Skip to content

Operator Guides

Documentation for operators and DevOps engineers managing the CharlieHub infrastructure.

🔧 Making Changes

  • How-To Guide - Step-by-step procedures for changing the system
  • Covers hotfixes, emergency database updates, and configuration changes
  • Includes pre/post-change checklists and audit trail verification

🔐 Security & Safeguards

  • Safeguards Policy - Security policies and enforcement mechanisms
  • API-first enforcement, change procedures, and incident response
  • Audit trail requirements and compliance procedures
  • Security Maintenance - Quarterly credential rotation procedures
  • Pre-rotation checklists, execution procedures, emergency rollback, troubleshooting
  • Quarterly schedule: Last Sunday of each quarter @ 02:00 UTC

Key Concepts

File Permissions

Files are read-only by default to prevent accidental corruption: - Code files: 444 (r--r--r--) - Config files: 440 (r--r-----) - Directories: 555 (r-xr-xr-x)

To modify: 

sudo chmod 644 /path/to/file     # Make writable
nano /path/to/file               # Edit
sudo chmod 440 /path/to/file     # Protect again
git add /path/to/file            # Commit change
git commit -m "Your message"

Database Access

PostgreSQL localhost access is blocked by UFW firewall rule. For emergencies: 

# Access database via docker (doesn't expose port)
sudo docker exec charliehub-postgres psql \
  -U charliehub \
  -d charliehub_domains \
  -c "SELECT * FROM domains WHERE id = 24;"

Audit Trail

Everything is logged automatically: 

charliehub-audit-report              # View audit summary
sudo journalctl -u sudo              # See sudo commands
git log                              # See code changes
sudo ausearch -k domain_manager_code # See file modifications

Common Scenarios

Scenario 1: Quick Hotfix to API Code

→ Read: How-To - Scenario 1

Steps: chmod → edit → test → chmod → git commit → restart

Scenario 2: Emergency Database Fix

→ Read: How-To - Scenario 2

Only if API is completely broken. Use docker exec, not direct psql.

Scenario 3: Update Configuration

→ Read: How-To - Scenario 3

Prefer API changes over file edits. Generated files auto-regenerate.

Scenario 4: Emergency Permissions Change

→ Read: How-To - Scenario 4

Rare situation. Make temporary, fix issue, protect immediately.

Verification Commands

Check what changed: 

# Last 24 hours of sudo commands
sudo journalctl -u sudo --since "24 hours ago" | grep domain_manager

# File permission changes
sudo journalctl | grep chmod

# Git commits
git log --since="24 hours ago"

# Docker commands
sudo journalctl -u docker | grep charliehub

Pre-Change Checklist

Before modifying anything: - [ ] Understand what I'm changing - [ ] Have documented the reason - [ ] Tested changes locally if possible - [ ] Have backups / can roll back - [ ] Notified the team - [ ] Will commit changes to git - [ ] Will document in incident report if needed

Post-Change Checklist

After modifying anything: - [ ] Change works as expected - [ ] No services are broken - [ ] Changes are committed to git - [ ] Changes are pushed to remote - [ ] Audit trail is clean - [ ] Team has been notified - [ ] Incident report is filed (if needed)

Escalation Path

If something goes wrong after your changes:

  1. API is down 

    sudo docker logs charliehub_domain_manager_v3

  2. Database is corrupted 

    git log                    # See what changed
git revert <commit_hash>   # Revert the change

  3. Traefik routing broken 

    cat /opt/charliehub/traefik/config/dynamic/static-routes.yml
curl -X POST http://172.19.0.5:8001/api/deploy-all \
  -H "X-API-Key: $API_KEY"  # Redeploy

  4. Need to rollback 

    git revert <commit_hash>
git push origin main
# Services will pick up the new code automatically

Policies

API-First Enforcement

  • Agents use API exclusively (files are read-only)
  • Changes via API are validated and logged
  • API is the only supported way to modify infrastructure

Change Authorization

  • All changes tracked in git
  • Every change has an author and message
  • Sudo commands logged to journalctl
  • Audit trail is law

Incident Response

  • Document all emergencies
  • Create incident reports
  • Perform post-mortems
  • Prevent recurrence

Questions?

  • How to make changes? → Read How-To Guide
  • What are the policies? → Read Safeguards
  • Need to check audit trail? → Run charliehub-audit-report
  • Need to rollback? → Use git revert

Last updated: 2026-02-08