Operator How-To: Making Changes to the System¶

For humans on the hub2 node who need to modify code, configuration, or respond to emergencies.

Overview¶

Files are read-only by default to prevent accidental or malicious corruption. But you can still modify them intentionally with sudo.

Read-only file → sudo chmod to make writable → edit → sudo chmod back to read-only

Every step is logged in the audit trail.

Scenario 1: Quick Hotfix to API Code¶

You found a bug in domain-manager and need to fix it immediately.

Step 1: Make File Writable¶

sudo chmod 644 /opt/charliehub/domain-manager/app/models.py

Step 2: Edit the File¶

# Use your favorite editor
nano /opt/charliehub/domain-manager/app/models.py

# Or use sed for scripted changes
sudo sed -i 's/old_logic/new_logic/g' /opt/charliehub/domain-manager/app/models.py

Step 3: Test Your Changes¶

# Verify Python syntax
python3 -m py_compile /opt/charliehub/domain-manager/app/models.py

# Run unit tests if available
cd /opt/charliehub/domain-manager
python3 -m pytest tests/test_models.py -v

Step 4: Re-Protect the File¶

sudo chmod 440 /opt/charliehub/domain-manager/app/models.py

Step 5: Commit to Git¶

cd /opt/charliehub
git add domain-manager/app/models.py
git commit -m "Fix: Correct auth validation logic in models.py

Details of the fix...
"
git push origin main

Step 6: Restart the Service¶

# Option A: Restart container (pulls from disk)
sudo docker restart charliehub_domain_manager_v3

# Option B: Wait for automatic reload (depends on setup)
sudo docker logs charliehub_domain_manager_v3 -f

Step 7: Verify It Works¶

# Check API is healthy
curl -s http://172.19.0.5:8001/health | jq .

# Test the specific functionality you fixed
curl -H "X-API-Key: $API_KEY" http://172.19.0.5:8001/api/domains | head

Audit trail automatically created for: - ✅ sudo chmod 644 - permission change logged - ✅ nano edits - if you used git/IDE with audit - ✅ git commit - change author + message logged - ✅ docker restart - container restart logged

Scenario 2: Emergency Database Update¶

Something went wrong with a domain configuration and the API is failing. You need to fix it directly.

⚠️ ONLY DO THIS IF:¶

The API is completely broken
You cannot fix it via API
You need to restore from a bad state

Step 1: Check the Problem¶

# Is the API running?
curl http://172.19.0.5:8001/health

# What domains are broken?
curl -H "X-API-Key: $API_KEY" http://172.19.0.5:8001/api/domains | jq '.[] | select(.status=="error")'

# Can you fix via API instead?
curl -X PUT http://172.19.0.5:8001/api/domains/24 \
  -H "X-API-Key: $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"domain":"...","backend_host":"...",...}'

Step 2: If API Fix Failed, Document the Emergency¶

# Create an incident log
cat > /tmp/incident_$(date +%s).md << 'LOG'
# INCIDENT: Database corruption 2026-02-08 20:45 UTC

## Symptoms
- Domain 24 has corrupted backend_host value
- API update fails with 500 error
- Traffic broken for control.trevarn.com

## Root Cause
- Unknown (investigate after fix)

## Direct DB Access Reason
- API completely broken, needed emergency fix
- Operator: $(whoami)
- Timestamp: $(date)

## Changes Made
[Document what was changed]

## Recovery
[How to prevent this happening again]
LOG

cat /tmp/incident_$(date +%s).md

Step 3: Access Database (Inside Container)¶

# IMPORTANT: Use docker exec, don't expose postgres port
# This connects through the docker network, not localhost

sudo docker exec charliehub-postgres psql \
  -U charliehub \
  -d charliehub_domains \
  -c "SELECT id, domain, backend_host FROM domains WHERE id = 24;"

# Output:
#  id |       domain        |   backend_host
# ----+---------------------+----------------
#  24 | control.trevarn.com | corrupted_value

Step 4: Fix the Specific Record¶

# Update the corrupted record
sudo docker exec charliehub-postgres psql \
  -U charliehub \
  -d charliehub_domains << 'SQL'
UPDATE domains SET backend_host = '10.44.1.214' WHERE id = 24;
SELECT id, domain, backend_host FROM domains WHERE id = 24;
SQL

# Verify:
# UPDATE 1
#  id |       domain        | backend_host
# ----+---------------------+---------------
#  24 | control.trevarn.com | 10.44.1.214

Step 5: Verify API Works Again¶

curl -H "X-API-Key: $API_KEY" http://172.19.0.5:8001/api/domains/24 | jq .backend_host
# Output: "10.44.1.214"

# Deploy to Traefik
curl -X POST http://172.19.0.5:8001/api/deploy-all -H "X-API-Key: $API_KEY"

Step 6: Post-Incident Review¶

# Audit trail will show:
# - Time of docker exec command
# - SQL executed
# - Which user ran it
# - All details

# Review the incident
sudo journalctl -u docker | grep charliehub-postgres | tail -20

# Send to team for review
cat /tmp/incident_*.md | mail -s "INCIDENT REPORT: DB Emergency Fix" team@charliehub.net

Scenario 3: Update Configuration Files¶

You need to modify Traefik or domain-manager configuration.

Step 1: Understand the Current State¶

# These files are AUTO-GENERATED
ls -la /opt/charliehub/traefik/config/dynamic/static-routes.yml
# -r--r----- 1 root docker 2048 Feb 8 20:45 static-routes.yml

# Never edit auto-generated files directly
# Instead: Update via API, then redeploy

Step 2: Make Changes Via API (Preferred)¶

# Instead of editing YAML directly:
curl -X PUT http://172.19.0.5:8001/api/domains/24 \
  -H "X-API-Key: $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"domain":"control.trevarn.com","backend_host":"...",...}'

# Then deploy:
curl -X POST http://172.19.0.5:8001/api/deploy-all \
  -H "X-API-Key: $API_KEY"

# The file is automatically regenerated ✅

Step 3: Only Edit Source Code Files¶

# ✅ OK to edit: Source code in git
sudo chmod 644 /opt/charliehub/domain-manager/app/models.py
nano /opt/charliehub/domain-manager/app/models.py
sudo chmod 440 /opt/charliehub/domain-manager/app/models.py
git add domain-manager/app/models.py
git commit -m "Update: ..."
git push origin main

# ❌ DO NOT EDIT: Generated files (static-routes.yml)
# These are overwritten by the next deploy anyway

Scenario 4: Emergency Permissions Change¶

The system is in a bad state and you need to temporarily make files writable to fix things.

Step 1: Understand Why You're Doing This¶

# This should be RARE. Ask yourself:
# - Have I tried fixing via API? (yes/no)
# - Is the API completely broken? (yes/no)
# - Have I documented the incident? (yes/no)
# - Have I told the team? (yes/no)

# If all answers are "yes", proceed:

Step 2: Remove Read-Only Protection Temporarily¶

# Make directory writable
sudo chmod 755 /opt/charliehub/domain-manager/app/

# Make specific file writable
sudo chmod 644 /opt/charliehub/domain-manager/app/models.py

# ⚠️ IMMEDIATELY document this
echo "CHMOD CHANGE: Made /opt/charliehub/domain-manager/app/models.py writable at $(date)" \
  >> /var/log/admin_changes.log

Step 3: Make Your Changes¶

# Minimal changes only
sed -i 's/bad_value/good_value/g' /opt/charliehub/domain-manager/app/models.py

# Verify
python3 -m py_compile /opt/charliehub/domain-manager/app/models.py

Step 4: Re-Protect Immediately¶

# Restore read-only
sudo chmod 440 /opt/charliehub/domain-manager/app/models.py
sudo chmod 555 /opt/charliehub/domain-manager/app/

# Document the fix
echo "CHMOD RESTORE: Re-protected /opt/charliehub/domain-manager/app/ at $(date)" \
  >> /var/log/admin_changes.log

Step 5: Commit and Push¶

cd /opt/charliehub
git add domain-manager/app/models.py
git commit -m "HOTFIX: Emergency database corruption fix

Reason: [Why was this emergency fix needed]
Method: Direct file modification (API was unavailable)
Test: [How was this tested]
Next: [What should be done to prevent this in future]

See incident report: /tmp/incident_*.md
"
git push origin main

Complete Audit Trail Check¶

After making changes, verify everything is logged:

#!/bin/bash
# Check what was recorded

echo "=== SUDO Commands ==="
sudo journalctl -u sudo | grep -E "chmod|models.py|docker" | tail -20

echo ""
echo "=== Docker Commands ==="
sudo journalctl -u docker | grep "charliehub_domain_manager" | tail -10

echo ""
echo "=== Git Commits ==="
cd /opt/charliehub
git log --oneline -10

echo ""
echo "=== File Modification Times ==="
stat -c "%y %n" /opt/charliehub/domain-manager/app/models.py

echo ""
echo "=== Current Permissions ==="
ls -la /opt/charliehub/domain-manager/app/models.py

File Permission Reference¶

Operation	Permission	Command
Read only (default)	440	`sudo chmod 440 file`
Editable	644	`sudo chmod 644 file`
Executable scripts	755	`sudo chmod 755 file`
Sensitive data	600	`sudo chmod 600 file`

Pre-Change Checklist¶

Before modifying anything on hub2:

[ ] I understand what I'm changing
[ ] I have documented the reason
[ ] I have tested my changes locally (if possible)
[ ] I have backups / can roll back
[ ] I have notified the team
[ ] I will commit changes to git
[ ] I will document the change in an incident report
[ ] I understand the audit trail will record all of this

Post-Change Checklist¶

After modifying anything:

[ ] The change works as expected
[ ] No services are broken
[ ] Changes are committed to git
[ ] Changes are pushed to remote
[ ] Audit trail is clean and readable
[ ] Team has been notified
[ ] Incident report (if needed) has been filed
[ ] Root cause analysis is documented

Emergency Contacts¶

If something goes wrong after your changes:

API is down → Check sudo docker logs charliehub_domain_manager_v3
Database is corrupted → Check git log to see what changed
Traefik routing broken → Check /opt/charliehub/traefik/config/dynamic/static-routes.yml
Need to rollback → git revert <commit_hash> && redeploy

Key Principles¶

Read-only by default - Prevents accidents
Sudo for everything - Creates audit trail
Git is truth - All changes committed
API first - Use API when possible
Document always - Leave breadcrumbs for next person
Test before deploying - Verify it works
Audit trail is law - Every action is logged

Last updated: 2026-02-08 For agents using the API: See AGENT_API_GUIDE.md