Traefik Config Backup & Recovery System¶

Last Updated: 2026-02-09 Status: ACTIVE

Overview¶

This document describes the 3-layer defense system for Traefik configuration safety:

Backup - Hourly automatic snapshots
Detection - Real-time monitoring for unauthorized changes
Recovery - One-command restore from any backup point

Even though file permissions enforce read-only (444), this system provides defense in depth in case someone bypasses the mandate or permission system.

Quick Start¶

Create an hourly backup job (one-time setup)¶

/opt/charliehub/setup-traefik-backup-cron.sh

This will: - Create hourly backups at 00 minutes of each hour - Store in /opt/charliehub/.traefik-backups/ - Keep 30 days of history - Log to /var/log/charliehub/traefik-backups.log

List available backups¶

charliehub-restore-traefik

Output:

Available Traefik config backups:
==================================

  traefik-config_20260209_130000 [2026-02-09 13:00:00, 120K, 12 files]
  traefik-config_20260209_120000 [2026-02-09 12:00:00, 120K, 12 files]
  traefik-config_20260209_110000 [2026-02-09 11:00:00, 120K, 12 files]

Restore from a backup¶

charliehub-restore-traefik traefik-config_20260209_130000

Steps through: 1. Shows what will be restored 2. Asks for confirmation 3. Creates safety backup first 4. Restores files 5. Verifies checksums 6. Guides next steps

Monitor for unauthorized changes¶

charliehub-detect-traefik --status

Complete System¶

1. Automatic Hourly Backups¶

Script: /opt/charliehub/backup-traefik-configs.sh Command: charliehub-backup-traefik Frequency: Hourly (via cron) Setup: ./setup-traefik-backup-cron.sh

What It Does¶

Creates timestamped backup directory: traefik-config_YYYYMMDD_HHMMSS
Copies all .yml and .yaml files from /opt/charliehub/traefik/config/dynamic/
Generates checksum file for integrity verification
Creates metadata file with backup info
Deletes backups older than 30 days

Backup Location¶

/opt/charliehub/.traefik-backups/
├── traefik-config_20260209_130000/
│   ├── static-routes.yml
│   ├── middlewares.yml
│   ├── rodent-microshare.yml
│   ├── .checksums              ← SHA256 hashes
│   └── .backup-info             ← Metadata
├── traefik-config_20260209_120000/
│   ├── ...
└── ...

Log File¶

tail -f /var/log/charliehub/traefik-backups.log

Output:

[2026-02-09 13:00:01] Backing up Traefik configs...
[2026-02-09 13:00:01] Backup created: /opt/charliehub/.traefik-backups/traefik-config_20260209_130000
[2026-02-09 13:00:02] Cleaning up backups older than 30 days...
[2026-02-09 13:00:02] Recent backups:
  traefik-config_20260209_130000
  traefik-config_20260209_120000
  traefik-config_20260209_110000
[2026-02-09 13:00:02] Backup complete

2. Real-Time Change Detection¶

Script: /opt/charliehub/detect-traefik-changes.sh Command: charliehub-detect-traefik Frequency: Every 5 minutes (via cron) Setup: Manual (add to crontab)

What It Does¶

Monitors all Traefik YAML files for changes
Compares file sizes, timestamps, and content
Logs changes to /var/log/charliehub/traefik-changes.log
Also logs to systemd journal (searchable with journalctl)
Exits with code 1 if changes detected (can trigger external alerts)

Install¶

# Add to crontab:
crontab -e

# Add line:
*/5 * * * * /opt/charliehub/detect-traefik-changes.sh

# Save and exit

Check Status¶

charliehub-detect-traefik --status

Output:

Traefik Config Change Detection Status
======================================
Config directory: /opt/charliehub/traefik/config/dynamic
Last state file: /opt/charliehub/.traefik-last-state
Alert log: /var/log/charliehub/traefik-changes.log

Last checked: Sun Feb  9 13:45:22 2026
Current state:
. 1707500000 4096 .
./static-routes.yml 1707481234 2048 ./static-routes.yml
./middlewares.yml 1707481234 1024 ./middlewares.yml
...

Recent alerts:
[2026-02-09 13:45:22] ALERT: Traefik config files have been modified!
[2026-02-09 13:45:22] ALERT: Differences:
[2026-02-09 13:45:22] ALERT:   M rodent-microshare.yml

View Alerts¶

# Recent changes (last 50)
tail -50 /var/log/charliehub/traefik-changes.log

# Search for alerts
grep "ALERT" /var/log/charliehub/traefik-changes.log

# Real-time monitoring
journalctl -t charliehub-traefik -f

Reset Monitoring¶

If you make legitimate changes via the API and want to reset the state:

charliehub-detect-traefik --reset

This clears the change history so future detections start fresh.

3. One-Command Recovery¶

Script: /opt/charliehub/restore-traefik-configs.sh Command: charliehub-restore-traefik

List Backups¶

charliehub-restore-traefik

Restore from Backup¶

charliehub-restore-traefik traefik-config_20260209_130000

Steps: 1. ✅ Shows what will be restored 2. ✅ Asks for confirmation 3. ✅ Automatically creates safety backup first → traefik-config_SAFETY_20260209_144523 4. ✅ Restores all YAML files 5. ✅ Verifies checksums 6. ✅ Guides next steps

Safety Features¶

Before restoring: Creates a safety backup of current state
Checksums: Verifies file integrity after restore
Confirmation: Requires explicit "yes" confirmation before proceeding
Reversible: Can always restore the safety backup if needed

After Restore¶

After restoring from backup, you must redeploy Traefik:

charliehub-domain deploy

Or restart Traefik manually if needed:

sudo docker compose -f /opt/charliehub/docker-compose.yml restart traefik

Disaster Recovery Scenarios¶

Scenario 1: Someone Edited Traefik YAML Directly¶

Detection:

# Cron job detects it automatically, or you notice:
charliehub-detect-traefik --status
# Shows recent alerts about changes

Recovery:

# List backups from before the bad edit
charliehub-restore-traefik

# Restore good state
charliehub-restore-traefik traefik-config_20260209_130000

# Redeploy
charliehub-domain deploy

Scenario 2: Traefik Config is Corrupted¶

Symptoms: - Services return 502 Bad Gateway - Traefik container crashes - Invalid YAML syntax errors

Recovery:

# Restore from last known good backup
charliehub-restore-traefik traefik-config_20260209_100000

# Redeploy
charliehub-domain deploy

# Verify
curl https://yourdomain.charliehub.net/

Scenario 3: Accidental File Deletion¶

Recovery:

# All deleted files are in backups
charliehub-restore-traefik traefik-config_20260209_120000

Scenario 4: Restore Restore (Oops)¶

If you restored wrong backup:

# Restore the safety backup that was created
charliehub-restore-traefik traefik-config_SAFETY_20260209_144523

# Or restore from before that
charliehub-restore-traefik traefik-config_20260209_140000

Cron Jobs Reference¶

Backup Job (Hourly)¶

crontab -l | grep backup-traefik
# Output: 0 * * * * /opt/charliehub/backup-traefik-configs.sh >> /var/log/charliehub/traefik-backups.log 2>&1

Runs at: :00 of every hour (13:00, 14:00, 15:00, etc.)

Detection Job (Every 5 Minutes)¶

crontab -e
# Add: */5 * * * * /opt/charliehub/detect-traefik-changes.sh

Runs at: Every 5 minutes (13:00, 13:05, 13:10, etc.)

View All Jobs¶

crontab -l

Remove a Job¶

crontab -e
# Delete the line

Log Files¶

Backup Log¶

/var/log/charliehub/traefik-backups.log

What's logged: - When backups are created - Which files were backed up - Size and file count - Cleanup of old backups

Change Detection Log¶

/var/log/charliehub/traefik-changes.log

What's logged: - Timestamps of changes - Which files were modified - Git status if available - Full diff output

Systemd Journal¶

# Search all charliehub traefik alerts
journalctl -t charliehub-traefik

# Real-time tail
journalctl -t charliehub-traefik -f

# Last 100 entries
journalctl -t charliehub-traefik -n 100

File Permissions & Security¶

Why Backups are in `.traefik-backups/`¶

The . prefix means the directory is hidden: ls won't show it by default.

# See backups (need to use -a flag)
ls -la /opt/charliehub/.traefik-backups/

# Regular ls doesn't show it
ls /opt/charliehub/  # .traefik-backups not listed

Backup Directory Permissions¶

/opt/charliehub/.traefik-backups/
- Owner: ubuntu (or root)
- Permissions: 755 (rwxr-xr-x)
- Readable by: everyone (audit trail)
- Writable by: owner only

Original Config Permissions¶

/opt/charliehub/traefik/config/dynamic/*.yml
- Permissions: 444 (r--r--r--)
- Read-only for all users
- Even sudo can't write without explicit chmod

Troubleshooting¶

Backups not being created¶

# Check cron job exists
crontab -l | grep backup

# Check if directory is writable
ls -la /opt/charliehub/.traefik-backups/

# Check log file
tail -50 /var/log/charliehub/traefik-backups.log

# Run manually to see errors
/opt/charliehub/backup-traefik-configs.sh

Restore fails with "Backup not found"¶

# Make sure backup name is correct
charliehub-restore-traefik  # Shows correct names

# Check backup exists
ls -la /opt/charliehub/.traefik-backups/traefik-config_20260209_130000/

Changes not detected¶

# Check if detection job is running
crontab -l | grep detect-traefik

# Check status
charliehub-detect-traefik --status

# Reset and re-initialize
charliehub-detect-traefik --reset
sleep 5
charliehub-detect-traefik

Restore verification fails¶

# Checksums might not match perfectly (that's OK)
# Files are still restored

# After restore, always redeploy
charliehub-domain deploy

# Test services
curl https://yourdomain.charliehub.net/

Integration with Domain Manager API¶

The backup system is complementary to the API-only mandate:

API Changes (Recommended)
    ↓
charliehub-domain add/update/delete
    ↓
Changes go to: Database → Generated Traefik YAML files
    ↓
Backup Job (hourly)
    ↓
/opt/charliehub/.traefik-backups/

Change Detection (every 5 min)
    ↓
Monitors for unauthorized edits

How They Work Together¶

Primary workflow: Use charliehub-domain API
Backup: Automatic hourly snapshots capture current state
Detection: Alerts if someone bypasses the API
Recovery: Can restore from any backup point

This means: - ✅ If using API correctly: Backups just happen silently - ✅ If someone bypasses API: Detection alerts you, recovery available - ✅ If disaster happens: Restore with one command

Best Practices¶

✅ DO: - Use charliehub-domain for all changes (API-first) - Let backups run automatically (don't disable) - Check detection alerts regularly - Test restore procedure monthly - Keep backups for at least 30 days

❌ DON'T: - Edit YAML files directly (even with sudo) - Disable backup or detection jobs - Ignore change detection alerts - Keep backups for more than 90 days (storage) - Restore without confirmation

Commands Reference¶

Command	Purpose
`charliehub-backup-traefik`	Run manual backup now
`charliehub-restore-traefik`	List backups or restore
`charliehub-restore-traefik <name>`	Restore specific backup
`charliehub-detect-traefik`	Run detection check
`charliehub-detect-traefik --status`	Show status & recent alerts
`charliehub-detect-traefik --reset`	Reset detection state
`/opt/charliehub/setup-traefik-backup-cron.sh`	Setup automated backups

Support¶

For questions: - Backup: See /var/log/charliehub/traefik-backups.log - Detection: See /var/log/charliehub/traefik-changes.log - General: Ask DevOps team

For issues: - Restore not working? Check backup exists: charliehub-restore-traefik - Changes not detected? Reset: charliehub-detect-traefik --reset - Backups not creating? Check cron: crontab -l

Last updated: 2026-02-09 System deployed to protect against unauthorized Traefik configuration changes

Traefik Config Backup & Recovery System¶

Overview¶

Quick Start¶

Create an hourly backup job (one-time setup)¶

List available backups¶

Restore from a backup¶

Monitor for unauthorized changes¶

Complete System¶

1. Automatic Hourly Backups¶

What It Does¶

Backup Location¶

Log File¶

2. Real-Time Change Detection¶

What It Does¶

Install¶

Check Status¶

View Alerts¶

Reset Monitoring¶

3. One-Command Recovery¶

List Backups¶

Restore from Backup¶

Safety Features¶

After Restore¶

Disaster Recovery Scenarios¶

Scenario 1: Someone Edited Traefik YAML Directly¶

Scenario 2: Traefik Config is Corrupted¶

Scenario 3: Accidental File Deletion¶

Scenario 4: Restore Restore (Oops)¶

Cron Jobs Reference¶

Backup Job (Hourly)¶

Detection Job (Every 5 Minutes)¶

View All Jobs¶

Remove a Job¶

Log Files¶

Backup Log¶

Change Detection Log¶

Systemd Journal¶

File Permissions & Security¶

Why Backups are in .traefik-backups/¶

Backup Directory Permissions¶

Original Config Permissions¶

Troubleshooting¶

Backups not being created¶

Restore fails with "Backup not found"¶

Changes not detected¶

Restore verification fails¶

Integration with Domain Manager API¶

How They Work Together¶

Best Practices¶

Commands Reference¶

Support¶

Why Backups are in `.traefik-backups/`¶