Cron Schedules¶
All scheduled jobs for the CharlieHub cluster.
Last Updated: 2026-02-04
PBS Backup Schedule (Primary)¶
Incremental backups to Proxmox Backup Server in France.
| Time (UTC) | Job | Node | Storage | Retention |
|---|---|---|---|---|
| 22:00 | pbs-px3-daily | px3 | pbs-fr | 7d, 4w, 2m |
| 00:30 | pbs-px2-daily | px2 | pbs-fr | 7d, 4w, 2m |
| 03:00 | pbs-px1-daily | px1 | pbs-fr | 7d, 4w, 2m |
| Sun 07:00 | pbs-weekly | px1 | pbs-fr | 8w, 3m |
Vzdump Schedule (UK Local)¶
Full backups to local NAS for fast UK restores.
| Time (UTC) | Job | Node | Storage | Retention |
|---|---|---|---|---|
| 05:30 | uk-secondary | px2, px3 | px3-nas | 5d, 2w |
Legacy Vzdump (Parallel Run)¶
These jobs run alongside PBS during the validation period (Feb 4-18, 2026).
| Time (UTC) | Job | Node | Storage |
|---|---|---|---|
| 22:00 | pikvm-px3 | px3 | pikvm-backup |
| 00:30 | pikvm-px2 | px2 | pikvm-backup |
| 03:00 | pikvm-px1 | px1 | pikvm-backup |
| 03:15 | backup-55292acc | px1 | pikvm-backup |
| Sun 07:00 | weekly-archive | px1 | pikvm-backup |
After validation, these will be disabled.
PBS Maintenance¶
| Schedule | Job | Purpose |
|---|---|---|
| Sat 02:00 | GC (pbs-main) | Garbage collection |
| Sun 04:00 | Verify (pbs-main) | Backup verification |
Ceph Scrub Window¶
| Start | End | Purpose |
|---|---|---|
| 09:00 | 17:00 | Scrubs allowed (business hours only) |
No scrubs during overnight backup window (17:00-09:00).
System Maintenance¶
| Schedule | Job | Node |
|---|---|---|
| 03:10 daily | e2scrub_all | px1 |
| 03:30 Sunday | e2scrub_all_cron | px1 |
Security Maintenance¶
| Schedule | Job | Purpose | Duration | Downtime |
|---|---|---|---|---|
| Last Sunday of Q1, Q2, Q3, Q4 @ 02:00 UTC | rotate-secrets.sh | Quarterly credential rotation | ~30 min | < 5 min |
2026 Schedule: - Q1: March 29, 2026 @ 02:00 UTC - Q2: June 28, 2026 @ 02:00 UTC - Q3: September 27, 2026 @ 02:00 UTC - Q4: December 27, 2026 @ 02:00 UTC
Details: - Rotates 13 credentials across 4 files (database passwords, API keys, JWT secrets, encryption keys) - Pre-rotation checklist starts 1 week before - Full procedures: Security Maintenance Guide
Hub2 Backups¶
| Time | Target | Path |
|---|---|---|
| 03:00 UTC | px3 (UK) | /mnt/nas-backup/hub2-snapshots/ |
| 03:00 UTC | px5 (FR) | /mnt/pve/pikvm-backup/hub2-offsite/ |
For full details, see:
Last updated: 2026-02-04
px3 NAS Maintenance Jobs¶
Custom cron jobs on px3 for NAS data management. All wrapped with ionice -c2 -n7 nice -n 19 to protect Ceph osd.0 from I/O contention (applied 2026-02-07).
| Schedule | Job | Purpose |
|---|---|---|
| Hourly | nas-tier-sync.sh | Move old files from fast tier to NAS |
| Daily 03:00 | nas-mirror-sync.sh | rsync NAS primary → backup mirror |
| Daily 08:00 | backup-verification.sh | Verify backup integrity |
| Sunday 06:00 | nas-offsite-backup.sh | rsync NAS → pikvm (France offsite) |
I/O Priority
These jobs must keep the ionice -c2 -n7 nice -n 19 prefix.
Without it, nas-mirror-sync at 03:00 can saturate the SATA controller
and trigger Ceph osd.0 suicide timeouts.
Last updated: 2026-02-07