Cluster Overview¶
CharlieHub infrastructure consists of a central dedicated server (hub2) connected to homelab Proxmox clusters via WireGuard site-to-site VPN.
Architecture¶
Internet
│
▼
┌─────────────────────────────┐
│ hub2 (OVH Dedicated) │
│ 51.68.235.106 │
│ ┌───────────────────────┐ │
│ │ Traefik (Parent) │ │
│ │ Authelia SSO │ │
│ │ Domain Manager │ │
│ │ UniFi API │ │
│ │ Prometheus/Grafana │ │
│ │ Docs, Code Server │ │
│ └───────────────────────┘ │
└─────────────┬───────────────┘
│ WireGuard VPN
│
┌───────────────────┴───────────────────┐
│ │
▼ ▼
┌─────────────────────────────────────┐ ┌──────────────────────────┐
│ UK SITE (10.44.x.x) │ │ FRANCE SITE (10.35.x.x)│
│ ┌─────────────────────────────────┐ │ │ ┌──────────────────────┐ │
│ │ px1-silverstone px2-monza │ │ │ │ px5-lemans │ │
│ │ REDACTED_IP REDACTED_IP │ │ │ │ REDACTED_IP │ │
│ │ osd.2,osd.3 osd.1,osd.4 │ │ │ │ (DR site) │ │
│ │ │ │ │ │ │ │
│ │ px3-suzuka │ │ │ │ │ │
│ │ REDACTED_IP │ │ │ │ │ │
│ │ osd.0 │ │ │ │ │ │
│ │ │ │ │ │ │ │
│ │ ┌─────────────────────┐ │ │ │ │ │ │
│ │ │ CEPH POOL │ │ │ │ │ │ │
│ │ │ size=3 │ │ │ │ │ │ │
│ │ │ (all 3 nodes) │ │ │ │ │ │ │
│ │ └─────────────────────┘ │ │ │ │ │ │
│ └─────────────────────────────────┘ │ │ └──────────────────────┘ │
│ │ │ │
│ UniFi UCG: REDACTED_IP │ │ UniFi UCG: REDACTED_IP │
└─────────────────────────────────────┘ └──────────────────────────┘
Components¶
hub2 (OVH Dedicated Server)¶
The central hub running all public-facing services:
| Property | Value |
|---|---|
| Public IP | 51.68.235.106 |
| WireGuard IPs | wg-uk: REDACTED_IP, wg-fr: REDACTED_IP |
| SSH User | ubuntu (not root) |
| Location | OVH Datacenter |
| Purpose | Central services hub |
See hub2 Documentation for full details.
UK Homelab Nodes¶
| Node | Location | IP | Role | Storage |
|---|---|---|---|---|
| px1-silverstone | UK (Loft) | REDACTED_IP | Primary, Mon, OSD.2 + OSD.3 | 2TB SSD + 1.7TB NVMe |
| px2-monza | UK (Loft) | REDACTED_IP | Dev/Staging, Mon, OSD.1 + OSD.4 | 2TB SSD + 1.8TB NVMe |
| px3-suzuka | UK (Loft) | REDACTED_IP | NAS + Storage, Mon, OSD.0 | 2TB SSD (Ceph) |
France Cluster Node¶
| Node | Location | IP | Role | Storage |
|---|---|---|---|---|
| px5-lemans | France | REDACTED_IP | Active cluster member, Mon | Local NVMe (ZFS RAID1) |
Storage Architecture¶
Ceph RBD (Primary Storage)¶
- Pool:
ceph-pool - Replication: size=3, min_size=2 (data on ALL 3 UK nodes: px1, px2, px3)
- Total Capacity: ~8.9 TiB raw, ~2.0 TiB usable
- OSDs: 5 (osd.0 on px3, osd.1+osd.4 on px2, osd.2+osd.3 on px1)
- Used For: All critical VMs/CTs (automatic HA across UK nodes)
- Accessible From: All 4 cluster nodes including px5-lemans (over WAN)
Local Storage¶
- local-lvm: Per-node LVM for non-HA workloads
- backup-storage: USB drive on px1 for vzdump backups
- pikvm-backup: NFS from PiKVM for off-site backups
High Availability¶
Ceph Replication¶
All data on ceph-pool is automatically written to all 3 UK nodes. If any node fails, data remains available on the other 2.
Proxmox HA¶
Critical VMs are managed by Proxmox HA for automatic failover:
| VM/CT | Name | Node | HA Status |
|---|---|---|---|
| ct:1112 | prod-database-postgre | px1 | HA Enabled |
| ct:1113 | prod-iot-platform | px1 | HA Enabled |
| ct:1119 | vpn-wg-manager | px1 | HA Enabled |
| Linux Mint | isp-monitor | Direct to Technicolor | Migrated from CT1118 |
| ct:1935 | pescle-rodent | px1 | HA Enabled |
| ct:1945 | zoho-books-api | px1 | HA Enabled |
| vm:1123 | cbre-api-people-counting | px1 | HA Enabled |
| ct:2912 | CT2912 | px2 | HA Enabled |
| ct:2913 | difenn-sprint1 | px2 | HA Enabled |
| ct:2920 | trevarn-core | px2 | HA Enabled |
| ct:2929 | trevarn-brand | px2 | HA Enabled |
| ct:3102 | homelab-monitor | px3 | HA Enabled |
| vm:3970 | rpautoparts-store | px3 | HA Enabled |
Quorum¶
- Cluster Nodes: 4 nodes (1 vote each: px1, px2, px3, px5)
- QDevice: Corosync QDevice at REDACTED_IP (1 vote, France site for tie-breaker)
- Total Votes: 5 (4 nodes + 1 qdevice)
- Quorum Required: 3 votes
- Status: Fully quorate with WAN-resilient architecture over 20ms UK↔FR link
- Tie-breaker: QDevice ensures cluster stability if WAN partition occurs
Key Services¶
| Service | Location | IP/URL | Purpose |
|---|---|---|---|
| hub2 | OVH Dedicated | 51.68.235.106 | Central services hub |
| Corosync QDevice | France | REDACTED_IP | Cluster quorum tie-breaker (WAN-resilient) |
| PostgreSQL | CT1912 | REDACTED_IP | Databases |
| IoT Platform | CT1113 | REDACTED_IP3 | MQTT, ChirpStack |
| UniFi UCG (UK) | UK | REDACTED_IP | Network management |
| UniFi UCG (FR) | France | REDACTED_IP | Network management |
WireGuard VPN¶
hub2 connects to homelabs via WireGuard site-to-site VPN through the UniFi UCGs:
| Interface | Hub2 IP | Routes To | Peer |
|---|---|---|---|
| wg-uk | REDACTED_IP | REDACTED_SUBNET | uk-ucg |
| wg-fr | REDACTED_IP | REDACTED_SUBNET | fr-dnr-ucg |
Direct routing
WireGuard provides direct routing to homelab subnets. All px1-px5 nodes are reachable via their LAN IPs.
Networking¶
- UK LAN: 10.44.1.x (hosts, VMs, CTs on main LAN - VLAN 10 retired)
- France LAN: 10.35.1.x
- IoT: 10.x.5.x
- Cross-site: UniFi SD-WAN VPN between UK and FR
Quick Commands¶
# Connect to hub2 (uses ubuntu user)
ssh hub2 # with SSH config alias
ssh ubuntu@51.68.235.106 # direct
# Check WireGuard status (from hub2)
sudo wg show
# Cluster status (from any Proxmox node)
pvecm status
# Ceph status
ceph -s
# HA status
ha-manager status
# List all VMs/CTs
qm list && pct list
Migration History¶
| Date | Change |
|---|---|
| 2026-01-27 | px5-lemans successfully rejoined cluster post-migration, 4-node cluster now operational with WAN-resilient quorum |
| 2026-01-19 | hub2 deployed, replacing hub1 after DDoS |
| 2026-01-19 | WireGuard VPN configured for homelab connectivity |
| 2026-01-19 | All DNS records pointed to hub2 (51.68.235.106) |
| 2026-01-19 | Daily backups to UK (px3) and FR (px5) configured |
| 2026-01-05 | VM5111 migrated to hub1 |
| 2026-01-05 | DDNS disabled (static IP) |