SMTP Email Configuration

Complete guide to configuring and troubleshooting email delivery for alerts in CharlieHub.

Quick Reference

Setting	Value
SMTP Server	smtp.gmail.com
SMTP Port	587 (TLS)
Authentication	Gmail App Password
From Address	cpaumelle@eroundit.eu
Config Files	.env, alertmanager.yml

---

Configuration Files

1. Environment Variables (.env)

File: /opt/charliehub/.env

# Alert Recipients
ALERT_EMAIL=cpaumelle@eroundit.eu
ALERT_EMAIL_SECONDARY=chpa35@gmail.com
ALERT_EMAIL_CRITICAL_ONLY=cpaumelle@eroundit.eu

# SMTP Configuration
SMTP_HOST=smtp.gmail.com
SMTP_PORT=587
SMTP_USER=alerts@charliehub.net
SMTP_PASSWORD=kuum kiui aayk oqvj
SMTP_FROM=cpaumelle@eroundit.eu
SMTP_REQUIRE_TLS=true

2. Alertmanager Configuration (alertmanager.yml)

File: /opt/charliehub/monitoring/alertmanager/alertmanager.yml

global:
  resolve_timeout: 5m
  # SMTP settings for email alerts
  smtp_smarthost: 'smtp.gmail.com:587'
  smtp_from: 'cpaumelle@eroundit.eu'
  smtp_auth_username: 'alerts@charliehub.net'
  smtp_auth_password: 'kuum kiui aayk oqvj'
  smtp_require_tls: true
  smtp_hello: 'charliehub.net'

receivers:
  - name: 'email-alerts'
    email_configs:
      # Primary recipient
      - to: 'cpaumelle@eroundit.eu'
        send_resolved: true
        headers:
          Subject: '[{{ .Status | toUpper }}] {{ .GroupLabels.alertname }} - charliehub alert'

      # Secondary recipient (fallback)
      - to: 'chpa35@gmail.com'
        send_resolved: true
        headers:
          Subject: '[{{ .Status | toUpper }}] {{ .GroupLabels.alertname }} - charliehub (backup)'

⚠️ Known Issue: Email addresses are hardcoded in both files. They must be manually synchronized when changed.

---

Gmail App Password Setup

Prerequisites

• Gmail account with 2-factor authentication enabled
• Access to https://myaccount.google.com/apppasswords

Steps

1. Enable 2FA (if not already enabled)
2. Go to https://myaccount.google.com/security

3. Enable "2-Step Verification"

4. Generate App Password

5. Go to https://myaccount.google.com/apppasswords
6. Select "Mail" for app type
7. Select "Linux" for device type

8. Google will generate a 16-character password

9. Copy Password

10. Note the password format: abcd efgh ijkl mnop (spaces included)

11. Remove spaces when entering into configuration: abcdefghijklmnop

12. Update Configuration

    # Edit .env
    vi /opt/charliehub/.env
    # Update: SMTP_PASSWORD=<16-char-password>
    
    # Edit alertmanager.yml
    vi /opt/charliehub/monitoring/alertmanager/alertmanager.yml
    # Update: smtp_auth_password: '<16-char-password>'
    

13. Restart Alertmanager

    docker compose restart alertmanager
    docker compose logs alertmanager | tail -20
    

Troubleshooting Gmail App Password

Q: Where do I find my app passwords? A: https://myaccount.google.com/apppasswords

Q: I lost my app password, what do I do? A: Delete the old one and generate a new one in the Google Account settings.

Q: My password has spaces, do I include them? A: No, remove spaces. abcd efgh becomes abcdefgh

Q: How often do I need to renew the password? A: Gmail app passwords don't expire automatically.

---

Alternative Email Providers

SendGrid

Setup:

# Create SendGrid account and generate API key

# Update .env
SMTP_HOST=smtp.sendgrid.net
SMTP_PORT=587
SMTP_USER=apikey
SMTP_PASSWORD=SG.xxxxxxxxxxxxxxxxxxxxxxxxxxxxx
SMTP_FROM=noreply@yourdomain.com
SMTP_REQUIRE_TLS=true

Office 365 / Microsoft 365

Setup:

SMTP_HOST=smtp.office365.com
SMTP_PORT=587
SMTP_USER=your-email@company.com
SMTP_PASSWORD=your-office-password
SMTP_FROM=your-email@company.com
SMTP_REQUIRE_TLS=true

Postfix (Internal)

Setup:

SMTP_HOST=localhost
SMTP_PORT=25
SMTP_USER=           # Leave empty
SMTP_PASSWORD=       # Leave empty
SMTP_FROM=alerts@charliehub.net
SMTP_REQUIRE_TLS=false

See Postfix Setup for configuration details.

---

Email Delivery Testing

Test Email Sending

# Method 1: Trigger a test alert
docker compose stop prometheus
sleep 180  # Wait 3 minutes for PrometheusDown alert

# Check if Alertmanager has the alert
curl -s http://localhost:9093/api/v1/alerts | jq '.data[] | {alertname, status}'

# Restart Prometheus
docker compose start prometheus

# Wait for recovery alert (~5 minutes)

Verify SMTP Connectivity

# Test SMTP connection
docker exec charliehub_alertmanager \
  nc -zv smtp.gmail.com 587

# Response should be:
# smtp.gmail.com (142.251.x.x) open

Check Alertmanager Logs

# View recent logs
docker compose logs alertmanager | tail -30

# Filter for email-related messages
docker compose logs alertmanager | grep -i "email\|smtp\|mail"

# Filter for errors
docker compose logs alertmanager | grep -i "error"

---

Email Configuration Changes

Changing Alert Recipients

⚠️ Important: Changes must be made in TWO files

Step 1: Update .env

vi /opt/charliehub/.env

# Find and update:
ALERT_EMAIL=<new-primary-email>
ALERT_EMAIL_SECONDARY=<new-secondary-email>
SMTP_FROM=<new-from-address>

Step 2: Update alertmanager.yml

vi /opt/charliehub/monitoring/alertmanager/alertmanager.yml

# Find and update:
# Line 6: smtp_from: '<new-from-address>'
# Line 45: to: '<new-primary-email>'
# Line 51: to: '<new-secondary-email>'

Step 3: Restart Alertmanager

docker compose restart alertmanager
sleep 5
docker compose logs alertmanager | grep "Completed loading"

Step 4: Verify

# Check config was loaded
docker exec charliehub_alertmanager cat /etc/alertmanager/alertmanager.yml | grep "to:"

# Expected output:
# - to: '<new-primary-email>'
# - to: '<new-secondary-email>'

Changing SMTP Password

Step 1: Generate new password - Go to https://myaccount.google.com/apppasswords (if using Gmail) - Generate new 16-character password

Step 2: Update both files

# .env
vi /opt/charliehub/.env
# Update: SMTP_PASSWORD=<new-password>

# alertmanager.yml
vi /opt/charliehub/monitoring/alertmanager/alertmanager.yml
# Update line 8: smtp_auth_password: '<new-password>'

Step 3: Restart Alertmanager

docker compose restart alertmanager
docker compose logs alertmanager | tail -20

Adding Multiple Recipients

Option A: Primary + Secondary (Current)

Current configuration already supports this: - Primary recipient: Receives all alerts - Secondary recipient: Receives alerts after 5 minutes if primary unresponsive

Option B: Add More Recipients

Edit alertmanager.yml:

email_configs:
  - to: 'primary@example.com'
    send_resolved: true

  - to: 'secondary@example.com'
    send_resolved: true
    group_wait: 5m  # Delayed

  - to: 'critical-only@example.com'
    send_resolved: true
    match:
      severity: critical  # Only critical alerts

---

Troubleshooting Email Issues

Emails Not Arriving

Symptom: Alert fires in Prometheus, but no email received

Diagnosis Steps:

# 1. Is Alertmanager running?
docker compose ps alertmanager

# 2. Did Alertmanager start without errors?
docker compose logs alertmanager | grep -i "error\|failed"

# 3. Is there an active alert?
curl -s http://localhost:9093/api/v1/alerts | jq '.data[] | {alertname, status}'

# 4. Check SMTP connectivity
docker exec charliehub_alertmanager nc -zv smtp.gmail.com 587

# 5. Are credentials correct?
docker exec charliehub_alertmanager cat /etc/alertmanager/alertmanager.yml | grep -A 5 "smtp_"

Common Causes & Fixes:

Issue	Symptom	Fix
Wrong Gmail password	"authentication failed" in logs	Regenerate app password at myaccount.google.com/apppasswords
Gmail account locked	"Please log in via your web browser"	Unlock account at myaccount.google.com/security
2FA not enabled	"Application password not allowed"	Enable 2-factor authentication first
Email address typo	Alerts routed to wrong address	Double-check both files; restart Alertmanager
Firewall blocking	"Connection timeout"	Check network policy allows port 587 outbound
SMTP disabled	"Connection refused"	Enable SMTP in email provider settings

"SMTP Error: 535 5.7.8 Authentication failed"

Cause: Wrong password

Fix:

# 1. Regenerate Gmail app password
# Visit: https://myaccount.google.com/apppasswords
# Select Mail + Linux
# Copy 16-character password

# 2. Update both config files (no spaces in password)
SMTP_PASSWORD=abcdefghijklmnop  # No spaces

# 3. Restart
docker compose restart alertmanager

"554 5.5.1 Invalid address"

Cause: Invalid email address in configuration

Fix:

# Check email format
# ✓ user@example.com
# ✗ user@
# ✗ @example.com
# ✗ user example@example.com (spaces)

# Verify in both files
grep "to:" /opt/charliehub/monitoring/alertmanager/alertmanager.yml
grep "ALERT_EMAIL" /opt/charliehub/.env

"TLS 1.3 Connection failed"

Cause: SMTP_REQUIRE_TLS disabled or version mismatch

Fix:

# Ensure TLS is enabled
SMTP_REQUIRE_TLS=true

# Update alertmanager.yml
smtp_require_tls: true

# Restart
docker compose restart alertmanager

Emails Going to Spam

Cause: SMTP FROM address not SPF/DKIM signed

Fix: - Use email address verified in Gmail account - Set up SPF record for your domain (if sending from custom domain) - Use branded FROM address instead of generic noreply

# Example:
SMTP_FROM=noreply@yourdomain.com  # Not: noreply@charliehub.net

---

Configuration Validation

Validate Email Format

# Check all email addresses in configuration
grep -r "ALERT_EMAIL\|@.*\.com\|to:" /opt/charliehub/{.env,monitoring/alertmanager/}

# Expected pattern: user@domain.com (no spaces, valid domain)

Validate YAML Syntax

# Check alertmanager.yml syntax
docker run --rm -v /opt/charliehub/monitoring/alertmanager:/config \
  alpine:latest sh -c 'grep -E "^[a-z_]+:" /config/alertmanager.yml'

# Or manually inspect
cat /opt/charliehub/monitoring/alertmanager/alertmanager.yml | head -20

Test SMTP with telnet/nc

# Manual SMTP test
docker exec charliehub_alertmanager sh -c '
  (
    sleep 1
    echo "EHLO charliehub.net"
    sleep 1
  ) | nc smtp.gmail.com 587
'

# Expected: 220 smtp.google.com ESMTP response

---

Security Considerations

Password Management

✅ Do: - Use Gmail app passwords (specific to app/device) - Store passwords in .env (outside version control) - Regenerate if password ever appears in logs - Use strong passwords from password manager

❌ Don't: - Hardcode Gmail account passwords - Commit passwords to Git - Use personal Gmail password - Reuse password across services

Configuration Sync

⚠️ Current Issue: Email addresses hardcoded in two files

Risk: Configuration drift if values get out of sync

Mitigation:

# Always update both files together:
vi /opt/charliehub/.env
vi /opt/charliehub/monitoring/alertmanager/alertmanager.yml
docker compose restart alertmanager

Future: See TECHNICAL-DEBT.md for planned refactoring to use environment variable substitution.

---

References

• Alertmanager Email Configuration
• Gmail App Passwords Help
• SMTP RFC 5321
• Alert Configuration Guide

---

Last Updated: 2026-02-06 Status: Production Ready