Agent Troubleshooting Guide: "I Got an Error, Now What?"¶

Overview¶

You're reading this because you tried to do something and got an error. That's actually good - the error is there to guide you to the right way.

This guide explains every error you might encounter and exactly what to do instead.

Common Errors & Solutions¶

Error 1: "Permission denied" when editing files¶

$ nano /opt/charliehub/traefik/config/dynamic/static-routes.yml
nano: /opt/charliehub/traefik/config/dynamic/static-routes.yml: Permission denied

Why this happened: - Files are read-only by design - This prevents accidental or malicious corruption of the infrastructure

What you should do instead: - ✅ Use the REST API to make changes - ✅ See AGENT_API_GUIDE.md for complete examples

Quick example:

# Instead of editing the YAML file:
# ❌ nano /opt/charliehub/traefik/config/dynamic/static-routes.yml

# ✅ Use the API:
curl -X POST http://172.19.0.5:8001/api/domains \
  -H "X-API-Key: $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "domain": "myservice.charliehub.net",
    "backend_host": "10.44.1.100",
    "backend_port": 8080,
    ...
  }'

# Then deploy:
curl -X POST http://172.19.0.5:8001/api/deploy-all \
  -H "X-API-Key: $API_KEY"

See also: - AGENT_API_GUIDE.md - Complete API documentation - Section: "How to Do Everything via API"

Error 2: "Connection refused" when connecting to database¶

$ psql -h 127.0.0.1 -U charliehub -d charliehub_domains
psql: could not connect to server: Connection refused

Why this happened: - Direct database access is blocked by firewall - This prevents agents from bypassing the API and corrupting data directly - The API is the only validated way to modify domains

What you should do instead: - ✅ Use the REST API for all domain operations - ✅ The API handles all database operations safely

If you have a legitimate reason to query data:

# Instead of direct SQL:
# ❌ psql -h 127.0.0.1 -U charliehub

# ✅ Use the API:
curl -H "X-API-Key: $API_KEY" http://172.19.0.5:8001/api/domains | jq .

# Get specific domain:
curl -H "X-API-Key: $API_KEY" http://172.19.0.5:8001/api/domains/24 | jq .

See also: - AGENT_API_GUIDE.md - Section: "List All Domains" and "Get Details of One Domain"

Error 3: "Permission denied" when trying to modify code files¶

$ sed -i 's/old_config/new_config/g' /opt/charliehub/domain-manager/app/models.py
sed: couldn't open /opt/charliehub/domain-manager/app/models.py: Permission denied

Why this happened: - Code files are read-only to prevent corruption - Changes to code should go through a proper review and deployment process - This isn't something agents should be doing anyway

What you should do instead: - ✅ Use the API for configuration changes (not code changes) - ✅ If you found a bug in the code, file an issue instead - ✅ Let humans review and deploy code changes

If there's a bug in the API itself:

# Create an issue report instead
cat > /tmp/issue.md << EOF
**Issue**: Domain update fails with 500 error when backend_port is 443
**Steps to reproduce**:
1. POST /api/domains with backend_port: 443
2. See error: ...

**Expected**: Should accept port 443
**Actual**: Returns 500 Internal Server Error

**Environment**: charliehub production, 2026-02-08
EOF

# Send to: your DevOps/operations team

See also: - How to report bugs: Ask your team lead - Code changes are handled by humans with code review

Error 4: "docker: permission denied" trying to access docker socket¶

$ docker ps
docker: permission denied while trying to connect to the Docker daemon socket

Why this happened: - Only authorized users/services can access the Docker daemon - This prevents agents from modifying containers directly

What you should do instead: - ✅ Use the REST API for all domain/service management - ✅ Don't try to manually restart containers - ✅ Changes via API trigger proper deployment

Example: Redeploying a domain (the right way):

# Instead of: docker restart charliehub_domain_manager_v3
# ❌ This bypasses the controlled deployment process

# ✅ Use the API:
# First, make your change:
curl -X PUT http://172.19.0.5:8001/api/domains/24 \
  -H "X-API-Key: $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"domain": "...", "backend_host": "...", ...}'

# Then trigger deployment (which handles all container updates):
curl -X POST http://172.19.0.5:8001/api/deploy-all \
  -H "X-API-Key: $API_KEY"

See also: - AGENT_API_GUIDE.md - Section: "4. Deploy Changes to Production"

Error 5: "API returned 401 Unauthorized"¶

$ curl http://172.19.0.5:8001/api/domains
{"detail":"Invalid API key"}

Why this happened: - You either: - Didn't include an API key - Included an invalid/expired API key

What you should do instead:

# ✅ Make sure you have the API key
echo $API_KEY  # Should print something like: YOUR_API_KEY_HERE

# ✅ If empty, set it:
export DOMAIN_MANAGER_API_KEY="your_api_key_here"

# ✅ Make sure you're using it in the header:
curl -H "X-API-Key: $API_KEY" http://172.19.0.5:8001/api/domains

If you don't have an API key: - Ask your team lead or DevOps engineer for the key - Never hardcode it in your script - use environment variables or .env file - Never commit it to git - it will be exposed

See also: - AGENT_API_GUIDE.md - Section: "Getting Your API Key"

Error 6: "API returned 422 Unprocessable Entity"¶

$ curl -X POST http://172.19.0.5:8001/api/domains \
  -H "X-API-Key: $API_KEY" \
  -d '{"domain":"test"}'

{"detail":"Invalid request"}

Why this happened: - You forgot the Content-Type: application/json header - Or your JSON is malformed - Or you're missing required fields

What you should do instead:

# ✅ Always include Content-Type header:
curl -X POST http://172.19.0.5:8001/api/domains \
  -H "X-API-Key: $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "domain": "myservice.charliehub.net",
    "service_type": "http",
    "environment": "production",
    "backend_host": "10.44.1.100",
    "backend_port": 8080,
    "cors_enabled": false,
    "auth_required": true
  }'

# ✅ Validate your JSON before sending:
echo '{"domain":"test"}' | jq .  # jq will highlight syntax errors

Required fields for creating a domain: - domain - FQDN (e.g., "myservice.charliehub.net") - backend_host - IP address or hostname - backend_port - Port number (1-65535) - service_type - One of: api, http, spa, proxy, websocket, tcp-proxy, static, internal, manual - environment - Either: production or development

See also: - AGENT_API_GUIDE.md - Section: "1. Add a New Domain"

Error 7: "API returned 409 Conflict - Domain already exists"¶

$ curl -X POST http://172.19.0.5:8001/api/domains \
  -H "X-API-Key: $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"domain":"control.trevarn.com",...}'

{"detail":"Domain already exists"}

Why this happened: - A domain with that name already exists in the system - You can't create a duplicate

What you should do instead:

# ✅ Option 1: Use a different domain name
curl -X POST http://172.19.0.5:8001/api/domains \
  -H "X-API-Key: $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"domain":"newservice.charliehub.net",...}'

# ✅ Option 2: Update the existing domain instead (use PUT)
# First, find its ID:
curl -H "X-API-Key: $API_KEY" http://172.19.0.5:8001/api/domains | \
  jq '.[] | select(.domain=="control.trevarn.com") | .id'
# Output: 24

# Then update it:
curl -X PUT http://172.19.0.5:8001/api/domains/24 \
  -H "X-API-Key: $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"domain":"control.trevarn.com","backend_host":"10.44.1.200",...}'

See also: - AGENT_API_GUIDE.md - Section: "1. Add a New Domain" and "2. Update an Existing Domain"

Error 8: "API returned 404 Not Found"¶

$ curl -H "X-API-Key: $API_KEY" http://172.19.0.5:8001/api/domains/999
{"detail":"Domain not found"}

Why this happened: - The domain ID you're trying to access doesn't exist - You might have the wrong ID number

What you should do instead:

# ✅ List all domains to find the right ID:
curl -H "X-API-Key: $API_KEY" http://172.19.0.5:8001/api/domains | jq '.[] | {id, domain}'
# Output:
# {"id":1,"domain":"control.trevarn.com"}
# {"id":24,"domain":"api.trevarn.com"}
# ... etc

# ✅ Then use the correct ID:
curl -H "X-API-Key: $API_KEY" http://172.19.0.5:8001/api/domains/24 | jq .

See also: - AGENT_API_GUIDE.md - Section: "5. List All Domains"

Error 9: "Domain is unreachable" or "503 Service Unavailable"¶

$ curl https://myservice.charliehub.net/
curl: (7) Failed to connect to myservice.charliehub.net
# OR
HTTP/1.1 503 Service Unavailable

Why this happened: - The backend service isn't responding - Or the domain wasn't deployed yet - Or the backend host/port is wrong

What you should do instead:

# ✅ Step 1: Verify domain was created
curl -H "X-API-Key: $API_KEY" http://172.19.0.5:8001/api/domains | \
  jq '.[] | select(.domain=="myservice.charliehub.net")'

# ✅ Step 2: Verify it was deployed
curl -X POST http://172.19.0.5:8001/api/deploy-all \
  -H "X-API-Key: $API_KEY"

# ✅ Step 3: Check the domain details
curl -H "X-API-Key: $API_KEY" http://172.19.0.5:8001/api/domains/24 | jq .

# ✅ Step 4: Verify backend is actually running
ping 10.44.1.100  # Replace with your backend_host
curl http://10.44.1.100:8080/  # Replace with your backend_port

Common causes: - ❌ Backend service is down or crashed - ❌ Backend host is wrong (e.g., typo in IP address) - ❌ Backend port is wrong - ❌ Forgot to call /api/deploy-all after creating domain

See also: - AGENT_API_GUIDE.md - Section: "4. Deploy Changes to Production" and "Checking If Your Change Worked"

Errors Not Listed Here?¶

If you're getting an error not listed above:

Read the full error message - It usually has useful information
Check AGENT_API_GUIDE.md - Look for "Troubleshooting" section
Check OPERATOR_HOWTO.md - If it's an infrastructure issue
Ask your team - There might be a known issue or recent change

Quick Reference: What to Do Instead¶

You wanted to...	❌ Don't do this	✅ Do this instead
Add a new domain	Edit YAML files	POST /api/domains
Change a domain's backend	Modify config files	PUT /api/domains/{id}
Make a domain public	Edit traefik config	PUT /api/domains/{id} with auth_required=false
Query domains	Connect to database with psql	GET /api/domains
View one domain's config	SELECT from database	GET /api/domains/{id}
Delete a domain	DELETE from database	DELETE /api/domains/{id}
Apply changes	docker restart	POST /api/deploy-all
Check if changes work	Manually test	curl the domain, check Traefik

Summary¶

The pattern is simple:

❌ You try to modify files/database directly
📛 You get a "Permission denied" error
✅ You use the REST API instead
🎉 Everything works and is audited

Remember: - Read-only files aren't obstacles - they're guardrails - Blocked database isn't restrictive - it's protective - API-only isn't limitation - it's the safe, validated way

Getting Help¶

API Documentation: AGENT_API_GUIDE.md - Complete with examples
Python Examples: AGENT_API_GUIDE.md - Ready-to-use DomainManagerClient class
Error Help: You're reading it!
Team Support: Ask your DevOps engineer or team lead

Last updated: 2026-02-08 If this guide isn't helping, you may have found a bug in the error messages. Please report it.