Advanced guide / Chapter 7 of 7
Operations, reliability and troubleshooting.
If limits cannot be checked
Closed is the safe default, and with it nothing is ever sent unchecked. Open keeps your agents running through an outage, and each call that goes out that way is logged as "sent unchecked". Free plans always fail closed. A caller can always ask for closed, but asking for open only works if the team allows it.
Every error code
| Status | Code | Meaning | What to do |
|---|---|---|---|
| 401 | missing_api_key, invalid_api_key, expired_api_key, key_revoked | The ACPpotamia key is missing, wrong or revoked | Check the key; create a new one |
| 401 | missing_upstream_key | No provider key sent | Send X-Upstream-Key (or use a test key) |
| 402 | budget_exhausted, token_limit_reached | The team's monthly budget or tokens are used up | Raise the limit or wait for the 1st |
| 402 | key_ / tenant_ / session_budget_exhausted | That key, customer or task reached its own budget | Raise it; others are unaffected |
| 403 | model_not_allowed | Outside the model policy | Use an allowed model |
| 403 | team_suspended | The team is suspended | Write to acp@examopotamia.com |
| 429 | rate_limited, key_ / tenant_ / session_rate_limited | Too many requests a minute | Wait for Retry-After |
| 429 | loop_detected, session_killed, team_killed | Stopped by loop detection or the kill switch | Fix the cause, then revive; retrying does not help |
| 502 | upstream_unreachable | The provider could not be reached | Retry; nothing was charged |
| 503 | control_plane_unavailable | Limits could not be checked and fail mode is closed | Retry after Retry-After |
Monitoring
- Service status checks every component live. Point your uptime monitor at
/status.json, which returns 503 when anything is down. - Every response carries
X-ControlPlane-Overhead-MsandX-ControlPlane-Request-Id. Log both. You'll want the request id the day something goes wrong. - In the console, Overview shows spend, blocked calls and stopped agents, and Activity lists every call.
Troubleshooting checklist
| Symptom | Check |
|---|---|
| Every call gets 401 | Is the key the acp_ key (not the provider key)? Was it revoked? For Anthropic, is it in X-ACP-Key? |
| Calls work in tests but not in production | Test keys never call providers. Production needs a real acp_ key and X-Upstream-Key. |
| An agent stopped on its own | Activity → filter by agent: look for loop_detected or a budget code. Revive on the Agents page. |
| One customer blocked, others fine | That tenant reached its budget: Tenants page or GET /v1/controlplane/tenants/{id}. |
| Answers are shorter than expected | A budget is nearly used up, so max_tokens was lowered. See X-ControlPlane-Budget-Remaining-USD. |
| Changes do not seem to apply | Limits, policies, kill switches and revocations apply to the next call. Plan upgrades apply within a minute. |
Still stuck? Email acp@examopotamia.com with the X-ControlPlane-Request-Id.