We set a Slack alert at 80% of our monthly LLM budget. The alert fired at 2 AM on a Saturday. Nobody saw it until Monday morning, by which point we were 340% over budget from a retry loop that had been running the entire weekend. The alert did exactly what it was built to do — notify someone. It just couldn't do the other job we actually needed, which was stop the spend before it happened.
1. Alerts notify; they don't enforce. The gap between an alert firing and a human acting is where runaway spend actually happens.
2. A hard block at the proxy layer stops the spend before it happens, but risks breaking production if it fires on legitimate traffic.
3. The pattern that held up: alert at 75%, hard block at 100%+, both enforced where the request actually passes through — not bolted on after the fact.
Why Alert-Only Failed for Us
The weekend incident: A retry loop on a flaky downstream dependency kept resubmitting the same request pattern for roughly 30 hours. Our 80% budget alert fired Saturday at 2 AM. Nobody was watching Slack. By Monday morning, spend was 340% over the monthly budget — the alert had done its job perfectly and prevented nothing.
The structural problem isn't that our team was careless — it's that an alert-only system has a single point of failure: a human, awake, watching the right channel, at the exact moment it matters. Weekends, overnight windows, and busy sprints are precisely when that assumption breaks, and they're also disproportionately when runaway loops and bugs actually occur, because they're the times nobody's actively watching the system.
Alert vs. Hard Block, Honestly
Alert-Only
Pro: Never breaks legitimate traffic. Safe default for low-stakes or still-calibrating budgets.
Con: Only as good as the person watching it, at the moment it fires. Zero protection outside business hours unless someone's specifically on call for it.
Hard Block
Pro: Actually stops the spend — no human required, works at 2 AM on a Saturday exactly the same as at 2 PM on a Tuesday.
Con: If it triggers on a legitimate spike (a real traffic surge, a launch), it can break production for real users, not just waste.
Neither mode is strictly better — they fail differently. Alert-only fails by doing nothing when nobody's watching. Hard block fails, rarely but visibly, by stopping something legitimate. The pattern that held up in practice uses both, at different thresholds.
Want hard limits, not just alerts?
Set an alert threshold and a hard-block threshold, enforced where the request actually passes through — the proxy, not a dashboard nobody's watching.
Set Your First Budget Limit — FreeHard-block or alert. Preto is one of the few proxies that enforces hard budget limits, not just alerts.
The Two-Threshold Pattern
75% — alert only. Gives a human time to look at what's happening and decide whether it's expected (a real traffic spike, a planned launch) or not, before anything gets blocked. This threshold exists to catch problems while there's still room to react calmly.
100%+ — hard block. By this point, either the alert was ignored, or spend accelerated too fast for a human to catch it in time — either way, the risk of continuing to spend outweighs the risk of blocking a legitimate request. This is the floor that actually held during our weekend incident: with a hard block in place, the retry loop would have stopped at the configured ceiling instead of running for 30 unmonitored hours.
Why Enforcement Location Matters
OpenAI's own project-level budgets exist, but there are documented community reports of those limits not being consistently respected, with real overages exceeding the configured cap. Relying on provider-native budget controls alone means trusting an enforcement mechanism you don't control and can't verify in real time. A limit enforced at the proxy layer — before the request ever reaches the provider — is a check you own, can test, and can see the exact behavior of when it fires.
This connects directly to token-based rate limiting, which caps sustained per-identity usage — budget enforcement caps the workspace total regardless of which identity is responsible. The two are complementary layers of the same underlying discipline: don't rely on someone noticing a problem to stop it.
Frequently Asked Questions
Why don't Slack alerts prevent LLM cost overruns?
What's the risk of a hard budget block in production?
Does OpenAI have native budget enforcement?
Should every team use hard blocks, or is alert-only sometimes right?
Set a limit that actually holds at 2 AM.
Alert at 75%, hard block at 100% — configured once at the proxy layer, enforced whether or not anyone's watching Slack.
Set Your First Budget Limit — FreeFree forever up to 10K requests. No credit card required.
