Why Request-Count Rate Limiting Is Costing You Money (And What to Use Instead)

"100 requests per minute per user" sounds like a reasonable rate limit. It caps abuse, it's simple to implement, and every API gateway supports it out of the box. It's also nearly useless as a cost control, because it treats a 10-token request and a 4,000-token request as identical — and on an LLM API, token count is what determines what you actually pay.

The Gap, Concretely

Two users, both under a 100-requests-per-minute limit, both fully compliant:

Both users are equally "compliant" under a request-count limit. User B has generated 40x the token volume — and 40x the cost exposure — while never once triggering the rate limit that was supposed to control usage. This isn't a hypothetical edge case; it's the normal outcome any time request size varies across your user base, which is nearly always.

Why This Matters More Than It Seems

Request-count limiting was inherited from traditional API rate limiting, where the cost driver really is request volume — a database query or a REST endpoint call costs roughly the same regardless of payload size, so counting requests is a reasonable proxy for load. LLM APIs break that assumption completely: cost scales with tokens, not requests, and token count per request can vary by two or three orders of magnitude depending on what a user is doing. A rate limit built for the first model of cost doesn't protect against the second.

The practical failure mode: a single user or a single misbehaving integration sending large-payload requests can burn through a disproportionate share of budget while staying technically within every configured limit — exactly the kind of quiet cost exposure that shows up as a surprise line item weeks later, not as an alert.

Token-Based Rate Limiting: The Architecture

The pattern is a small variation on standard rate limiting — track a running total instead of a running count, and measure in tokens instead of requests:

import redis
import time

r = redis.Redis()

def check_token_budget(user_id, estimated_tokens, budget=500_000, window_seconds=3600):
    key = f"tokenlimit:{user_id}:{int(time.time() // window_seconds)}"
    current = int(r.get(key) or 0)

    if current + estimated_tokens > budget:
        return False  # reject or queue

    r.incrby(key, estimated_tokens)
    r.expire(key, window_seconds)
    return True

def record_actual_usage(user_id, actual_tokens, estimated_tokens, window_seconds=3600):
    # Correct the counter after the real token count is known
    key = f"tokenlimit:{user_id}:{int(time.time() // window_seconds)}"
    delta = actual_tokens - estimated_tokens
    if delta != 0:
        r.incrby(key, delta)

Two things matter in this pattern beyond the obvious counter swap:

Estimate before, measure after. You need a fast token estimate before the call to make the rate-limit decision without adding latency — a lightweight tokenizer on the prompt text is usually enough. Output length is unpredictable until generation completes, so the running counter needs a correction step once the actual usage is known, or the budget will drift over many requests.

Window granularity should match your actual risk. An hourly window catches sustained overuse without punishing normal burstiness the way a per-minute window would. Pick the window based on how quickly you need to react to a runaway pattern versus how much normal variance you want to tolerate.

What This Doesn't Replace

Token-based rate limiting caps sustained per-identity usage. It doesn't replace budget enforcement at the workspace or organization level — a hard cutoff when total monthly spend crosses a threshold, regardless of which user is responsible. The two are complementary: token-based limits prevent any single user or integration from consuming a disproportionate share of the budget, while workspace-level budget enforcement caps the total regardless of how usage is distributed. Teams that implement only one tend to be surprised by the failure mode the other one would have caught.

This kind of enforcement lives at the proxy layer — see the full architecture behind LLM proxies for where rate limiting and budget checks sit relative to caching, routing, and logging. And if you haven't run a structured pass over your cost data recently, a weekly cost review is where a rate-limiting gap like this usually gets caught first.

Gaurav Dagade

Founder of Preto.ai. 11 years engineering leadership. Previously Engineering Manager at Bynry. Building the cost intelligence layer for AI infrastructure.

LinkedIn · Twitter