Streaming SSE Proxying for LLM APIs: The Hard Parts

OpenAI streaming looks simple from the outside. Set stream: true, iterate the response, pipe it to the client. One afternoon of work.

Then you ship it. A client disconnects mid-generation and you eat 2,000 tokens nobody received. A slow mobile client causes your proxy's memory to climb. An OpenAI rate limit hits after you've already sent a 200. Here's what actually happens when you proxy SSE at scale — and the patterns that fix each failure mode.

How LLM SSE Actually Works

OpenAI's streaming response is plain HTTP with Content-Type: text/event-stream. The body is a sequence of newline-delimited frames:

data: {"id":"chatcmpl-...","choices":[{"delta":{"content":"Hello"},...}]}

data: {"id":"chatcmpl-...","choices":[{"delta":{"content":" world"},...}]}

data: [DONE]

Each event is data: {JSON}\n\n — a line starting with data: , then a blank line as a separator. The stream ends with the literal data: [DONE]\n\n sentinel.

A proxy needs to read this upstream, optionally inspect or annotate the frames, and write them downstream to the client without adding latency or modifying the framing. That's the theory. Here's what breaks in practice.

Failure 1: The Chunk Boundary Problem

If you only need to pass the stream through without inspecting it, io.Copy is fine — TCP handles reassembly. The problem comes when you need to inspect each SSE frame: to extract token counts, inject annotations, or filter events.

The wrong approach: read into a fixed buffer and parse what you get. A single Read() call may return half an event, two events, or anything in between.

The right approach: use Go's bufio.Scanner with a custom split function that understands SSE framing:

func proxySSE(upstream io.ReadCloser, w http.ResponseWriter, onEvent func([]byte)) {
  scanner := bufio.NewScanner(upstream)
  scanner.Split(scanSSEEvents) // custom split — see below

  flusher := w.(http.Flusher)

  for scanner.Scan() {
    line := scanner.Bytes()
    w.Write(line)
    w.Write([]byte("\n\n"))
    flusher.Flush() // push each event to the client immediately

    if onEvent != nil && bytes.HasPrefix(line, []byte("data: ")) {
      onEvent(line[6:]) // strip "data: " prefix before passing to handler
    }
  }
}

// scanSSEEvents splits on double-newline (SSE event boundary)
func scanSSEEvents(data []byte, atEOF bool) (advance int, token []byte, err error) {
  if atEOF && len(data) == 0 {
    return 0, nil, nil
  }
  if i := bytes.Index(data, []byte("\n\n")); i >= 0 {
    return i + 2, bytes.TrimRight(data[:i], "\n"), nil
  }
  if atEOF {
    return len(data), bytes.TrimRight(data, "\n"), nil
  }
  return 0, nil, nil // request more data
}

The Flusher call on every event is critical. Without it, Go's http.ResponseWriter buffers writes and the client gets chunks in batches — defeating the purpose of streaming.

Failure 2: Token Leaks on Client Disconnect

This is the most expensive failure mode. When a client closes the connection mid-stream — browser tab closed, mobile app backgrounded, network timeout — a naive proxy keeps the upstream request to OpenAI running. OpenAI finishes generating the full completion. You're billed for every token.

At 1,000 req/s with a 5% disconnect rate and 500 average output tokens: that's 25,000 wasted tokens per second — hundreds of dollars per day at GPT-4.1-nano pricing, more at GPT-4.1.

The fix is Go context propagation. Pass the client's request context to the upstream HTTP call. When the client disconnects, Go's net/http server cancels the request context, which cascades to the upstream call:

func (p *Proxy) handleStream(w http.ResponseWriter, r *http.Request) {
  // r.Context() is cancelled automatically when the client disconnects
  ctx := r.Context()

  upstreamReq, err := http.NewRequestWithContext(ctx, "POST",
    "https://api.openai.com/v1/chat/completions",
    r.Body,
  )
  if err != nil {
    http.Error(w, "upstream error", 502)
    return
  }
  upstreamReq.Header = r.Header.Clone()

  resp, err := p.client.Do(upstreamReq)
  if err != nil {
    if errors.Is(err, context.Canceled) {
      // client disconnected — upstream cancelled, no token leak
      return
    }
    http.Error(w, "upstream error", 502)
    return
  }
  defer resp.Body.Close()

  // ... proxy the stream
}

The key: http.NewRequestWithContext(ctx, ...) instead of http.NewRequest. When the client context is cancelled, Go's HTTP client aborts the upstream connection. OpenAI stops generating. You stop paying.

Failure 3: Backpressure and Unbounded Buffering

OpenAI streams tokens at roughly 50–100 tokens per second for GPT-4.1. A client on a fast connection reads faster than that — no problem. A client on a slow connection, or one that's processing each chunk before reading the next, can fall behind.

If your proxy writes to the client's ResponseWriter synchronously, Go's HTTP server buffers unread data in kernel socket buffers. Those buffers are bounded (typically 64KB–256KB per connection). When they fill, the Write() call blocks — which blocks your upstream reader — which causes the upstream TCP window to fill — which causes OpenAI's server to pause sending. The stream stalls.

The dangerous alternative is an unbounded in-memory buffer between the upstream reader and the downstream writer. Under load with many slow clients, this causes OOM.

Our approach: a bounded channel between reader and writer goroutines, with a timeout on the write side:

func (p *Proxy) streamWithBackpressure(ctx context.Context,
  upstream io.ReadCloser, w http.ResponseWriter) {

  eventCh := make(chan []byte, 64) // bounded: 64 events max in-flight
  flusher  := w.(http.Flusher)

  // Reader goroutine: upstream → channel
  go func() {
    defer close(eventCh)
    scanner := bufio.NewScanner(upstream)
    scanner.Split(scanSSEEvents)
    for scanner.Scan() {
      select {
      case eventCh <- append([]byte{}, scanner.Bytes()...):
      case <-ctx.Done():
        return
      case <-time.After(5 * time.Second):
        // client too slow — abort
        return
      }
    }
  }()

  // Writer goroutine: channel → client
  for event := range eventCh {
    w.Write(event)
    w.Write([]byte("\n\n"))
    flusher.Flush()
  }
}

The 5-second timeout on the channel send is the backpressure relief valve. If the downstream writer hasn't consumed the last event within 5 seconds, the reader goroutine exits — closing the channel — which causes the writer loop to exit cleanly. The upstream connection is cancelled via context.

Failure 4: Mid-Stream Errors After HTTP 200

HTTP status codes are sent in the response header — before the body. Once you've written 200 OK and started streaming, you cannot send a 429 Too Many Requests or 503 if something goes wrong upstream. The client already accepted the response as successful.

This happens in real production traffic. OpenAI sends a 200 header and begins streaming, then hits an internal rate limit or context overflow mid-generation. The stream ends abruptly. Without handling, the client sees a truncated response with no indication of what happened — and typically retries, paying for partial tokens twice.

The SSE spec has no built-in error channel. The convention (used by OpenAI themselves in their error handling) is to send a final event with an error payload before closing:

func writeSSEError(w http.ResponseWriter, code string, message string) {
  flusher, ok := w.(http.Flusher)
  if !ok {
    return
  }
  errPayload := fmt.Sprintf(
    `data: {"error":{"code":"%s","message":"%s"}}`+"\n\n",
    code, message,
  )
  w.Write([]byte(errPayload))
  flusher.Flush()
}

// In your stream handler, after the scanner loop:
if err := scanner.Err(); err != nil {
  if !errors.Is(err, context.Canceled) {
    writeSSEError(w, "stream_error", "upstream stream interrupted")
  }
}

Clients handling streaming responses should check every data: payload for an error key, not just the HTTP status. This is especially important for retry logic — a truncated stream with an in-band error should retry differently from a clean stream that completed normally.

Cost Tracking Without Blocking the Stream

One more wrinkle: you can't calculate output token cost until the stream completes. Without the final token count, you'd need to build your own token counter — and you'd still be estimating mid-stream.

The solution: pass stream_options: {"include_usage": true} in your OpenAI request body. OpenAI sends a final chunk before [DONE] with the exact usage stats:

data: {"id":"...","choices":[],"usage":{"prompt_tokens":142,"completion_tokens":387,"total_tokens":529}}

data: [DONE]

In your onEvent handler, parse each chunk and look for the usage field. When you find it, record the cost asynchronously (fire the log entry into your channel) and pass the chunk through to the client unmodified. Zero latency added — the usage chunk arrives after all content tokens have already streamed.

At Preto, we capture this usage chunk and attribute cost to the calling feature, team, and model — giving you per-feature cost breakdown that the OpenAI dashboard doesn't provide. See how we store and aggregate those logs at scale.

Gaurav Dagade

Founder of Preto.ai. 11 years engineering leadership. Previously Engineering Manager at Bynry. Building the cost intelligence layer for AI infrastructure.

LinkedIn · Twitter