Streaming Through MonoRouter: How We Keep Latency Low

When Claude generates a 2,000-token response, you don't want to wait for all 2,000 tokens before seeing anything. Streaming sends tokens as they're generated, and MonoRouter preserves this end-to-end.

Why Streaming Matters for Code

In a chat UI, streaming is a nice-to-have — it makes the interface feel responsive. In a coding workflow, it's more than that. Claude Code shows you what it's thinking and doing in real time. Without streaming, you'd stare at a blank terminal for 30 seconds waiting for a complex response.

Agent frameworks like LangChain and LangGraph also depend on streaming for intermediate steps. A multi-step agent that takes 2 minutes to complete is unusable without streaming — you have no idea what it's doing or whether it's stuck.

Streaming with the Python SDK

The Anthropic SDK's streaming interface works identically through MonoRouter. Here's a basic example:

import anthropic

client = anthropic.Anthropic(
    api_key="mr_live_abc123",
    base_url="https://api.monorouter.dev/v1",
)

with client.messages.stream(
    model="claude-sonnet-4-20250514",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Explain async/await in Python"}],
) as stream:
    for text in stream.text_stream:
        print(text, end="", flush=True)

Every token arrives the moment Claude generates it. MonoRouter adds no buffering — it forwards each SSE event as it comes off the upstream connection.

Streaming in TypeScript

The TypeScript SDK uses an async iterator pattern:

import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic({
  apiKey: "mr_live_abc123",
  baseURL: "https://api.monorouter.dev/v1",
});

const stream = client.messages.stream({
  model: "claude-sonnet-4-20250514",
  max_tokens: 1024,
  messages: [{ role: "user", content: "Write a React hook for debounced search" }],
});

for await (const event of stream) {
  if (
    event.type === "content_block_delta" &&
    event.delta.type === "text_delta"
  ) {
    process.stdout.write(event.delta.text);
  }
}

const finalMessage = await stream.finalMessage();
console.log("\n\nTokens used:", finalMessage.usage);

The event types are identical to what you'd get from the Anthropic API directly — message_start, content_block_start, content_block_delta, content_block_stop, and message_stop. MonoRouter doesn't modify, reorder, or filter any events.

The Proxy Challenge

Proxying streaming responses is harder than proxying regular HTTP. A normal request-response cycle is simple: receive request, forward it, get response, send it back. Streaming adds a persistent connection that needs to stay open for the duration of the response.

MonoRouter handles this with Server-Sent Events (SSE), the same protocol the Anthropic API uses. When your SDK sends a streaming request, MonoRouter opens a streaming connection to Claude and pipes events through as they arrive. Each token, each content block, each tool use — forwarded with minimal added latency.

Under the hood, a raw SSE frame from MonoRouter looks like this:

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"Here"}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"'s"}}

Each data: line is forwarded within microseconds of arrival. No batching, no rewriting.

Latency Overhead

We measured the added latency of routing through MonoRouter vs. hitting the API directly. For streaming responses, the overhead is consistently under 50ms for the first token and negligible for subsequent tokens. The bottleneck is always Claude's generation speed, not the proxy.

Metric	Direct API	MonoRouter	Overhead
Time to first token	~320ms	~360ms	+40ms
Inter-token latency	~12ms	~12ms	<1ms
Total (1K tokens)	~12.3s	~12.35s	negligible

Error Handling in Streams

Streams can fail mid-response. The browser session might drop, the connection might timeout, or Claude might hit an internal error. MonoRouter detects these failures and sends a proper SSE error event so your SDK can handle it cleanly instead of hanging on a broken connection.

try:
    with client.messages.stream(
        model="claude-sonnet-4-20250514",
        max_tokens=4096,
        messages=messages,
    ) as stream:
        for text in stream.text_stream:
            print(text, end="", flush=True)
except anthropic.APIStatusError as e:
    if e.status_code == 529:
        print("All sessions busy — try again in a moment")
    else:
        print(f"Stream error: {e.message}")
except anthropic.APIConnectionError:
    print("Connection dropped mid-stream")

For non-streaming requests, MonoRouter can retry on a different session. For streaming requests that fail mid-response, there's no clean way to retry — the partial response is already sent. MonoRouter surfaces the error and lets your application decide what to do.

If you're building an agent loop that needs resilience, wrap your streaming calls in retry logic and handle partial outputs. The SDK's built-in retry works for connection failures before the stream starts, but mid-stream failures need application-level handling.