Author: vitorcherulli

Enterprise teams build AI chat interfaces, showcase smooth demos, and watch them break in production. The root cause often lies not in the model but in the frontend architecture. Connecting raw token streams directly to React components causes slow renders, lag, and broken state. This article explores why this happens and presents a robust solution.

The Core Problem: Streams vs. UI Rendering

AI chat operates differently from standard interfaces. Responses arrive as continuous event streams while users cancel requests, switch chats, or trigger tools simultaneously. The common approach of reading a POST response stream inside a React component—whether manually or with hooks like Vercel’s useChat—creates three critical couplings:

• Schedule coupling: The component controls when the UI updates, but it shouldn’t.
• Concurrency coupling: The component manages which stream may write, but this logic belongs elsewhere.
• Protocol coupling: The component interprets raw bytes, but this is not its responsibility.

How These Couplings Fail

These couplings manifest in three main failure modes:

1. Interaction Latency That Scales with Session Length

As the transcript grows, each update takes more work to render. Once a task exceeds 50 ms, it blocks user input. When this delay pushes the interface beyond the 200 ms threshold for a good Interaction to Next Paint (INP), typing feels sluggish. Because the problem appears gradually, teams often blame the model instead of the architecture.

2. Concurrency Races on Stop, Regenerate, and Switch

When the current response lives in effect closures and isLoading booleans, there’s no central mechanism to decide which stream can update state. Switching conversations mid-stream can cause late events to update the wrong conversation. Open WebUI documents this: “Chat B’s content disappears—replaced by a loading state meant for Chat A.”

3. Protocol Flattening

AI providers send typed event streams, not plain text. For example, Anthropic’s protocol interleaves text, thinking, and tool-call deltas, with tool arguments arriving as partial JSON. Flattening these events into string concatenation loses meaning, leading to incomplete tool arguments, missed mid-stream errors, or truncated output stored as final content—corrupting the context for the next turn.

The Correction: Ingest, Log, Project

The solution introduces a boundary with three components, none of which import React:

• Stream Ingestor: Handles the connection and provider events, turns them into typed application events, validates order, and assigns a run ID. Starting, stopping, or switching conversations creates a new ID, and late events are dropped. AbortController lives here, separating cancellation from the component lifecycle.
• Conversation Log: The single source of truth. It records events in order and marks a response complete only when the provider sends its final message event (e.g., message_stop). A dropped connection cannot look like a completed answer. The log also serves as a test case and debugging trace.
• Projection Scheduler: Converts the log into the state React needs to display. Using useSyncExternalStore, it updates the UI on a controlled schedule (e.g., frame budget or ~50 ms interval) instead of on every token. Per-message views remain referentially stable, so memoized components skip unchanged work.

provider SSE / fetch stream
        |
        v
Typed protocol events
        |
        v
+-----------------------------+
| Conversation runtime (no React) |
| - Stream Ingestor: parse, validate, epoch-guard |
| - Conversation Log: append-only, replayable |
| - Projection Scheduler: frame-budget flush |
+-----------------------------+
        |
        v
useSyncExternalStore commit
        |
        v
React component tree

Trade-offs and When to Use This Approach

The UI trails the stream by one flush interval, and the architecture requires more code than a simple hook. For short prototype sessions without tool calls, it’s probably unnecessary. The event log also needs periodic compaction. However, the payoff is that interaction cost stops growing with conversation length.

Real-World Numbers

In a benchmark comparing three versions of the same React chat harness, the naive architecture showed a p95 heartbeat-task lateness of 20.9 ms at 12 messages—acceptable at demo scale. But at 48 messages, each UI update took three times longer, frame rate dropped, and user interactions waited 61 ms before handlers ran. The naive version attempted 100 UI updates per second but managed only 38, saturating the main thread. The projection variant committed 5× less often and spent 12× less main-thread time, at identical perceived text speed.

Conclusion

Treat an LLM conversation as distributed-systems data: an ordered, typed, cancellable event log. The frontend should control when the UI updates. Review streaming state architecture like an API contract: Who controls the update schedule? Who decides which stream can write? Which events mark a response as complete? Measure interactivity at realistic conversation lengths, not demo lengths. A chat interface that feels responsive at message ten should not degrade by message fifty.

References:

• WHATWG. HTML Living Standard, §9.2 Server-sent events.
• React Team. React v18.0 — automatic batching.
• Google web.dev. Interaction to Next Paint (INP); Optimize long tasks.
• Vercel. AI SDK UI: Chatbot—per-chunk renders and the throttle option.
• Anthropic. Streaming Messages—event types and ordering.
• Brysbaert, M. (2019). How many words do we read per minute? Journal of Memory and Language, 109.
• Artificial Analysis. LLM output speed comparisons.
• Open WebUI. Discussion #21462: chat switching causes race conditions.