How we streamed AI CareerBot responses and protected GigaChat from overload

Thousands of concurrent users is the point where the nice effect of “the response is typed out token by token” stops being a frontend task and becomes a task of protecting an external AI API.

In AI CareerBot, we implemented streaming responses: the model generates an answer, and the user sees it not all at once at the end, but gradually, like in ChatGPT. For transport, we used SSE/WebSocket. Under normal load, everything looked as expected: the request went out, tokens came back, and the interface rendered the text incrementally.

Then the product went viral.

Why token streaming was not the main problem

Streaming itself is fairly straightforward technically: keep the connection open, pass through response chunks, and on the client side append tokens to the current text. The important details are clean connection shutdown, handling disconnects, the “generation in progress” state, and the final signal so the UI knows the answer is complete.

But the weak point was not the typing animation. The weak point was the GigaChat API.

If every user starts generation directly, then when traffic spikes suddenly, the external API gets a burst of requests. For the user, this looks like “the bot is thinking”; for us, it means the risk of errors, timeouts, and a chain reaction of retries.

One practical detail: at first, we wanted to treat streaming mostly as a UX problem. We discussed which transport was more convenient — SSE or WebSocket — and how to deliver tokens more elegantly. But after the first signs of overload, the discussion quickly shifted toward the queue: who gets to go to GigaChat now, who waits, and to whom we show degradation honestly.

What we put between the user and GigaChat

The resulting setup had three layers of protection:

• a queue for AI requests so we don’t release the entire incoming stream outward at once;
• rate limiting to cap request frequency and avoid amplifying overload;
• graceful degradation so that at peak load the system doesn’t fail silently, but responds predictably.

The queue here is not there for architectural aesthetics. It turns a sharp spike into a controlled flow. The user request enters the system, waits its turn, and then generation starts and the streaming response opens.

Rate limiting cuts off cases where one segment of traffic starts consuming the shared limit. This is especially visible during public launches: some users refresh the page, repeat the request, or open several tabs. If you don’t limit anything, those repeated actions make the problem worse.

Graceful degradation is the most unpleasant, but also the most important layer. You need to decide ahead of time what happens when the queue is already too long or the external API is behaving unreliably. Don’t promise the user an immediate response if the system can’t provide one right now. It’s better to show a waiting state or fail gracefully than to keep an endless “the bot is typing...” screen.

Where the line between SSE and WebSocket actually is

For this task, both options make sense.

SSE fits naturally into the “server writes, client reads” model. For AI streaming responses, that’s the natural fit: the browser keeps the connection open, and the server sends token events. Less bidirectional logic, easier to reason about the stream.

WebSocket is convenient if, in addition to tokens, you need an active two-way channel: cancel generation, send additional commands, manage more complex session state. But that also increases the number of places where you need to track connections, reconnections, and resource cleanup.

The important conclusion for us was not which transport to pick forever. More important is that the transport itself does not protect the AI API. You can stream tokens perfectly and still take down the external service with an uncontrolled number of generations.

What we’d ask other teams about

In problems like this, there are a few decisions you have to make almost blind, until real load shows up:

• what queue size is considered normal;
• when to start degrading;
• what to show the user before generation starts;
• how to avoid provoking repeated requests;
• where to keep the streaming session state if the connection drops.

In AI CareerBot, we covered the basic scenario with a queue, rate limiting, and graceful degradation around the GigaChat API. The typing effect stayed, but generations stopped going out uncontrollably.

If you’ve built streaming AI responses under high load, how did you choose the thresholds for the queue and rate limiting: fixed values, dynamic tuning based on external API errors, or a separate backpressure model?