A/B Testing Prompts in AI CareerBot Without API Surprises

In the AI CareerBot chat, a user sends a question about a career transition and expects not a polished paragraph, but an answer they can act on: update a resume, rephrase experience, or understand the next step.

For the user, it looks simple: ask a question, get advice. For us, the important part at that moment is different: which system prompt was used, how useful the answer was, whether the request cost got out of hand, and whether we need to replace the response with a safe fallback.

In AI CareerBot, we built infrastructure for A/B testing system prompts in production. Not in a separate sandbox where every question is neat, but on real traffic, where users ask in different ways and often repeat the same topics.

What exactly we tested

The system prompt in a career bot has a strong effect on model behavior. One version gives more structure. Another sets a cautious tone. A third does a better job of keeping task context and does not turn the answer into a generic consultation.

We needed to compare these options not by gut feel in the dev chat, but on live traffic. So we built routing: different users or sessions were sent to different versions of the system prompt, and the responses were then linked to the variant that produced them.

The setup looked like this:

• there are several versions of the system prompt;
• a user request goes to one of the variants;
• the response is saved together with the variant ID;
• quality is measured through usefulness metrics;
• bad or incorrect responses are intercepted by moderation and fallback logic.

The main debate was not about “which wording sounds better.” The debate was about where usefulness ends and subjective preference begins. Career answers do not have a single gold-standard JSON to compare against. So the metrics had to help us make decisions, but not pretend to be absolute truth.

A viral launch quickly showed the cost of every experiment

After launch, traffic started to grow sharply. For the product, that is a good signal; for an LLM integration, it is the moment when the API cost table becomes much more interesting than you would like.

A/B testing amplifies this effect. If you do not control the flow, it is easy to start paying for experiments that no longer provide new information. Especially when users ask similar questions at scale: the model faithfully generates an answer every time, and the API counter grows just as faithfully.

We added three layers of protection.

The first was budgeting. Experiments should not exist separately from financial constraints. Tests need boundaries: how many requests a variant can handle, when to stop, and what to do when the limit is reached.

The second was limits. They protect the system from a situation where one scenario starts consuming too much. A limit is a boring thing until it saves your evening.

The third was caching common questions. CareerBot has recurring patterns: users ask about resumes, interviews, role changes, and how to describe experience. Where the question and context fit a typical scenario, there is no reason to burn API calls again each time. Caching reduces cost and makes behavior more stable for frequently seen requests.

Moderating the output, not just the input

People usually start with filtering user input. That matters, but for a career assistant, the model output is just as important. An answer can be confident, well written, and still incorrect: for example, too categorical, not suited to the task, or failing our quality rules.

So after generation, we added content moderation on the output. The response is checked before it reaches the user. If it fails the check, fallback kicks in.

Fallback here is not a dummy response for the sake of having one. Its job is to avoid leaving the user with bad advice. Depending on the situation, the system can return a safer answer, ask for more context, or avoid showing a response the model generated poorly.

This is especially important in A/B tests. A new system prompt can improve one type of response and suddenly make another worse. Without output moderation, that variant would reach users in a raw state.

Why prompt A/B tests need product scaffolding

It is easy to change a prompt in a file. It is hard to make that change measurable, keep it within cost limits, and roll it back safely.

For AI CareerBot, the A/B testing infrastructure became a separate layer around LLM calls. It includes variant assignment, usefulness metric collection, budgeting, limits, caching for common questions, output moderation, and fallback for incorrect responses.

This does not replace manual prompt review. That is still needed, especially for new scenarios. But production quickly surfaces questions that did not make it into the test set. So the next practical step is to look at which request groups most often end up in fallback and which prompt variants produce less bad output there.