Building a Production AI Agent in Spring Boot: Streaming Responses with SSE (Part 3)

"Is it stuck?"

The same tester who caught the memory bug in Part 2 found the next problem, and this one was harder to explain. She asked the e-commerce agent: "Can I return the blue jacket from my last order, and what does the return policy say?"

That question needs two tool calls. The order lookup tool runs, the policy tool runs, then the model writes a long answer with the jacket details, the order number, and the policy steps. On my setup that whole round trip took about ten seconds. For nine of those seconds, the screen showed nothing.

The agent was not stuck. The model was working the whole time. But a silent nine-second wait reads as broken, no matter how good the final answer is. I told her to wait, the text appeared, and she moved on. Every user after her did not get that instruction.

That is the gap this part of the series closes. Same agent, same tools, same memory from Parts 1 and 2. The only change is how the response travels: instead of one string arriving when the answer is complete, tokens reach the browser as the model produces them.

I am a Senior Software Engineer II at BS23 in Dhaka, and I have been building production AI agents with Spring Boot and Spring AI for over a year. Everything below comes from the agent I actually run, not a toy demo.

Streaming Changes Perception, Not Speed

Here is the honest part first: streaming does not make the model faster. The total time from request to complete answer stays roughly the same. What changes is what the user experiences.

The synchronous path from Parts 1 and 2 ends with .call(), which blocks until the entire response exists:

String answer = chatClient.prompt()
        .user(userMessage)
        .call()
        .content();

One string, one long wait, then everything at once. The streaming path returns a Reactor Flux immediately, and each element of the flux is a piece of text the model generated:

Flux<String> output = chatClient.prompt()
        .user("Tell me a joke")
        .stream()
        .content();

The ChatClient reference documents three return types for stream():

• Flux<String> content() gives you the text deltas as they are generated.
• Flux<ChatResponse> chatResponse() gives you each response chunk with metadata, for when you need more than text.
• Flux<ChatClientResponse> chatClientResponse() wraps the chunk with the execution context, which is how you can read what advisors did during the call, like the documents retrieved in a retrieval-augmented generation (RAG) flow.

For 95 percent of my UI work, content() is enough. The other two matter when you want to show retrieved sources or stream per-tool progress, which I get to below.

The measurable win is time to first token. On my agent, with the system prompt and nine tool schemas in context, the first token landed between two and five seconds after the request, and the full answer took ten to twelve. Users do not measure total time. They measure the moment the page starts responding. Streaming moves that moment from ten seconds to two.

The Controller: SseEmitter and Flux

The series runs on Spring MVC, so the browser connection is a SseEmitter, Spring's server-sent events class for the servlet stack. (If you are on WebFlux you would return the flux directly, but everything in this series uses Spring Web MVC.)

@RestController
public class ChatController {

    private static final Logger log = LoggerFactory.getLogger(ChatController.class);

    private final ChatClient chatClient;

    public ChatController(ChatClient chatClient) {
        this.chatClient = chatClient;
    }

    @GetMapping(path = "/chat/stream", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
    public SseEmitter stream(@RequestParam String message, @RequestParam String conversationId) {
        SseEmitter emitter = new SseEmitter(120_000L);

        Flux<String> tokens = chatClient.prompt()
                .user(message)
                .advisors(a -> a.param(ChatMemory.CONVERSATION_ID, conversationId))
                .toolContext(Map.of("conversationId", conversationId))
                .stream()
                .content();

        tokens.subscribe(
                token -> send(emitter, token),
                emitter::completeWithError,
                emitter::complete);

        return emitter;
    }

    private void send(SseEmitter emitter, String token) {
        try {
            emitter.send(token);
        } catch (IOException e) {
            // The client went away. Spring MVC and the container handle the cleanup.
            log.warn("SSE client disconnected mid-stream");
        }
    }
}

Three details in this controller are load-bearing.

The timeout is explicit. new SseEmitter(120_000L) sets a two-minute window. The container default for async requests is short (30 seconds in Tomcat), and a question that triggers two tool calls and a long answer can blow past that. When the emitter times out, the connection closes and the user loses the tail of the response.

The wiring from Part 2 is untouched. The conversation ID still goes through ChatMemory.CONVERSATION_ID, and the tools still get their context through toolContext. The advisor chain from the memory part runs during streaming exactly as it runs during a synchronous call. That is not a coincidence: built-in advisors like MessageChatMemoryAdvisor implement both the call and the stream paths, so memory keeps working token by token.

The error path is asymmetric. When the model stream itself fails, emitter::completeWithError is right, and the client gets a proper error event. But when emitter.send() throws an IOException, it means the browser disconnected. In that case the Spring reference docs are explicit: do not call complete or completeWithError yourself. The container detects the disconnect and initiates cleanup. My first version called completeWithError there too, and it produced a second error dispatch on a connection that was already gone. Log it and stop.

The Spring MVC async documentation covers the full SseEmitter behavior, including that client-disconnect rule.

Memory and Advisors Follow the Stream

Because Part 2 built memory as an advisor, this part barely needed changes. The advisors reference explains why: every advisor exposes a streaming variant. The call path uses adviseCall with a CallAdvisorChain, the stream path uses adviseStream with a StreamAdvisorChain that returns Flux<ChatClientResponse>.

What that means in practice: the memory advisor reads history for the conversation ID, appends it to the prompt, streams the response, and stores the new turn when the stream completes. A user can ask a follow-up question mid-conversation and the agent still knows what it said three messages ago, because the history handling is identical in both modes.

The one streaming-specific behavior to know: with a sliding window memory, the tokens that stream back are counted against the window on completion, not as they arrive. So the max message count stays predictable even for long streams.

Tools During Streaming: The Silent Gap

This is the part that cost me a weekend, and the answer is more subtle than the docs make it look at first.

In Spring AI 2.0, tool execution is handled by the ToolCallingAdvisor, which ChatClient registers automatically. When you call .stream() with tools enabled, the advisor still executes the tool loop for you. The model emits a tool-call chunk, the advisor runs the tool, and the stream resumes. That works.

The problem is the UI. While the tool runs, the model produces no text, so the stream goes quiet. The browser shows a cursor blinking over a half-finished sentence for several seconds. To the user, it looks like the answer died mid-sentence, which is worse than the original ten-second wait, because now they can see the stream and watch it stall.

I fix it with named SSE events. SseEmitter supports an event builder, so instead of sending only text, the server can send status events the client renders separately:

emitter.send(SseEmitter.event().name("status").data("Checking the order..."));
emitter.send(SseEmitter.event().name("status").data("Reading the return policy..."));

The client listens for the status event name and shows a small line under the streaming text. The model text keeps arriving in the default message events, and the status line explains the silence. It is a small change, and it is the difference between a stream that looks broken and a stream that looks busy.

To send those status events, you need to know when a tool runs. The default advisor does not tell you, so I switched that part of the flow to user-controlled tool execution, which the tools reference documents under "User-Controlled Tool Execution". The idea: disable the automatic advisor for the call, drive the tool loop yourself, and emit events between iterations.

ToolCallingManager toolCallingManager = ToolCallingManager.builder().build();
ToolCallback[] tools = ToolCallbacks.from(new ShoppingTools());
ChatOptions chatOptions = ToolCallingChatOptions.builder()
        .toolCallbacks(tools)
        .build();

Prompt prompt = new Prompt(userMessage); // the user's question
AtomicReference<ChatClientResponse> ref = new AtomicReference<>();

while (true) {
    new ChatClientMessageAggregator().aggregateChatClientResponse(
            chatClient.prompt()
                    .messages(prompt.getInstructions())
                    .options(chatOptions)
                    .advisors(AdvisorParams.toolCallingAdvisorAutoRegister(false))
                    .stream()
                    .chatClientResponse(),
            ref::set
    ).blockLast();

    ChatClientResponse response = ref.get();
    if (response.chatResponse() == null || !response.chatResponse().hasToolCalls()) {
        break;
    }

    // Tell the browser a tool is running, so the stream does not look dead.
    sendStatus(emitter, "Running a tool...");

    ToolExecutionResult result = toolCallingManager.executeToolCalls(prompt, response.chatResponse());
    prompt = new Prompt(result.conversationHistory(), chatOptions);
    ref.set(null);
}

sendStatus follows the same pattern as send above: build the event with SseEmitter.event().name("status").data(...), and wrap the emitter.send() call in a try/catch for the IOException that fires when the client disconnects.

The ChatClientMessageAggregator exists because tool calls span multiple stream chunks. You cannot tell from any single chunk whether the stream is done, so you aggregate the flux and inspect the result for hasToolCalls(). When it returns true, you execute the calls with the ToolCallingManager, build the next prompt from result.conversationHistory(), and stream again.

I keep the automatic advisor as the default for simple questions with no tools. The manual loop only activates for requests that involve tool calls. That keeps the common path simple and gives me the status events exactly when they matter.

The Client Side: EventSource

The browser side is deliberately boring. EventSource is built into every modern browser and speaks the SSE protocol natively:

const source = new EventSource(
  `/chat/stream?message=${encodeURIComponent(text)}&conversationId=${convId}`
);

source.onmessage = (event) => {
  appendToken(event.data);
};

source.addEventListener("status", (event) => {
  showStatus(event.data);
});

source.onerror = () => {
  // EventSource reconnects automatically. Show a note if it loops too long.
};

Two limits to know before you commit to this design.

EventSource is GET-only. You cannot send a POST body or custom headers with it. I pass the message in the query string, which means the message must fit in a URL, so my chat input caps at a few hundred characters. If you need long messages or auth headers, you switch to fetch() with a ReadableStream reader and parse the SSE format yourself. The Spring side does not care; it just writes text/event-stream.

Older browsers are a question. The Spring docs note that Internet Explorer never supported server-sent events. If your user base includes it, the fallback path is WebSocket with SockJS transports. For a developer-tools audience this is rarely a real constraint, but it is worth a line in your support matrix.

Three Gotchas I Hit in Production

Structured output and streaming do not mix. The ChatClient docs state it plainly: streaming is not supported when validateSchema() is active. My first pass tried to stream a JSON answer through the structured output converter and got an empty response. If you need both, you stream the raw text and validate the schema after aggregation, or you keep that one call synchronous.

The JDBC memory store silently drops tool messages. This one is in the chat memory documentation and it matters exactly because this agent uses tools: the JDBC-backed chat memory repository filters out assistant messages that contain tool calls and the tool response messages, on save. Your agent's history then looks like the tool never ran. The docs point tool-using applications to the Spring AI Session project with its JDBC session store instead. If you followed the Part 2 advice to persist memory in a shared backend, verify your chosen store keeps tool messages before you rely on it.

Watch the disconnect path in tests. When you test the controller with a client that closes early, the IOException path fires, and the Flux subscription must be cancelled cleanly. I have a test that opens the stream, reads two tokens, and closes. It caught two bugs: an unhandled exception in send(), and a leaked subscription that kept running the model call after the client was gone.

The Streaming Checklist

If you take nothing else from this part, take this list. It is the difference between a stream that impresses and a stream that confuses:

• Measure time to first token, not total time. It is the number your users actually feel.
• Set an explicit emitter timeout. The container default is short; tool-heavy answers outlive it.
• Keep the advisor wiring identical between call and stream. Memory, tools, and context should not care which mode you use.
• Emit named status events during tool execution, or accept a stream that visibly stalls.
• Do not call complete or completeWithError on client disconnect. The container owns that path.
• Check your memory store keeps tool messages. JDBC-backed chat memory filters them out by default.
• Remember validateSchema() disables streaming. Plan your structured output path around it.
• Test with a client that disconnects mid-stream. It is the failure mode you will meet first in production.

What Comes Next

I planned to fit observability and the multi-agent pattern into this part, and they did not fit. Streaming alone needed the space, so Part 4 covers observability for tool calls and latency, and Part 5 covers the pattern where one agent delegates to another.

Have you streamed tokens to your users yet? What did you do about the silent gap while tools run, and did your memory store survive tool calls? I read every response.

I write about Java, Spring Boot, and AI agents every week. Subscribe, it's free. Part 4 goes out soon.

If this was useful, bookmark it. The checklist at the end is the part you will reach for again when the stream works in development and stalls in production.