Spring AI Retries and Embeddings: Failed Answers and Full Re-indexes — LLM Cost Control 4/4

Across the first three parts of this series, the goal has stayed the same: send fewer tokens without losing what the model actually needs to answer well. Part 1 started with measurement and model choice, so you know what you are paying for before you optimise anything. Part 2 looked at the tokens you already send: how long the answer runs, how much history you carry on every turn, and how caching can make repeated content cheaper. Part 3 tuned retrieval and tool calling, so a request now carries only the context that question actually needs.

All of that assumes each request is sent only once. In practice, a failed answer usually means a retry, and a retry sends everything again — so one usable answer can cost you several requests.

This part is about controls for failed responses and for the indexing side of RAG. It also closes the series: all ten drivers are listed on one map at the end.

• Driver #8 — Malformed output: paying full price for a failed answer
• Driver #9 — Embeddings at scale: small numbers, big multipliers
• The one-glance map: all ten drivers

Prices: list prices where a ratio matters, an example rate of $1 per million input tokens elsewhere. Full note in Part 1.

Driver #8 — Malformed output: paying full price for a failed answer

When the model returns JSON that your code cannot parse, the usual fix is to retry the call. Each retry resends the request context — system prompt, conversation history, RAG context, and tool schemas — so the input tokens are billed again and the failed answer is billed as output in full, cut off or not. For example, if 5% of requests require one retry, input token usage increases by roughly 5% without producing additional business value. A retry policy allowing three retries can make worst-case requests cost up to four times more.

Spring AI's .entity() convenience method on ChatClient's CallResponseSpec traditionally used a prompt-based structured output approach: the StructuredOutputConverter generated format instructions (including the JSON schema) and added them to the prompt. The model was then expected to return JSON matching that schema, which Spring AI parsed into the target Java type afterwards.

Spring AI 2.0 adds two per-call controls through EntityParamSpec: .validateSchema() and .useProviderStructuredOutput(). The first enables schema validation; the second switches from prompt-based instructions to the provider's native structured output mechanism, where supported.

Ticket ticket = chatClient.prompt()
    .user(text)
    .call()
    .entity(Ticket.class, spec -> spec
        .useProviderStructuredOutput()  // schema enforced by the provider
        .validateSchema());             // fallback: validate and retry

.useProviderStructuredOutput() sends the schema to the provider as an API-level rule instead of as prompt text. The provider runtime enforces schema conformance for supported models, so invalid structured responses are prevented during generation. The schema instructions also disappear from the prompt, reducing input token usage on every call. Spring AI 2.0 supports this for providers with native structured output capabilities, including OpenAI, Anthropic, Google GenAI, Mistral AI, and model-dependent Ollama support depending on the model. Check the limitations for your provider before relying on it: OpenAI, for example, rejects top-level array schemas, so requesting a List<T> fails, and Ollama models with reasoning mode may still return plain text.

.validateSchema() adds StructuredOutputValidationAdvisor: the response is checked against the schema, and if validation fails, the validation error is appended to the user message and the model is called again, up to 3 retry attempts by default. Each retry is another model call and consumes tokens, so use validation as a recovery mechanism rather than the primary strategy. Combine it with provider-native structured output when available to prevent failures before they happen.

Driver #9 — Embeddings at scale: small numbers, big multipliers

Embedding calls look cheap on a per-request basis, but a RAG pipeline runs them across your entire document collection, and may run them again whenever documents are added, updated, or a full re-index is required.

Three things drive that cost: the embedding model's price, the dimensionality of the vectors (which affects storage and retrieval costs), and how much text you actually need to embed during each indexing run. A fourth source of embedding calls has nothing to do with documents at all — that one comes at the end.

The embedding model

Model price comes first. According to OpenAI's pricing (August 2026), text-embedding-3-small costs $0.02 per million input tokens, compared to $0.13 per million for text-embedding-3-large — making the small model about 6.5× cheaper. Embedding a 50-million-token document collection therefore costs about $1 with the small model versus $6.50 with the large one each time you re-embed the entire corpus, such as during a full re-index. For many retrieval workloads, the smaller model delivers sufficient retrieval quality. Rather than assuming the larger model is necessary, evaluate both using the same fixed test set of questions introduced in Driver #6, Part 3.

Vector dimensionality

The second lever is vector dimensionality. EmbeddingOptions exposes a dimensions setting for providers that support it, such as OpenAI's text-embedding-3 models (including deployments on Azure). Reducing the number of dimensions does not lower the embedding API cost, which is based on input tokens, but it significantly reduces vector storage requirements. Assuming float32 vectors (4 bytes per dimension), one million vectors at 3,072 dimensions occupy about 12 GB of raw storage, while 512-dimensional vectors require about 2 GB. Smaller vectors also reduce index size and memory usage and often reduce similarity search latency because less data must be stored and processed.

spring.ai.openai.embedding.model=text-embedding-3-small
spring.ai.openai.embedding.dimensions=512

Set the number of dimensions before you start loading data. A vector index normally requires all vectors to have the same dimensionality, so changing this setting later usually requires creating a new index and re-embedding the existing documents.

Batching and incremental indexing

Third: batch requests, and skip what has not changed. When you add documents through a VectorStore, Spring AI uses the default TokenCountBatchingStrategy, which groups documents into batches based on token count. It keeps batches below the embedding model's token limit (OpenAI's 8,191-token limit with a default 10% safety margin) instead of sending each chunk as a separate embedding request. You can replace this with your own BatchingStrategy bean if needed.

What the framework cannot know automatically is which documents actually changed. Store a content hash in each Document's metadata and re-embed only chunks whose hash differs. If only 2% of chunks require updating each week, embedding work can be roughly 50× smaller than a full re-index.

Chat memory is an embedding cost too

Everything above assumes you are embedding documents. There is a second source, and it is easy to miss because it does not look like indexing at all.

Part 2, Driver #4 offered VectorStoreChatMemoryAdvisor as an option for very long conversations. Instead of resending the whole chat on every turn, it saves the messages in a vector store and adds back only the ones that match the current question. Each request then stays the same size, however long the session runs. That is a real saving on the chat side.

The cost does not disappear, though. It moves. Every message the advisor saves has to be turned into a vector first, and the search itself embeds the question too — so each turn brings its own embedding cost. A document collection is embedded once and updated from time to time; a busy support chat produces new messages all day, so here the number of embedding calls follows your traffic, not the size of your collection. The controls are the same — a smaller model, fewer dimensions — but count the calls before you choose this advisor over a message window.

Wrap-up: the one-glance map

Every driver above follows the same basic pattern: tokens repeat or grow somewhere, and Spring AI gives you a control to cut them. Here is the full map:

Where to start

Ten drivers is a list, not a plan. Working through them in order is the slowest way to use this map, because the drivers are not equal — in most applications two or three of them carry almost the whole bill, and which ones depends on what you built.

So start at Driver #0. Set up your metrics, watch prompt and completion tokens, and let the graph choose your next move.

One last thing worth saying: every control in this series trades something away. A smaller window forgets more. A tighter topK misses more facts. A lower reasoning effort reasons less. None of these is free, and a cheap application that gives poor answers is not cheap — users simply ask again. The main point here is knowing what each token buys you, and paying only for the ones that buy something.

That is the series. If you have hit a cost driver that is not on the map, or your numbers came out differently from the estimates here, the comments are the right place for it — several ideas from readers of Part 1 ended up shaping the later parts.