devLap

Asking a RAG in the wrong language

python dev.to

Every article in this series so far assumed that the person asking speaks the language of the documents. The SEPA rulebooks my RAG indexes exist only in English, and so did every question in my golden set. But a payments engineer in Madrid or Kyiv asks in their own language, and I had never measured what that does to retrieval. I can read Spanish and Russian myself, so those two were the cheap place to start.

The question for this piece is narrow: what happens to the retriever when the question changes language and the corpus does not. It stops at whether the right page still lands in the top 5; answer quality and the generator wait for the next piece.

Hypothesis

Three things I expected going in:

  • Vector retrieval would lose something cross-lingually, since the embedder has to map a Spanish question next to an English paragraph, but not much. Vendors describe these models as multilingual.
  • Hybrid retrieval (vector plus Postgres full-text search, fused by reciprocal rank fusion) would quietly stop helping, because the full-text index is built with the english configuration. My guess was that a Spanish query would match nothing and the fusion would fall back to the vector ranking alone.
  • Translating the question to English first, then running the normal English pipeline, would recover most of the loss for the cost of one small LLM call.

The second one turned out wrong in an instructive way.

Development

The corpus is the production index untouched: 484 chunks from the two 2025 EPC rulebooks, with the contextual blurbs from the previous improvement round already in the stored vectors. The eval reads it read-only.

The golden set is the 20 answerable SEPA questions from the model matrix article, each now labelled with the page that answers it, and each translated into Spanish and Russian. Scheme and message names (SCT, SCT Inst, PSP, Recall, Return) stay in English in all three versions, because that is how practitioners actually talk about them. It also turned out to matter for what follows.

Every question, in each of its three languages, goes through four retrieval paths, and each path hands back a top 5 that is scored against the page labels:

  1. Vector only: embed the question, take the 5 nearest chunks. Run once per embedder.
  2. Full-text only: Postgres full-text search, the 5 best lexical matches. Run once per language configuration.
  3. Hybrid: take the top 20 from vector and the top 20 from full-text, fuse the two rankings, keep the top 5. This is the production hybrid path with one change: the language configuration is a variable instead of the hardcoded english.
  4. Translate first: Claude Haiku turns the Spanish or Russian question into English, then path 1 runs with the production embedder. 
flowchart LR
    Q["question<br/>(en / es / ru)"] --> V["1. vector only"] --> T5a["top 5"]
    Q --> F["2. full-text only"] --> T5b["top 5"]
    Q --> H["3. hybrid: vector top-20 + full-text top-20,<br/>fused by reciprocal rank"] --> T5c["top 5"]
    Q --> TR["4. translate to English,<br/>then vector only"] --> T5d["top 5"]
    T5a & T5b & T5c & T5d --> S["scored against<br/>the page labels"]

The fusion in path 3 is reciprocal rank fusion, the same one production uses. Each chunk scores 1 / (60 + its rank) in every list it appears in, and the two scores are added. A chunk at rank 1 in one list and absent from the other gets 1/61; a chunk at rank 10 in both gets 2/70, which is more. So the fusion rewards agreement between the two lists and treats both lists as equally trustworthy. That second property is the one this piece is about.

The three embedders are the ones from the matrix piece: text-embedding-3-small (production, its vectors read straight from the database), voyage-4 and gemini-embedding-001. The full-text configurations tried were english, spanish, russian and simple (no stemming at all).

Two metrics. hit@5 against the page labels is the real one; with 20 questions every hit is worth 0.05. The second needs no labels at all: how much of the top 5 for the Spanish or Russian question overlaps with the top 5 for the same question in English. It says whether the retriever is even looking at the same pages.

Note: the production /ask endpoint runs plain vector retrieval. Hybrid exists behind an eval flag, since the round where I measured it landed at exactly the same recall as vector alone. So nothing below describes a live failure; it describes what turning that flag on would do.

Results

hit@5 over 20 questions, production embedder unless stated:

leg en es ru
vector, text-embedding-3-small 0.80 0.70 0.65
vector, gemini-embedding-001 0.85 0.75 0.70
vector, voyage-4 0.70 0.70 0.60
full-text alone, english config 0.50 0.35 0.35
hybrid (vector + full-text), english config 0.65 0.60 0.55
hybrid, spanish config 0.65 0.60 0.60
translate to English, then vector n/a 0.70 0.75

Top-5 overlap with the English question: 0.72 for Spanish, 0.63 for Russian on the production embedder; 0.81 and 0.73 on the Gemini embedder; 0.82 and 0.76 on Voyage.

The cross-lingual loss is real and small. Spanish costs the production embedder 2 questions in 20, Russian 3. English itself misses 4 of the 20; Spanish keeps all four of those misses and adds two, Russian keeps three of them and adds four, so the language penalty sits mostly on top of what English already gets wrong. Gemini also loses 2 and 3; Voyage starts lower in English and loses 0 and 2. Meaning, the embedder you already have is not the problem here, or at least not the biggest one.

Hybrid costs recall in every language, English included. Against plain vector it drops 0.80 to 0.65 in English, 0.70 to 0.60 in Spanish, 0.65 to 0.55 in Russian. In English it loses 3 hits and gains none; in Spanish it loses 2 and gains none; in Russian it loses 3 and gains 1. The English drop is new since the round where hybrid was at parity: contextual retrieval lifted the vector leg from 0.60 to 0.80 in between, and the full-text leg stayed where it was, so fusing them now dilutes a better ranking with a worse one.

The mechanism is where my second hypothesis broke. I expected a Spanish question to match nothing in an English full-text index. It matched 192 of 484 chunks. The production query rewrites the full-text search from AND to OR (a natural-language question shares only some words with a terse rulebook paragraph, so requiring every word matched nothing), and under OR any shared token is enough. For the question about Recall deadlines the english stemmer turned the Spanish text into this:

'en' | 'qué' | 'plazo' | 'debe' | 'enviars' | 'un' | 'recal' | 'de' | 'sct' | 'por' | 'duplicidad' | 'o' | 'error' | 'técnico' | 'frent' | 'fraud'
Enter fullscreen mode Exit fullscreen mode

recal, sct, error and fraud survive translation, so 192 chunks match, ranked by how often they mention those tokens. The full-text leg returned all 20 of its 20 candidates on every one of the Spanish questions, and on 19 of the 20 Russian ones (one Russian question shared no token at all and matched zero, which is the behaviour I had expected everywhere). The top of that list for the Recall question was the SCT Inst page listing Recall reasons, the change log, and an annex, all of them about Recalls and none of them the page with the deadlines. Reciprocal rank fusion then weights that list equally with the vector list, so twenty plausible but wrong pages get the same say as the twenty the embedder picked.

flowchart LR
    Q["Spanish question<br/>¿En qué plazos debe<br/>enviarse un Recall...?"] --> TS["plainto_tsquery('english'),<br/>AND rewritten to OR<br/>recal | sct | error | fraud | plazo | ..."]
    TS --> FT["192 of 484 chunks match<br/>full-text top-20 by ts_rank:<br/>SCT Inst Recall p.40, change log p.8,<br/>annex p.134, ..."]
    Q --> VEC["vector top-20<br/>deadline page p.31 at rank 3"]
    FT --> RRF["RRF fusion<br/>both lists weighted equally"]
    VEC --> RRF
    RRF --> OUT["hybrid top-5<br/>five SCT Inst Recall pages,<br/>p.31 pushed out"]

For that question the vector leg alone had the deadline page at rank 3. After fusion the top 5 were five pages from the SCT Inst Recall section, and the deadline page was gone.

Switching the full-text configuration to spanish or russian changed nothing for Spanish or Russian questions (0.60 and 0.55 to 0.60, within one question). The documents are English, so a Spanish stemmer has nothing to stem on the index side; the only cross-language matches were the acronyms, and those match under any configuration. The fix is not in the stemmer.

Translating first helped Russian and did nothing for Spanish, which looked odd, since Spanish is the closer language. Claude Haiku turned the 40 questions into English for $0.0071 total, about $0.0002 and 0.7 seconds per query. Russian went from 0.65 to 0.75; Spanish stayed at 0.70 with exactly the same hits. Top-5 overlap with the English question rose to 0.87 and 0.81 for both, so translation did put the retriever back on the same pages even where the count did not move.

The explanation is in the ranks, not in the languages. The two questions that separate the Spanish and Russian translate legs are ones where the original English question itself only just makes it: the answer page sits at rank 4 for one and rank 5 for the other. A translation is a paraphrase, and any paraphrase can move a rank-5 page to rank 6. On the value-limits question the Russian back-translation came out word for word identical to my English original (rank 4, hit); the Spanish one came out as "establish a maximum amount per transaction" (rank 6, miss). On the execution-time question the Spanish back-translation dropped the word "credit" from "credit transfer", and the page went from rank 5 in English to rank 13. Russian also had more to recover: native Spanish already found two pages (settlement certainty, currency) at ranks 1 and 3 that native Russian had at rank 7. Meaning, translation buys back what the language cost, and then the fragile English questions cost it right back.

Note: I also wanted to say there were no false positives, and I cannot. The golden set carries 5 questions whose answer is verifiably not in the corpus (they are about direct debits, card fees, SWIFT messages and TARGET2 hours). I ran them in all three languages through the production answer path: 14 of 15 were refused correctly, in the language of the question. The 15th was the Russian version of the direct-debit refund question, which came back with "13 months" and a citation. Thirteen months is the window for recalling a SEPA Credit Transfer, a different payment instrument entirely; the English and Spanish versions of the same question were refused. One hallucination in fifteen, and it appeared only cross-lingually.

Where the same knob lives on other stacks, for anyone running one of the systems from the comparison with lexical search turned on (the comparison itself ran them vector-only):

  • Postgres: the text search configuration is the second argument of to_tsvector and plainto_tsquery; the index is built with one of them too, so a per-query change means either a second index or a sequential scan.
  • LlamaIndex: BM25Retriever takes stemmer=Stemmer.Stemmer("english") and language="english" for the stop-word list, both defaulting to English.
  • Haystack: the in-memory BM25 lowercases and tokenizes with the regex (?u)\b\w+\b, and that is all. No stemming, no stop words, no language to set, so a Spanish question matches an English corpus on exactly the shared tokens this piece is about.
  • LangChain: BM25Retriever takes a preprocess_func; the default splits on whitespace, and any stemming or stop-word handling is yours to write.
  • OpenAI file search: the docs say retrieval is "semantic and keyword search" and expose no language, tokenizer or keyword setting. NotebookLM exposes nothing at all.

In every one of these the lexical leg has a language baked in somewhere, and none of them will tell you when the question stops matching it.

Conclusion

  • Cross-lingual retrieval with the production embedder loses 2 to 3 questions in 20 against English. Small, consistent across three embedders, and sitting on top of the 4 questions English misses on its own.
  • Hybrid search is not neutral to language, and not for the reason I guessed. It does not fall back to vector when the query language changes; it fuses in a full list of confident lexical matches on shared acronyms, and with an OR-joined query that list is always full. It cost 2 to 3 questions in every language, English included.
  • The full-text language configuration is the wrong knob when the documents and the questions disagree. The right knob is fusion: skip or down-weight the lexical leg when the query language is not the corpus language, or require more than one matching term before a chunk counts.
  • Translating the question first is cheap ($0.0002, 0.7 s) and recovered Russian to within one question of English. Where it did not help, the reason was two English questions that already sit at rank 4 and 5, so a paraphrase pushes them out. Translation is a paraphrase.
  • The refusal boundary that held for all 20 trap questions in the matrix piece let one through here, in Russian only. Cross-lingual questions deserve their own refusal traps in the golden set.
  • Twenty questions means every hit is 0.05; I am reporting deltas of 2 and 3 questions and would not defend anything smaller. Honest rather than optimized.

Out of Scope

  • The generator, beyond the 15 refusal checks above. Whether Claude answers a Spanish question correctly from English passages, and in which language, is the next measurement.
  • A corpus that is itself not in English. That is a different problem: the chunker and the embedder both see different text, and it is the subject of the next piece. The obvious shortcut, translating the whole corpus into one language before indexing, is a document-translation problem at scale, where every sentence has to stay correct and nobody signs off on it. The tooling that exists for that is quality estimation, models like CometKiwi that score a translation without a reference, and even its authors warn about reading the scores as guarantees. A recent cross-lingual retrieval study (Goworek et al., 2025) found that dense retrievers trained for cross-lingual use "derive little benefit from document translation" and recommends multilingual embeddings over translation pipelines. So the honest position for now: do not translate the corpus, measure the embedder on it.
  • A fusion rule that is language-aware. Argued above, not implemented or measured.

Sources

Source: dev.to

arrow_back Back to Tutorials