Most RAG systems ship without a single metric. Module 4 of LLM Zoomcamp fixes that.
Here's what I built and what the numbers showed.
The Setup
Knowledge base: 72 course lesson pages pulled from GitHub at a fixed commit so everyone works with the same data.
Ground truth: 360 questions generated by an LLM — 5 per page, worded differently from the source so keyword overlap doesn't inflate scores.
Three search methods under test: keyword search with minsearch, vector search with ONNX embeddings, and hybrid search combining both via Reciprocal Rank Fusion.
The Metrics
Hit Rate — did the right page appear in the top 5 results? Binary per question, averaged across 360.
MRR — how high up was the right page? First place = 1.0, second = 0.5, third = 0.33. Penalizes systems that find the answer but bury it.
The Numbers
Text search: Hit Rate 0.76, MRR 0.59
Vector search: Hit Rate 0.73, MRR 0.55
Hybrid search (k=1): Hit Rate 0.84, MRR 0.65
Text beat vector. Hybrid beat both. Tuning k from 60 to 1 in RRF added measurable MRR improvement.
The Takeaway
# benchmark any search function in one call
results = evaluate(text_search, ground_truth)
results = evaluate(vector_search, ground_truth)
results = evaluate(hybrid_search, ground_truth)
Once you have a ground truth dataset and an evaluate() function, every search decision becomes quantitative. Change a parameter, re-run, see the delta. No more guessing.
Code
github.com/Derrick-Ryan-Giggs/llm-zoomcamp-2026
Free course: github.com/DataTalksClub/llm-zoomcamp