5 exercises — Use the precise vocabulary for faithfulness, context recall, hallucination, LLM-as-judge, and eval dataset curation.
0 / 11 completed
1 / 11
A colleague explains their RAG evaluation setup: "We run two RAGAS metrics — faithfulness and answer relevance. They keep getting confused in our team docs." What is the precise distinction between these two metrics?
Faithfulness and answer relevance measure orthogonal failure modes.
Faithfulness asks: "Is every claim in the generated answer supported by the retrieved context?" A faithful answer contains no invented facts — it sticks to what the retriever found. A low faithfulness score means the model is hallucinating content not present in the context chunks.
Answer relevance asks: "Does the answer actually address the user's question?" A perfectly faithful answer (grounded in context) can still be irrelevant if the retriever returned off-topic chunks and the model answered them faithfully but missed the real question.
In practice, a production RAG system needs both high faithfulness (no hallucination) and high answer relevance (on-topic responses). RAGAS computes both with LLM-as-judge calls.
Key vocabulary:
• faithfulness — the degree to which every claim in the answer is supported by the retrieved context
• answer relevance — the degree to which the answer addresses the user's actual question
• RAGAS — an open-source framework for evaluating RAG pipelines with LLM-assisted metrics
• hallucination — generating content not supported by the provided context or factual evidence
2 / 11
In a post-mortem, an engineer reports: "Context recall was 0.6 — we had the answer in the corpus but only retrieved 60% of the relevant chunks. Context precision was 0.9 — almost everything we retrieved was relevant." Which retrieval problem does the low context recall indicate?
Context recall measures coverage of the relevant corpus; low recall means missed evidence.
Context recall answers: "Of all the chunks in the corpus that are relevant to this query, what fraction did the retriever actually return?" A score of 0.6 means 40% of the evidence was left in the database — the generator never saw it and must either hallucinate or give an incomplete answer.
Context precision answers: "Of the chunks I did retrieve, what fraction were actually relevant?" A high precision (0.9) means the retrieved set is clean with little noise, even though the set is incomplete.
Together: high precision + low recall = a clean but incomplete retrieved context. The generator has accurate information but not enough of it to answer fully.
Key vocabulary:
• context recall — fraction of all relevant corpus chunks that were retrieved; measures coverage
• context precision — fraction of retrieved chunks that are relevant; measures noise level
• corpus — the full set of documents/chunks stored in the vector database
• retrieval gap — the relevant evidence that exists in the corpus but was not retrieved
3 / 11
A team lead writes in the incident report: "The model hallucinated a citation — it fabricated a paper titled 'Smith et al. 2019' that does not exist. The groundedness score for this response was 0.2." What does groundedness score measure?
Groundedness quantifies how well the answer is anchored in evidence.
A groundedness score (also called a "factual consistency" or "attribution" score) measures whether the claims in a generated response are supported by the provided source documents or retrieved context. A score of 0.2 means 80% of the response content has no traceable support in the evidence — it was invented by the model.
The hallucinated citation "Smith et al. 2019" is a classic example: it sounds plausible and is formatted correctly, but it is a factual inconsistency — a statement that cannot be verified against any real source.
Groundedness is distinct from confidence (hedging language like "I think" does not affect groundedness) and from similarity scores between query and context.
Key vocabulary:
• hallucination — generating factually incorrect or fabricated content presented as true
• groundedness score — metric for how much of a response is traceable to real source evidence
• factual inconsistency — a specific claim in the output that contradicts or is absent from the source
• citation hallucination — fabricating a plausible-looking but non-existent reference
4 / 11
An engineer proposes scaling evaluation: "Manual review is a bottleneck — we have 10,000 test cases. Let's use LLM-as-judge: we pass the question, context, and answer to GPT-4o and ask it to score faithfulness." What is the primary advantage of LLM-as-judge over human evaluation?
LLM-as-judge trades some accuracy for massive evaluation scale.
Human evaluation is the gold standard but cannot scale: annotating 10,000 samples per day requires significant headcount and time. An LLM judge — a model prompted to score dimensions like faithfulness, relevance, or coherence — can evaluate thousands of samples per hour at API cost fractions of a cent per call.
LLM-as-judge does have known limitations: it can exhibit position bias (favouring the first option in pairwise comparisons), self-enhancement bias (rating its own family's outputs higher), and verbosity bias (preferring longer answers). Good LLM-as-judge prompts include calibration examples, clear rubrics, and chain-of-thought scoring.
The key claim is scalability, not infallibility — LLM judges should be validated against human labels at a representative sample.
Key vocabulary:
• LLM-as-judge — using a language model to evaluate the outputs of another language model
• position bias — tendency of LLM judges to favour responses presented first in pairwise comparisons
• calibration — aligning the judge's scoring scale to human ground-truth labels
• evaluation rubric — explicit criteria the judge uses to assign a score to each dimension
5 / 11
A tech lead describes their quality framework: "Our eval suite checks 5 dimensions on every release: faithfulness, answer relevance, context recall, latency, and toxicity. We curate the eval dataset from production queries monthly." Why is eval dataset curation described as a continuous process rather than a one-time task?
Eval dataset curation is continuous because real usage patterns evolve.
An eval dataset built at launch reflects the query distribution at that moment. As the product grows, users ask questions the original dataset never anticipated: new product features, edge cases from domain shifts, adversarial inputs, seasonal topics. A static eval dataset gives false confidence — your scores stay high while the real failure modes are not being tested.
Production query mining — sampling real user queries, annotating them (manually or with LLM-as-judge), and adding them to the eval set — keeps the dataset aligned with actual usage. Monthly or per-release curation is standard in mature LLM teams.
This is distinct from model deprecations or vector database management — it is a data quality and representativeness practice.
Key vocabulary:
• eval suite — the complete set of test cases, metrics, and scoring logic used to assess model quality
• eval dataset curation — the ongoing process of selecting, annotating, and refreshing evaluation samples
• query distribution drift — gradual change in the topics and phrasing of real user queries over time
• production query mining — sampling real user inputs to represent genuine usage in the eval set
6 / 11
Sarah from the QA team sends you this Slack message: 'Hey, I'm seeing a lot of low scores on our LLM app's response to user queries about product features. The evaluation framework is flagging high toxicity in the responses, even when they seem perfectly reasonable. What's likely causing this issue?'
The question highlights a common issue: evaluation datasets can inadvertently contain biases or examples that trigger the toxicity detection model. LLMs don't inherently produce toxic content; it's often a result of the data they were trained on or how the toxicity system is configured. Adjusting the sensitivity of the toxicity detection system alone won't solve the underlying problem – you need to address the biased dataset.
7 / 11
David from the Engineering team sends this PR description:
'Evaluated the LLM's response to user queries about our new API. The evaluation framework returned a 'faithfulness' score of 0.8 and an 'answer relevance' score of 0.7. I've flagged potential issues with the context retrieval, as it seems to be missing key details from the documentation.
What does the term 'answer relevance' primarily measure in this context?
'Answer relevance' focuses on whether the LLM's response actually answers the user's question effectively. It doesn't necessarily assess factual correctness (faithfulness) or stylistic elements. A high score indicates the response is directly pertinent to the query; a low score suggests it misses key aspects of the user's need.
8 / 11
John from the LLM team sends this Slack message: 'The new evaluation pipeline is throwing errors when processing long context windows. It's timing out on queries exceeding 4096 tokens. We need to optimize for efficiency.' What does 'timing out' refer to in this context?
'Timing out' in this scenario means the evaluation process exceeded its maximum allowed execution time. This often happens when dealing with large contexts or computationally intensive tasks, triggering a system-level termination to prevent resource exhaustion. It's distinct from hardware failures or network issues.
9 / 11
Maria, a data scientist, is explaining the concept of 'few-shot learning' to her team. She says: 'We're providing the LLM with just a few examples of correct answers to guide its responses.' What is the primary benefit of using this approach?
The core benefit of few-shot learning is that it leverages the LLM's existing knowledge while guiding its response based on a small set of illustrative examples. This reduces the reliance on large datasets and enables faster adaptation to new tasks without extensive retraining – a common misconception is that it *increases* parameter count.
10 / 11
Reviewer: 'This PR description lacks specific details about the evaluation metrics used. It only mentions 'faithfulness' but doesn't specify the score threshold or how it was determined.' What is the primary issue highlighted in this reviewer's comment?
The reviewer's comment focuses on a critical deficiency in the PR description: the lack of specifics concerning the evaluation metrics. Without knowing the score threshold and how it was determined, it's impossible to assess the validity or effectiveness of the evaluation results. This highlights the importance of detailed reporting during evaluations.
11 / 11
During a standup update, Liam from the engineering team says: 'We're using LLM-as-judge to evaluate our chatbot's responses. We're feeding it the user query and the relevant context, and asking it to score for toxicity.' What is the purpose of using an LLM as a judge in this scenario?
The key purpose of using an LLM as a judge here is to mitigate human bias in evaluating toxicity. Human judgment can be subjective and prone to errors or inconsistencies. An LLM, trained on vast datasets, can provide a more objective assessment of potentially harmful content, though it's still important to monitor its performance.
What will I practise in "LLM Application Evaluation Language — LLM App Development"?
5 advanced exercises on LLM evaluation vocabulary — faithfulness, context recall, hallucination, LLM-as-judge, and eval suite curation.
How many exercises are in this module?
This module has 11 multiple-choice exercises, each with instant feedback and a full explanation of the correct answer.
Is this exercise free to use?
Yes. Every exercise on CoderSlingo, including this one, is free to use with no account, sign-up, or paywall.
Do I need to create an account to do these exercises?
No account is required. Just click an option to answer — your score for this session is tracked automatically in the progress bar above.
What happens if I choose the wrong answer?
You'll immediately see which answer was correct, plus a full explanation covering the vocabulary and reasoning behind it — mistakes are where most of the learning happens.
Can I retry the exercises if I want a higher score?
Yes — use the "Try again" button on the results screen to reset and go through all the questions again.
Is my progress saved if I close the page?
No. Progress is tracked only for your current visit; reloading or leaving the page resets the counter. This keeps the exercise simple and account-free.
Where can I find more LLM App Development exercises?
Browse the full LLM App Development hub for related drills, or check the "Next up" link below to continue with a connected topic.
How is this different from reading an article on the same topic?
Articles explain vocabulary and concepts in prose; this exercise tests and reinforces that vocabulary through active recall with immediate feedback — the two work best together.
Who writes these exercises?
Every exercise is written by the CoderSlingo team, drawing on real workplace English used in IT roles, then reviewed for accuracy and clarity.