Practise vocabulary for evaluating recommendation systems: precision@k, recall@k, NDCG, A/B testing, and offline vs. online evaluation.
0 / 10 completed
1 / 10
Precision@k measures the proportion of ___ items in the top-k recommendations shown to a user.
Precision@k = relevant items in top-k / k. If 7 of the top-10 recommendations are relevant, precision@10 = 0.7. It measures recommendation accuracy without considering position within the top-k.
2 / 10
Recall@k measures the proportion of ___ relevant items that appear in the top-k recommendations.
Recall@k = relevant items in top-k / total relevant items. If there are 20 relevant items and 12 appear in top-20, recall@20 = 0.6. High recall means the recommender covers most of what the user would find relevant.
3 / 10
Offline evaluation using ___ evaluates recommendation quality on historical data — faster but doesn't account for counterfactual feedback.
Offline evaluation splits historical interaction data into train/test sets and measures how well the model predicts held-out interactions. It's fast and cheap but may not reflect online performance — the test set only contains observed interactions, not counterfactual (unexplored) ones.
4 / 10
Online A/B testing for recommendations measures the ___ impact of a new recommender by exposing different user groups to different recommendation strategies.
Online A/B testing measures actual business impact: CTR, conversion rate, revenue, session length, return visits. It captures the true effect on user behaviour — offline metrics (precision, recall) don't always correlate with business outcomes.
5 / 10
Catalogue ___ measures how many distinct items from the total catalogue the recommender exposes to users — low coverage indicates a 'popularity bias'.
Catalogue coverage = unique items recommended / total catalogue items. Low coverage means the recommender concentrates on a small set of popular items, leaving the 'long tail' unexposed. This hurts item providers and limits discovery for users.
6 / 10
Sarah (Senior Developer) just submitted a PR with a new collaborative filtering recommendation engine. During the code review, Mark asks: 'How are we measuring the diversity of recommendations here? I'm concerned that it's only pushing variations on JavaScript tutorials.' Which metric should Sarah focus on to address Mark's concern?
Recall@k directly addresses whether relevant items (diverse tutorials) are appearing within the top-k recommendations. While Precision@k focuses on accuracy, it doesn't inherently capture diversity. Catalogue Coverage is related but doesn't explain why Mark believes the recommendations are homogenous; NDCG is a more sophisticated ranking metric and not the primary tool for assessing recommendation diversity in this scenario.
7 / 10
David, a data scientist, sends a Slack message to the team: 'I'm running offline evaluation on the new hybrid model using Recall@5. The initial results are promising – it's achieving 85% recall at k=5 for users who purchased books in the past year. However, I suspect this might be inflated because we only used historical data from the last six months.' What is David primarily concerned about?
David's message highlights that Recall@5 is being evaluated on a dataset with limited temporal scope (six months). This creates an artificially inflated recall score because the model has only learned from recent trends. A longer historical period would provide a more realistic and robust measure of performance; bias and NDCG are not directly addressed in this specific Slack conversation.
8 / 10
Elena is documenting the API response for the recommendation service. The response includes a JSON payload with a 'diversity_score' field. What does this field most likely represent?
The diversity_score, as indicated by its name, is most likely a metric designed to quantify the variety of items within the recommended set. This often involves counting distinct categories or item types. The other options represent different facets of recommendation quality that wouldn't be directly encapsulated in a single 'diversity' score.
9 / 10
Ben writes the PR description for a change to the recommendation system: 'Implemented a new algorithm that uses collaborative filtering. The goal is to improve Recall@10 and increase user engagement by promoting less popular items. We'll track this using online A/B testing, comparing the new version against the existing baseline.' What is Ben primarily focusing on when describing the change?
Ben explicitly states that they will use online A/B testing to measure the impact. This approach is crucial for evaluating recommendation changes in a real-world setting, by comparing user behavior between the new and existing strategies. While recall@10 and the algorithm itself are relevant details, the focus here is on *measurement* of success.
10 / 10
Chloe, a product manager, notices that the recommendation engine consistently recommends only the top 3 most popular books. 'This is creating a popularity bias,' she says. What does she mean by 'popularity bias' in the context of recommendations?
'Popularity bias' refers to the tendency of a recommendation system to disproportionately recommend highly popular items. This occurs when the model doesn't adequately consider less-popular but potentially relevant items, creating an echo chamber where only the most well-known choices are presented. The other options describe different aspects of how recommendations are generated, not the core issue of unbalanced representation.
What will I practice in "Recommendation Evaluation Vocabulary"?
This is a Recommendation Systems Language exercise set. It walks through 10 scenario-based multiple-choice questions built around real usage of recommendation systems language terminology that IT professionals encounter on the job.
Is this exercise free to use?
Yes. Every exercise on CoderSlingo, including this one, is free to complete with no account, sign-up, or paywall.
How many questions are in this exercise?
This set contains 10 questions. Each one shows immediate feedback and a detailed explanation after you answer, so you learn the correct usage right away rather than waiting for a final score.
Do I need prior experience to complete this exercise?
No prior experience is required. Each question includes a full explanation covering the reasoning behind the correct answer, so the exercise itself teaches the recommendation systems language vocabulary as you go.
Can I retry the exercise if I get questions wrong?
Yes — use the "Try again" button on the results screen to reset your answers and go through all the questions again. There is no limit on attempts.
Is my progress saved?
Your answers and score for the current session are tracked in the browser as you go. No account or login is needed, and there is nothing to install.
What if I don't understand a term used in a question?
Read the explanation shown after you answer each question — it breaks down the correct term in plain English with a real-world example. You can also check the site Glossary for quick definitions.
How is this different from reading a blog article on the topic?
Exercises like this one are interactive drills that test and reinforce specific vocabulary through multiple-choice questions, while blog articles explain concepts in prose. Practising here after reading builds active recall, not just passive recognition.
Where can I find more Recommendation Systems Language exercises?
See the Recommendation Systems Language exercises hub for the full set of related pages, or browse all exercise categories from the main Exercises index.
Can I use this exercise to prepare for a technical interview?
Yes — recommendation systems language vocabulary comes up often in technical discussions and interviews. Pair this exercise with our dedicated Interview Preparation section for role-specific practice.