5 exercises — Master the vocabulary of running LLM applications in production: prompt versioning, model registry, observability, A/B testing, and quality drift monitoring.
0 / 11 completed
1 / 11
A team lead proposes: "We need to stop hardcoding prompts in the application repository. Let's move them to a prompt registry and version-control them separately so we can roll back bad prompts without a code deployment."
What does a prompt registry provide?
A prompt registry decouples prompt management from code deployments.
A prompt registry (also called a prompt store or prompt management platform) stores prompt templates as versioned, named artifacts — independently of the application codebase. This lets teams roll back to a previous prompt version when quality degrades, audit who changed a prompt and why, and run side-by-side experiments, all without touching application code. Tools such as LangSmith, Langfuse, PromptLayer, and MLflow Prompt Registry provide this capability.
Embedding storage (A) is for vector search, not prompt versioning. Injection linting (B) is a validation step, not a registry. Feature flags (C) control availability but don't version or store prompt content.
Key vocabulary:
• prompt registry — a versioned, centralised store for named prompt templates, separate from application code
• prompt versioning — tracking every change to a prompt with a version identifier and metadata
• rollback — reverting a prompt to a previous version when a newer one causes quality regression
• audit trail — a log of who changed a prompt, when, and why
2 / 11
A senior ML engineer writes in a deployment doc: "We registered the new fine-tuned model in our model registry under the alias production. Once evaluation passes, we'll do a blue-green swap — point the alias at the new version and keep the old one live temporarily."
What does "registering a model" mean in this context?
Registering a model creates a versioned, named entry in the model catalog.
A model registry (MLflow, Vertex AI Model Registry, SageMaker Model Registry) catalogues trained or fine-tuned model artifacts with version numbers, evaluation metrics, tags, and lineage metadata. Aliases like production, staging, or champion are mutable labels that point to specific versions — swapping an alias changes which model version is active in production without modifying any application code.
A blue-green swap means both the old (blue) and new (green) model versions are live simultaneously during the transition; traffic shifts to green once quality is confirmed.
Key vocabulary:
• model registry — a central catalog that stores versioned model artifacts with metadata
• registered model — a model artifact that has been catalogued with a name and version
• alias — a mutable label (e.g., production) pointing to a specific model version
• blue-green deployment — running two versions side-by-side, then switching traffic from old (blue) to new (green)
3 / 11
An LLMOps dashboard alert reads: "p99 TTFT rose from 1.1 s to 4.8 s over the last hour. Average: 42 prompt tokens, 310 completion tokens. CPR now $0.0031."
What does TTFT specifically measure?
TTFT (Time To First Token) is the primary perceived-latency metric for streaming LLM UIs.
In streaming setups, the user sees the first token appear before the full response is generated. TTFT measures the waiting time before that first token arrives. A high p99 TTFT (4.8 s here) means 1% of requests wait nearly 5 seconds for any response — a poor UX. p99 is the 99th-percentile value: 99% of requests are faster, 1% are slower.
CPR (cost per request) is calculated from token usage × per-token price. TTFT increase without a change in token counts typically points to provider-side latency spikes, cold starts, or queue congestion.
Key vocabulary:
• TTFT — Time To First Token; streaming latency from request submission to first response token
• p99 latency — the 99th-percentile latency; 99% of requests complete within this duration
• prompt tokens — tokens in the input sent to the model (system prompt + context + user message)
• completion tokens — tokens the model generates in its response
• CPR — cost per request; token usage multiplied by per-token pricing
4 / 11
A team runs prompt experiment exp-042: they route 50% of production traffic to the existing system prompt (control) and 50% to a new version with a chain-of-thought preamble (treatment). Quality scores are measured on 500 sampled traces from each group.
What is the control in this experiment?
In any A/B experiment, the control is the existing production baseline.
The control is the current state — the prompt already in production. The treatment is the new variant being tested. Measuring both under the same conditions (same traffic mix, same time window) isolates the effect of the prompt change. "Measuring quality score on production traces" means evaluating real-world inputs rather than a static benchmark dataset — which is more representative of genuine user needs.
Key vocabulary:
• control — the existing production variant; the baseline for comparison in an A/B experiment
• treatment — the new prompt variant being tested against the control
• A/B test (prompt experiment) — an experiment comparing two prompt variants by splitting traffic between them
• quality score — a metric (human rating, LLM-as-judge, RAGAS metric) quantifying LLM output quality
• production trace — a recorded input/output pair from live traffic, used for offline evaluation
5 / 11
An on-call engineer receives an automated alert: "Faithfulness score on live traffic dropped from 0.87 to 0.61 over 48 hours — LLM quality drift detected. Investigate RAG pipeline."
What is the most accurate description of LLM quality drift in a production system?
Quality drift is a decline in output quality detected through continuous evaluation of live traffic.
LLM quality drift refers to gradual degradation in the quality of model outputs over time — distinct from latency or infrastructure issues. Common causes: upstream model version changes by the provider, RAG retrieval degradation (stale or corrupted vector index), prompt-data distribution shift (queries change in ways the prompt doesn't handle well), or context window misuse.
Detecting drift requires continuous evaluation — running faithfulness, answer relevance, or custom quality metrics on sampled production requests and triggering alerts when scores cross a threshold.
Key vocabulary:
• quality drift — gradual degradation in LLM output quality detected in production
• faithfulness score — a RAG evaluation metric measuring whether the answer is grounded in retrieved context
• distribution shift — a change in the statistical distribution of incoming queries; stale prompts may not handle new query patterns well
• continuous evaluation — automated periodic scoring of live traffic to detect quality regressions early
• alert threshold — the minimum acceptable quality score; crossing it triggers an incident alert
6 / 11
Liam (Senior ML Engineer) comments on a PR describing the deployment of a new LLM application: 'I've added a step to automatically trigger a canary rollout. We'll start with 5% of traffic directed to the updated model, and if performance metrics remain satisfactory for 24 hours, we'll gradually increase it to 100%.' What does a 'canary rollout' primarily aim to achieve?
A canary rollout is a strategic deployment technique used to minimize risk. It involves routing a small percentage of live traffic to the new version before fully deploying it. This allows engineers to monitor its performance and identify any issues in a controlled setting, without impacting the majority of users. The core goal is early detection – finding problems *before* they affect everyone.
7 / 11
Maya (Code Reviewer) flags a comment in a PR description: 'The prompt for the chatbot is being dynamically generated based on user input. We're using an LLM to create personalized prompts that are then fed into our main model.' What does this describe?
This description outlines the use of an LLM to generate prompts dynamically. This is common in modern LLM applications where personalized or context-aware prompts are needed. The key element here is the 'dynamic generation' – the prompt isn't fixed but created on the fly based on specific user input, enhancing relevance and potentially improving performance.
8 / 11
Sarah (Lead ML Engineer) sends a Slack message to the team: 'Just noticed a spike in latency for queries using the llama2-70b model. It's currently averaging 15 seconds – significantly higher than our target of 3 seconds. We need to investigate potential bottlenecks.' What is Sarah primarily concerned about?
Sarah's message focuses on latency – specifically the response time. High latency directly impacts user experience and can lead to increased operational costs. While accuracy and RAG integration are important, the immediate issue highlighted is performance degradation. The Slack conversation emphasizes a practical operational concern.
9 / 11
David (Junior Developer) is writing a PR description for deploying a new LLM application. He includes the following text: 'We're using a blue-green deployment strategy to minimize downtime and risk. After thorough testing in staging, we will switch traffic from the 'staging' environment to the 'production' environment automatically via DNS updates.' What does 'blue-green deployment' primarily refer to?
'Blue-green deployment' is a specific technique where you maintain two identical environments – one 'live' (e.g., 'production') and the other in standby mode ('green'). Traffic is switched to the 'green' environment after testing, allowing for rapid rollback if necessary without disrupting users. It's about minimizing downtime through parallel operation.
10 / 11
Ben (ML Engineer) sends a Slack message: 'Our LLM app's response latency is spiking – currently averaging 8 seconds for simple questions. We suspect the model isn't efficiently utilizing its context window. What does 'context window' primarily refer to in this situation?',
The context window is a crucial concept in LLMs. It represents the maximum amount of text—both user query and retrieved information—that the model can consider when generating a response. A smaller window limits the model's ability to effectively utilize relevant knowledge, leading to slower processing times. Option A is irrelevant; options C & D are related to training but not directly the definition.
11 / 11
Maria (Code Reviewer) comments on a PR description: 'This commit adds logging around prompt generation. We're now capturing the exact prompt string being sent to the LLM and its resulting token count. This will help us debug issues with prompt performance.' Which of the following best describes Maria's intention?
Maria is focused on understanding and improving the *performance* of the prompt. Tracking token counts directly relates to monitoring API usage costs – a key metric in LLMOps. While data privacy (A) and malicious input detection (D) are important considerations, they aren't the primary focus of this logging implementation.
What will I practise in "LLMOps Workflow Language — LLM App Development Exercises"?
Practice the English vocabulary of LLMOps: prompt versioning, model registry, LLM observability, A/B testing prompts, and quality drift monitoring. 5 advanced exercises.
How many exercises are in this module?
This module has 11 multiple-choice exercises, each with instant feedback and a full explanation of the correct answer.
Is this exercise free to use?
Yes. Every exercise on CoderSlingo, including this one, is free to use with no account, sign-up, or paywall.
Do I need to create an account to do these exercises?
No account is required. Just click an option to answer — your score for this session is tracked automatically in the progress bar above.
What happens if I choose the wrong answer?
You'll immediately see which answer was correct, plus a full explanation covering the vocabulary and reasoning behind it — mistakes are where most of the learning happens.
Can I retry the exercises if I want a higher score?
Yes — use the "Try again" button on the results screen to reset and go through all the questions again.
Is my progress saved if I close the page?
No. Progress is tracked only for your current visit; reloading or leaving the page resets the counter. This keeps the exercise simple and account-free.
Where can I find more LLM App Development exercises?
Browse the full LLM App Development hub for related drills, or check the "Next up" link below to continue with a connected topic.
How is this different from reading an article on the same topic?
Articles explain vocabulary and concepts in prose; this exercise tests and reinforces that vocabulary through active recall with immediate feedback — the two work best together.
Who writes these exercises?
Every exercise is written by the CoderSlingo team, drawing on real workplace English used in IT roles, then reviewed for accuracy and clarity.