LLM Deployment Vocabulary
5 exercises — inference servers and vLLM, quantisation tradeoffs, ONNX interoperability, latency metrics (TTFT vs. total generation time), and continuous batching.
0 / 26 completed
1 / 26
An infrastructure ticket says: "We're moving from calling a hosted API to running our own inference server with vLLM." What is an inference server, and why might a team choose to self-host one?
Understanding the distinction between training and inference is foundational: training is the (extremely expensive) process of teaching a model from data; inference is the (comparatively cheap per-request, but still resource-intensive) process of using an already-trained model to generate a response to a given input.
vLLM is specifically engineered to make inference serving efficient at scale, using techniques like continuous batching (dynamically grouping multiple requests together to maximise GPU utilisation, rather than processing one request fully before starting the next) and PagedAttention (an efficient memory management technique for the attention mechanism's key-value cache, reducing wasted GPU memory).
Why self-host instead of calling a hosted API (like OpenAI's or Anthropic's): data never needs to leave your own infrastructure (important for regulated industries), you can use open-weight models with no per-token vendor pricing, and you gain full control over latency, hardware allocation, and model version — at the cost of needing to manage your own GPU infrastructure, scaling, and reliability.
vLLM is specifically engineered to make inference serving efficient at scale, using techniques like continuous batching (dynamically grouping multiple requests together to maximise GPU utilisation, rather than processing one request fully before starting the next) and PagedAttention (an efficient memory management technique for the attention mechanism's key-value cache, reducing wasted GPU memory).
Why self-host instead of calling a hosted API (like OpenAI's or Anthropic's): data never needs to leave your own infrastructure (important for regulated industries), you can use open-weight models with no per-token vendor pricing, and you gain full control over latency, hardware allocation, and model version — at the cost of needing to manage your own GPU infrastructure, scaling, and reliability.
2 / 26
A model deployment plan mentions "quantising the model to 4-bit before deployment to reduce memory usage." What does quantisation mean, and what's the tradeoff?
Quantisation is one of the most impactful techniques for making large models practically deployable on limited hardware. A model's weights are normally stored as floating-point numbers (commonly 16-bit or 32-bit); quantising to, say, 4-bit integers can reduce the model's memory footprint by roughly 4x compared to 16-bit, which can be the difference between a model fitting on a single consumer GPU versus requiring an expensive multi-GPU server.
The precision-vs-quality tradeoff: naively rounding every weight to lower precision would meaningfully degrade output quality, but modern quantisation techniques (like GPTQ, AWQ, or the GGUF format's various quantisation levels) are specifically designed to minimise quality loss — often keeping degradation small enough to be acceptable for many use cases, though this should always be evaluated empirically for a specific task rather than assumed.
Common vocabulary in quantisation discussions: Q4, Q8 (common shorthand for bit-width in tools like llama.cpp), post-training quantisation (applied after training completes, the most common approach) versus quantisation-aware training (accounting for quantisation during the training process itself, for even better quality retention).
The precision-vs-quality tradeoff: naively rounding every weight to lower precision would meaningfully degrade output quality, but modern quantisation techniques (like GPTQ, AWQ, or the GGUF format's various quantisation levels) are specifically designed to minimise quality loss — often keeping degradation small enough to be acceptable for many use cases, though this should always be evaluated empirically for a specific task rather than assumed.
Common vocabulary in quantisation discussions: Q4, Q8 (common shorthand for bit-width in tools like llama.cpp), post-training quantisation (applied after training completes, the most common approach) versus quantisation-aware training (accounting for quantisation during the training process itself, for even better quality retention).
3 / 26
A team discusses converting a model to "ONNX format for deployment across different hardware backends." What problem does a format like ONNX solve?
The ML ecosystem has many training frameworks (PyTorch, TensorFlow, JAX) and many deployment targets (different cloud providers, edge devices, mobile phones, specialised inference chips) — without a common interchange format, deploying a model to a new target often meant rewriting significant portions of the inference code for that framework/hardware combination.
ONNX addresses this by defining a common, portable representation of a model's computation graph and weights. Once a model is exported to ONNX, it can be run by any ONNX-compatible runtime — software optimised to execute that graph efficiently on a specific hardware target — without depending on the original training framework being present.
Practical benefit in deployment conversations: being able to say "we exported to ONNX so we're not locked into PyTorch's runtime, and we can target whichever hardware backend is most cost-effective" signals an understanding of production ML infrastructure decisions that goes beyond just training a model — deployment format and runtime choice are separate, consequential engineering decisions from model architecture choice.
ONNX addresses this by defining a common, portable representation of a model's computation graph and weights. Once a model is exported to ONNX, it can be run by any ONNX-compatible runtime — software optimised to execute that graph efficiently on a specific hardware target — without depending on the original training framework being present.
Practical benefit in deployment conversations: being able to say "we exported to ONNX so we're not locked into PyTorch's runtime, and we can target whichever hardware backend is most cost-effective" signals an understanding of production ML infrastructure decisions that goes beyond just training a model — deployment format and runtime choice are separate, consequential engineering decisions from model architecture choice.
4 / 26
A performance review of an LLM API notes: "Latency to first token is 200ms, but total generation time for a long response is 8 seconds." How would you explain the difference between these two latency metrics, and why both matter?
Distinguishing these two latency metrics precisely matters because they're driven by different technical factors and matter for different user experiences.
Time to First Token (TTFT) — dominated by the initial "prefill" phase, where the model processes the entire input prompt before generating any output. A longer input prompt increases TTFT. This is the metric that most directly affects perceived responsiveness in an interactive chat UI, since users see the response start appearing.
Total generation time (and the related "tokens per second" throughput metric) — dominated by the "decode" phase, generating each subsequent output token one at a time. This scales roughly with the length of the response being generated, and matters most when the complete response is needed before the user (or a downstream system) can act on it.
Why both matter in a performance report: a chat product optimising for user experience might prioritise TTFT (get something on screen fast, then stream the rest), while a backend pipeline generating a structured JSON output that must be fully parsed before use cares primarily about total generation time — precise vocabulary here prevents a team from optimising the wrong metric for their actual use case.
Time to First Token (TTFT) — dominated by the initial "prefill" phase, where the model processes the entire input prompt before generating any output. A longer input prompt increases TTFT. This is the metric that most directly affects perceived responsiveness in an interactive chat UI, since users see the response start appearing.
Total generation time (and the related "tokens per second" throughput metric) — dominated by the "decode" phase, generating each subsequent output token one at a time. This scales roughly with the length of the response being generated, and matters most when the complete response is needed before the user (or a downstream system) can act on it.
Why both matter in a performance report: a chat product optimising for user experience might prioritise TTFT (get something on screen fast, then stream the rest), while a backend pipeline generating a structured JSON output that must be fully parsed before use cares primarily about total generation time — precise vocabulary here prevents a team from optimising the wrong metric for their actual use case.
5 / 26
A cost-optimisation discussion mentions "continuous batching improved our GPU utilisation and reduced cost per request by 40%." What is continuous batching, and why does it improve efficiency compared to processing one request at a time?
GPUs achieve their computational efficiency through parallelism — they're most efficient when processing many similar computations simultaneously, which is exactly what serving multiple LLM requests together (a "batch") enables, compared to processing each user's request one-by-one, which leaves most of the GPU's capacity idle.
The problem with naive (static) batching: if you wait to collect a fixed batch of, say, 8 requests, then process all 8 together — but responses vary wildly in length (one user asks a yes/no question, another requests a long essay) — the entire batch is stuck waiting for the longest response to finish before the GPU can start a new batch, wasting significant capacity once shorter requests in the batch have already completed.
Continuous batching solves this by treating the batch as dynamic: as soon as any request in the current batch finishes generating, a new waiting request can immediately take its place in the batch, keeping the GPU's batch size — and therefore its utilisation — consistently high, rather than fluctuating between full and mostly-idle. This is one of the core techniques (alongside quantisation and efficient memory management like PagedAttention) that modern inference servers like vLLM use to dramatically improve cost-per-request at scale.
The problem with naive (static) batching: if you wait to collect a fixed batch of, say, 8 requests, then process all 8 together — but responses vary wildly in length (one user asks a yes/no question, another requests a long essay) — the entire batch is stuck waiting for the longest response to finish before the GPU can start a new batch, wasting significant capacity once shorter requests in the batch have already completed.
Continuous batching solves this by treating the batch as dynamic: as soon as any request in the current batch finishes generating, a new waiting request can immediately take its place in the batch, keeping the GPU's batch size — and therefore its utilisation — consistently high, rather than fluctuating between full and mostly-idle. This is one of the core techniques (alongside quantisation and efficient memory management like PagedAttention) that modern inference servers like vLLM use to dramatically improve cost-per-request at scale.
6 / 26
Mark is referring to a 'system prompt,' which is a crucial technique in LLM prompting. It's a dedicated piece of text provided as context *before* any user input, defining the model's role, style, and constraints. This helps steer the generated responses towards desired formats and reduces ambiguity—a common misconception is that system prompts are just part of the regular prompt; they establish the foundational behavior for the entire interaction.
7 / 26
PR Description
Subject: Initial Deployment of Gemini-Pro Model – Version 1.0
This PR deploys the Gemini-Pro model (v1.0) to production using a managed service. We've configured prompt templates for common use cases and implemented basic logging.
Which of the following best describes the *purpose* of the 'prompt templates' mentioned in this PR description? A. To automatically scale the model deployment based on incoming requests. B. To define a standardized format for user input, ensuring consistency and facilitating prompt engineering. C. To directly control the model's parameters during inference, allowing for fine-grained adjustments. D. To encrypt all prompts before sending them to the model, enhancing security.
The correct answer is B. Prompt templates are reusable sets of instructions and examples given to the LLM as part of a prompt. They ensure that user input is consistently formatted for the model, which is crucial for effective prompt engineering – providing clear context and structure to the AI. Options A, C, and D describe unrelated functionalities like autoscaling, parameter control, and encryption, respectively. The PR focuses on establishing a consistent *input* format.
8 / 26
During a standup update, the AI deployment engineer says: 'We're using 'prompt chaining' to break down complex user queries into smaller steps for the LLM. We then combine the outputs of each step to generate the final response.' What does 'prompt chaining' refer to in this context?
'Prompt chaining' describes a technique where a complex task is broken down into a series of simpler prompts. Each prompt focuses on a specific part of the problem, and the results are combined to create a complete answer – this mirrors the engineer's explanation. This approach is particularly useful for LLMs when dealing with intricate requests that require multiple reasoning steps; option A is translation, C is a single complex prompt, and D refers to system prompts.
9 / 26
Reviewing the PR description for the Gemini-Pro model deployment, the team mentions 'prompt templates'. Considering the context of deploying an LLM and common practices, which of the following best explains their role?
The correct answer is B. 'Prompt templates' in this scenario define a structured format for user prompts – this is crucial for prompt engineering. Incorrect options misinterpret their role; they don't control training parameters (A), GPU allocation (C), or secure data transmission (D). Using standardized prompts allows developers to experiment with techniques like few-shot learning more effectively.
10 / 26
Mark is referring to a 'system prompt,' which is a crucial technique in LLM prompting. It's a dedicated piece of text provided as context *before* any user input, defining the model's role, style, and constraints. This helps steer the generated responses towards desired formats and reduces ambiguity—a common misconception is that system prompts are just part of the regular prompt; they establish the foundational behavior for the entire interaction.
11 / 26
PR Description
Subject: Initial Deployment of Gemini-Pro Model – Version 1.0
This PR deploys the Gemini-Pro model (v1.0) to production using a managed service. We've configured prompt templates for common use cases and implemented basic logging.
Which of the following best describes the *purpose* of the 'prompt templates' mentioned in this PR description? A. To automatically scale the model deployment based on incoming requests. B. To define a standardized format for user input, ensuring consistency and facilitating prompt engineering. C. To directly control the model's parameters during inference, allowing for fine-grained adjustments. D. To encrypt all prompts before sending them to the model, enhancing security.
The correct answer is B. Prompt templates are reusable sets of instructions and examples given to the LLM as part of a prompt. They ensure that user input is consistently formatted for the model, which is crucial for effective prompt engineering – providing clear context and structure to the AI. Options A, C, and D describe unrelated functionalities like autoscaling, parameter control, and encryption, respectively. The PR focuses on establishing a consistent *input* format.
12 / 26
During a standup update, the AI deployment engineer says: 'We're using 'prompt chaining' to break down complex user queries into smaller steps for the LLM. We then combine the outputs of each step to generate the final response.' What does 'prompt chaining' refer to in this context?
'Prompt chaining' describes a technique where a complex task is broken down into a series of simpler prompts. Each prompt focuses on a specific part of the problem, and the results are combined to create a complete answer – this mirrors the engineer's explanation. This approach is particularly useful for LLMs when dealing with intricate requests that require multiple reasoning steps; option A is translation, C is a single complex prompt, and D refers to system prompts.
13 / 26
Reviewing the PR description for the Gemini-Pro model deployment, the team mentions 'prompt templates'. Considering the context of deploying an LLM and common practices, which of the following best explains their role?
The correct answer is B. 'Prompt templates' in this scenario define a structured format for user prompts – this is crucial for prompt engineering. Incorrect options misinterpret their role; they don't control training parameters (A), GPU allocation (C), or secure data transmission (D). Using standardized prompts allows developers to experiment with techniques like few-shot learning more effectively.
14 / 26
Mark is referring to a 'system prompt,' which is a crucial technique in LLM prompting. It's a dedicated piece of text provided as context *before* any user input, defining the model's role, style, and constraints. This helps steer the generated responses towards desired formats and reduces ambiguity—a common misconception is that system prompts are just part of the regular prompt; they establish the foundational behavior for the entire interaction.
15 / 26
PR Description
Subject: Initial Deployment of Gemini-Pro Model – Version 1.0
This PR deploys the Gemini-Pro model (v1.0) to production using a managed service. We've configured prompt templates for common use cases and implemented basic logging.
Which of the following best describes the *purpose* of the 'prompt templates' mentioned in this PR description? A. To automatically scale the model deployment based on incoming requests. B. To define a standardized format for user input, ensuring consistency and facilitating prompt engineering. C. To directly control the model's parameters during inference, allowing for fine-grained adjustments. D. To encrypt all prompts before sending them to the model, enhancing security.
The correct answer is B. Prompt templates are reusable sets of instructions and examples given to the LLM as part of a prompt. They ensure that user input is consistently formatted for the model, which is crucial for effective prompt engineering – providing clear context and structure to the AI. Options A, C, and D describe unrelated functionalities like autoscaling, parameter control, and encryption, respectively. The PR focuses on establishing a consistent *input* format.
16 / 26
During a standup update, the AI deployment engineer says: 'We're using 'prompt chaining' to break down complex user queries into smaller steps for the LLM. We then combine the outputs of each step to generate the final response.' What does 'prompt chaining' refer to in this context?
'Prompt chaining' describes a technique where a complex task is broken down into a series of simpler prompts. Each prompt focuses on a specific part of the problem, and the results are combined to create a complete answer – this mirrors the engineer's explanation. This approach is particularly useful for LLMs when dealing with intricate requests that require multiple reasoning steps; option A is translation, C is a single complex prompt, and D refers to system prompts.
17 / 26
Reviewing the PR description for the Gemini-Pro model deployment, the team mentions 'prompt templates'. Considering the context of deploying an LLM and common practices, which of the following best explains their role?
The correct answer is B. 'Prompt templates' in this scenario define a structured format for user prompts – this is crucial for prompt engineering. Incorrect options misinterpret their role; they don't control training parameters (A), GPU allocation (C), or secure data transmission (D). Using standardized prompts allows developers to experiment with techniques like few-shot learning more effectively.
18 / 26
Mark is referring to a 'system prompt,' which is a crucial technique in LLM prompting. It's a dedicated piece of text provided as context *before* any user input, defining the model's role, style, and constraints. This helps steer the generated responses towards desired formats and reduces ambiguity—a common misconception is that system prompts are just part of the regular prompt; they establish the foundational behavior for the entire interaction.
19 / 26
PR Description
Subject: Initial Deployment of Gemini-Pro Model – Version 1.0
This PR deploys the Gemini-Pro model (v1.0) to production using a managed service. We've configured prompt templates for common use cases and implemented basic logging.
Which of the following best describes the *purpose* of the 'prompt templates' mentioned in this PR description? A. To automatically scale the model deployment based on incoming requests. B. To define a standardized format for user input, ensuring consistency and facilitating prompt engineering. C. To directly control the model's parameters during inference, allowing for fine-grained adjustments. D. To encrypt all prompts before sending them to the model, enhancing security.
The correct answer is B. Prompt templates are reusable sets of instructions and examples given to the LLM as part of a prompt. They ensure that user input is consistently formatted for the model, which is crucial for effective prompt engineering – providing clear context and structure to the AI. Options A, C, and D describe unrelated functionalities like autoscaling, parameter control, and encryption, respectively. The PR focuses on establishing a consistent *input* format.
20 / 26
During a standup update, the AI deployment engineer says: 'We're using 'prompt chaining' to break down complex user queries into smaller steps for the LLM. We then combine the outputs of each step to generate the final response.' What does 'prompt chaining' refer to in this context?
'Prompt chaining' describes a technique where a complex task is broken down into a series of simpler prompts. Each prompt focuses on a specific part of the problem, and the results are combined to create a complete answer – this mirrors the engineer's explanation. This approach is particularly useful for LLMs when dealing with intricate requests that require multiple reasoning steps; option A is translation, C is a single complex prompt, and D refers to system prompts.
21 / 26
Reviewing the PR description for the Gemini-Pro model deployment, the team mentions 'prompt templates'. Considering the context of deploying an LLM and common practices, which of the following best explains their role?
The correct answer is B. 'Prompt templates' in this scenario define a structured format for user prompts – this is crucial for prompt engineering. Incorrect options misinterpret their role; they don't control training parameters (A), GPU allocation (C), or secure data transmission (D). Using standardized prompts allows developers to experiment with techniques like few-shot learning more effectively.
22 / 26
Reviewer Alex comments on a PR draft: 'The prompt for this summarization task is too vague. It needs more context about the document's purpose and desired length.' Considering this feedback, what does 'prompt context' primarily refer to in LLM deployment?
'Prompt context' in this scenario refers to the supplementary details included within the prompt that help the LLM understand the task and generate a relevant output. It's about providing enough information for the model to accurately interpret the query – options A, C, and D relate to technical infrastructure or documentation, not the content *within* the prompt itself.
23 / 26
During a Slack channel discussion about debugging an LLM-powered customer support bot, David writes: 'I'm seeing inconsistent results when using zero-shot prompting. I'm going to try adding a few example conversations into the prompt – that might improve its understanding.' What technique is David primarily employing here?
David is utilizing 'few-shot learning' by including example conversations in the prompt. This technique leverages past interactions to provide the LLM with a better understanding of the desired behavior and expected output format, directly influencing its response – options C and D represent different prompting strategies focusing on reasoning or external data retrieval.
24 / 26
The API response from our LLM deployment service for a user query about 'quantum physics' is: `{"model_confidence": 0.75, "generated_text": 'Quantum physics deals with the behavior of matter and energy at the atomic and subatomic levels.'}`. Based on this response, what does the 'model confidence' score primarily indicate?
The 'model confidence' score represents the LLM's self-assessment of its own output. It's a probabilistic measure indicating how likely the model believes its response is accurate and relevant to the user query – options A, C, and D are related to different metrics or processes involved in generating the response.
25 / 26
A developer submits a PR with the following description: 'This update fine-tunes the LLM's responses for product support queries. We've incorporated 'prompt templates' to ensure consistency across all interactions.' What is the *primary* purpose of 'prompt templates' in this context?
'Prompt templates' are designed to enforce consistency by establishing a predetermined format and content for prompts. This ensures that the LLM receives consistent instructions, leading to predictable and standardized responses – options A, C, and D represent alternative uses of structured prompts but don't capture their core function.
26 / 26
During a daily stand-up, the AI deployment engineer says: 'We're implementing 'chain-of-thought prompting' to help the LLM break down complex user requests into smaller, more manageable steps. This allows us to improve accuracy and reduce hallucinations.' What does 'chain-of-thought prompting' primarily aim to achieve?
'Chain-of-thought prompting' focuses on structuring the prompt to encourage the LLM to reason step-by-step. This mimics human problem-solving by guiding the model through a logical process – option A is incorrect as it suggests bypassing reasoning entirely; options C and D relate to efficiency or error correction, not the core technique itself.
Frequently Asked Questions
What will I practice in "LLM Deployment Vocabulary — AI Prompting English Exercise"?
This is an AI Prompting exercise set. It walks through 26 scenario-based multiple-choice questions built around real usage of AI Prompting terminology that IT professionals encounter on the job.
Is this exercise free to use?
Yes. Every exercise on CoderSlingo, including this one, is free to complete with no account, sign-up, or paywall.
How many questions are in this exercise?
This set contains 26 questions. Each one shows immediate feedback and a detailed explanation after you answer, so you learn the correct usage right away rather than waiting for a final score.
Do I need prior experience to complete this exercise?
No prior experience is required. Each question includes a full explanation covering the reasoning behind the correct answer, so the exercise itself teaches the AI Prompting vocabulary as you go.
Can I retry the exercise if I get questions wrong?
Yes — use the "Try again" button on the results screen to reset your answers and go through all the questions again. There is no limit on attempts.
Is my progress saved?
Your answers and score for the current session are tracked in the browser as you go. No account or login is needed, and there is nothing to install.
What if I don't understand a term used in a question?
Read the explanation shown after you answer each question — it breaks down the correct term in plain English with a real-world example. You can also check the site Glossary for quick definitions.
How is this different from reading a blog article on the topic?
Exercises like this one are interactive drills that test and reinforce specific vocabulary through multiple-choice questions, while blog articles explain concepts in prose. Practising here after reading builds active recall, not just passive recognition.
Where can I find more AI Prompting exercises?
See the AI Prompting exercises hub for the full set of related pages, or browse all exercise categories from the main Exercises index.
Can I use this exercise to prepare for a technical interview?
Yes — AI Prompting vocabulary comes up often in technical discussions and interviews. Pair this exercise with our dedicated Interview Preparation section for role-specific practice.