5 exercises — inference servers and vLLM, quantisation tradeoffs, ONNX interoperability, latency metrics (TTFT vs. total generation time), and continuous batching.
0 / 5 completed
1 / 5
An infrastructure ticket says: "We're moving from calling a hosted API to running our own inference server with vLLM." What is an inference server, and why might a team choose to self-host one?
Understanding the distinction between training and inference is foundational: training is the (extremely expensive) process of teaching a model from data; inference is the (comparatively cheap per-request, but still resource-intensive) process of using an already-trained model to generate a response to a given input.
vLLM is specifically engineered to make inference serving efficient at scale, using techniques like continuous batching (dynamically grouping multiple requests together to maximise GPU utilisation, rather than processing one request fully before starting the next) and PagedAttention (an efficient memory management technique for the attention mechanism's key-value cache, reducing wasted GPU memory).
Why self-host instead of calling a hosted API (like OpenAI's or Anthropic's): data never needs to leave your own infrastructure (important for regulated industries), you can use open-weight models with no per-token vendor pricing, and you gain full control over latency, hardware allocation, and model version — at the cost of needing to manage your own GPU infrastructure, scaling, and reliability.
2 / 5
A model deployment plan mentions "quantising the model to 4-bit before deployment to reduce memory usage." What does quantisation mean, and what's the tradeoff?
Quantisation is one of the most impactful techniques for making large models practically deployable on limited hardware. A model's weights are normally stored as floating-point numbers (commonly 16-bit or 32-bit); quantising to, say, 4-bit integers can reduce the model's memory footprint by roughly 4x compared to 16-bit, which can be the difference between a model fitting on a single consumer GPU versus requiring an expensive multi-GPU server.
The precision-vs-quality tradeoff: naively rounding every weight to lower precision would meaningfully degrade output quality, but modern quantisation techniques (like GPTQ, AWQ, or the GGUF format's various quantisation levels) are specifically designed to minimise quality loss — often keeping degradation small enough to be acceptable for many use cases, though this should always be evaluated empirically for a specific task rather than assumed.
Common vocabulary in quantisation discussions:Q4, Q8 (common shorthand for bit-width in tools like llama.cpp), post-training quantisation (applied after training completes, the most common approach) versus quantisation-aware training (accounting for quantisation during the training process itself, for even better quality retention).
3 / 5
A team discusses converting a model to "ONNX format for deployment across different hardware backends." What problem does a format like ONNX solve?
The ML ecosystem has many training frameworks (PyTorch, TensorFlow, JAX) and many deployment targets (different cloud providers, edge devices, mobile phones, specialised inference chips) — without a common interchange format, deploying a model to a new target often meant rewriting significant portions of the inference code for that framework/hardware combination.
ONNX addresses this by defining a common, portable representation of a model's computation graph and weights. Once a model is exported to ONNX, it can be run by any ONNX-compatible runtime — software optimised to execute that graph efficiently on a specific hardware target — without depending on the original training framework being present.
Practical benefit in deployment conversations: being able to say "we exported to ONNX so we're not locked into PyTorch's runtime, and we can target whichever hardware backend is most cost-effective" signals an understanding of production ML infrastructure decisions that goes beyond just training a model — deployment format and runtime choice are separate, consequential engineering decisions from model architecture choice.
4 / 5
A performance review of an LLM API notes: "Latency to first token is 200ms, but total generation time for a long response is 8 seconds." How would you explain the difference between these two latency metrics, and why both matter?
Distinguishing these two latency metrics precisely matters because they're driven by different technical factors and matter for different user experiences.
Time to First Token (TTFT) — dominated by the initial "prefill" phase, where the model processes the entire input prompt before generating any output. A longer input prompt increases TTFT. This is the metric that most directly affects perceived responsiveness in an interactive chat UI, since users see the response start appearing.
Total generation time (and the related "tokens per second" throughput metric) — dominated by the "decode" phase, generating each subsequent output token one at a time. This scales roughly with the length of the response being generated, and matters most when the complete response is needed before the user (or a downstream system) can act on it.
Why both matter in a performance report: a chat product optimising for user experience might prioritise TTFT (get something on screen fast, then stream the rest), while a backend pipeline generating a structured JSON output that must be fully parsed before use cares primarily about total generation time — precise vocabulary here prevents a team from optimising the wrong metric for their actual use case.
5 / 5
A cost-optimisation discussion mentions "continuous batching improved our GPU utilisation and reduced cost per request by 40%." What is continuous batching, and why does it improve efficiency compared to processing one request at a time?
GPUs achieve their computational efficiency through parallelism — they're most efficient when processing many similar computations simultaneously, which is exactly what serving multiple LLM requests together (a "batch") enables, compared to processing each user's request one-by-one, which leaves most of the GPU's capacity idle.
The problem with naive (static) batching: if you wait to collect a fixed batch of, say, 8 requests, then process all 8 together — but responses vary wildly in length (one user asks a yes/no question, another requests a long essay) — the entire batch is stuck waiting for the longest response to finish before the GPU can start a new batch, wasting significant capacity once shorter requests in the batch have already completed.
Continuous batching solves this by treating the batch as dynamic: as soon as any request in the current batch finishes generating, a new waiting request can immediately take its place in the batch, keeping the GPU's batch size — and therefore its utilisation — consistently high, rather than fluctuating between full and mostly-idle. This is one of the core techniques (alongside quantisation and efficient memory management like PagedAttention) that modern inference servers like vLLM use to dramatically improve cost-per-request at scale.
What will I practice in "LLM Deployment Vocabulary — AI Prompting English Exercise"?
This is an AI Prompting exercise set. It walks through 5 scenario-based multiple-choice questions built around real usage of AI Prompting terminology that IT professionals encounter on the job.
Is this exercise free to use?
Yes. Every exercise on CoderSlingo, including this one, is free to complete with no account, sign-up, or paywall.
How many questions are in this exercise?
This set contains 5 questions. Each one shows immediate feedback and a detailed explanation after you answer, so you learn the correct usage right away rather than waiting for a final score.
Do I need prior experience to complete this exercise?
No prior experience is required. Each question includes a full explanation covering the reasoning behind the correct answer, so the exercise itself teaches the AI Prompting vocabulary as you go.
Can I retry the exercise if I get questions wrong?
Yes — use the "Try again" button on the results screen to reset your answers and go through all the questions again. There is no limit on attempts.
Is my progress saved?
Your answers and score for the current session are tracked in the browser as you go. No account or login is needed, and there is nothing to install.
What if I don't understand a term used in a question?
Read the explanation shown after you answer each question — it breaks down the correct term in plain English with a real-world example. You can also check the site Glossary for quick definitions.
How is this different from reading a blog article on the topic?
Exercises like this one are interactive drills that test and reinforce specific vocabulary through multiple-choice questions, while blog articles explain concepts in prose. Practising here after reading builds active recall, not just passive recognition.
Where can I find more AI Prompting exercises?
See the AI Prompting exercises hub for the full set of related pages, or browse all exercise categories from the main Exercises index.
Can I use this exercise to prepare for a technical interview?
Yes — AI Prompting vocabulary comes up often in technical discussions and interviews. Pair this exercise with our dedicated Interview Preparation section for role-specific practice.