Practise vocabulary for ML serving topologies: online vs. batch inference, model gateways, streaming inference, and hardware choices.
0 / 10 completed
1 / 10
___ inference serves predictions synchronously per request with low latency — the model must respond within milliseconds.
Online inference (real-time serving) serves predictions in real time, request by request. Each call waits for a response — latency is critical. Used in search ranking, recommendation, fraud detection.
2 / 10
___ inference runs predictions on large datasets asynchronously, typically on a schedule, without latency constraints.
Batch inference scores large datasets (e.g., nightly predictions for all users) in bulk. It's CPU/GPU-efficient but not suitable for real-time use cases. Results are stored for later consumption.
3 / 10
A ___ acts as a single entry point for inference requests, routing to the correct model version and managing authentication and rate limiting.
A model gateway (inference gateway) routes requests to model endpoints, handles versioning, A/B routing, rate limiting, and authentication. It decouples clients from specific model instances.
4 / 10
Running the new model in ___ mode means it receives production traffic and makes predictions, but its outputs are discarded — not shown to users.
Shadow mode mirrors production traffic to the new model without affecting users. It allows safe validation of model behaviour, latency, and output distribution before cutover.
5 / 10
An ___ is specialised hardware (GPU, TPU, inference chip) used to accelerate model computations during serving.
Accelerators (GPUs, TPUs, AWS Inferentia, NVIDIA A100) dramatically reduce inference latency for deep learning models. Choosing the right accelerator is a key serving architecture decision.
6 / 10
Reviewer: 'The PR description mentions 'batch inference' but doesn't explain the trade-offs. It just says it's used for high throughput. Can you elaborate on how this impacts latency and resource utilization compared to online serving?', What is the reviewer most concerned about?
The reviewer is rightly concerned about the *trade-offs* inherent in batch inference. While it can achieve high throughput, this often comes at the expense of increased latency – a key factor when comparing to online serving where low latency is crucial. Batch inference also introduces complexities around resource utilization that need careful consideration.
7 / 10
Dev A (in Slack): 'Just deploying the new model to staging with asynchronous batch inference. We're using Kafka for data ingestion and a Celery worker to process the batches. Monitoring queue length is key.' What's Dev A primarily focused on?
Dev A is emphasizing the *data pipeline* and its efficiency. Asynchronous batch inference relies heavily on a robust and responsive data ingestion system (Kafka) and a worker process (Celery) to handle the large datasets. Monitoring queue length directly relates to the performance of this processing chain – it's a vital indicator.
8 / 10
Which of the following best describes the significance of the 'latency' field in this response?
The 'latency' field in this API response specifically refers to the *time taken* by the inference server to process the request and return the prediction. While it's important for performance, it doesn't directly relate to training or data quality – those are measured through different metrics.
9 / 10
What is the primary purpose of deploying in 'shadow mode'?
Shadow deployment in 'shadow mode' is a crucial step for *risk mitigation*. It enables you to observe and evaluate the new model's predictions against live production traffic without exposing those predictions to actual users. This allows thorough validation before any changes are made visible.
10 / 10
Dev B (in a stand-up): 'We're using NVIDIA TensorRT to accelerate inference for the fraud detection model. It's significantly reduced prediction latency and allowed us to handle a much higher request volume.' What technology is Dev B primarily referring to?
Dev B's update highlights the use of NVIDIA TensorRT, which is a *specialized hardware* and software library designed to accelerate deep learning inference. It's focused on optimizing the computational graph for speed, allowing the model to process requests faster and handle more traffic – not a general-purpose orchestration or scaling solution.
What will I practise in "ML Model Serving Architectures — Vocabulary"?
This module focuses on ML Model Serving — real workplace phrasing you'll use on the job. It contains 10 scenario-based multiple-choice questions with instant feedback.
Is this exercise free to use?
Yes. Every exercise on CoderSlingo, including this one, is free to use with no account or sign-up required.
How many questions does this exercise have?
This module includes 10 questions. Each one gives an immediate right/wrong result plus a full explanation of the correct phrasing.
What happens if I answer a question incorrectly?
You'll see the correct answer highlighted straight away, along with a plain-English explanation of why it's right and why the other options don't fit — mistakes are part of the learning here.
Can I retry the exercise if I want a better score?
Yes — use the 'Try again' button on the results screen to reset your score and go through the questions again. There's no limit on attempts.
Who is this ML Model Serving exercise for?
It's aimed at IT professionals with working English who want to sound more natural and precise around ml model serving — useful whether you're preparing for real conversations at work or just building confidence with the vocabulary.
Do I need an account to track my progress?
No account is needed. Your progress through the exercise is tracked locally in your browser for the current session, and you can replay the module at any time.
How is this different from reading a blog article?
This exercise is an interactive drill that tests and reinforces specific phrasing through multiple-choice questions with instant feedback, while blog articles explain concepts and vocabulary in prose. The two work well together.
Where can I find more ML Model Serving exercises?
See the ML Model Serving hub for more modules like this one, or browse the full Exercises page for other IT-English topics.
Can I complete this exercise on my phone?
Yes — every exercise on CoderSlingo is fully responsive and works on phones and tablets, so you can practise anywhere.