Practise vocabulary for scaling ML inference: GPU utilisation, dynamic batching, autoscaling, cold starts, and throughput vs latency trade-offs.
0 / 10 completed
1 / 10
Grouping several incoming requests so they run together on the GPU in a single forward pass is called ___.
Dynamic batching waits a few milliseconds to collect multiple requests and process them as one batch, raising GPU throughput at a small latency cost.
2 / 10
The fraction of time the GPU is actively computing rather than idle is its ___.
GPU utilisation measures how busy the device is; low utilisation under load usually signals a CPU/IO bottleneck or poor batching.
3 / 10
The delay before a freshly scaled-up replica can serve traffic because it must load the model into GPU memory is the ___.
Cold start latency comes from spinning up the container and loading large model weights; keeping warm replicas avoids it.
4 / 10
Automatically adding or removing inference replicas based on request load is ___.
Autoscaling adjusts the replica count (often on GPU utilisation or queue depth) so you pay for capacity roughly proportional to demand.
5 / 10
Increasing batch size usually improves ___ but can worsen per-request ___.
Larger batches amortise overhead and lift throughput, but individual requests wait longer to be batched and processed, raising latency.
6 / 10
Code Review Comment: 'I'm seeing some performance bottlenecks on the inference endpoint. The team mentioned using batching to improve throughput. What does 'batching' typically refer to in this context?'
Batching in this scenario refers to combining multiple independent requests into a single batch before processing them on the GPU. This significantly reduces overhead compared to handling each request individually, leveraging the parallel processing capabilities of the GPU. Option A is about model adaptation; C is database optimization; and D describes caching.
7 / 10
Slack Message: 'Hey team, we're scaling the image recognition API. I've been monitoring GPU utilization and it's hovering around 65%. What metric is most directly related to that?'
GPU Utilization represents the proportion of time a GPU is actively performing computations rather than being idle. Monitoring this metric provides insight into how effectively the GPU resources are being utilized for inference tasks, which directly impacts performance. Latency measures response time; throughput measures request volume; and memory footprint describes resource consumption.
8 / 10
PR Description: 'Implemented a new autoscaling policy for the object detection model. The system now automatically adjusts the number of GPU replicas based on incoming request volume. This is achieved through dynamic scaling.'
Dynamic scaling describes the automated process of adjusting the number of GPU replicas in response to fluctuating demand. This is crucial for handling peak loads efficiently and minimizing costs when demand decreases. The load balancer distributes requests; manual intervention is not part of this described policy; and circuit breakers prevent failures.
9 / 10
Standup Update: 'We've been experimenting with increasing the batch size for our fraud detection model. Initially, we observed a significant improvement in throughput. However, we're now seeing an increase in per-request latency.'
Increasing batch size typically boosts throughput by processing multiple requests in one go, leveraging GPU parallelism. However, it can also increase per-request latency because the model needs to be executed for each item within that larger batch. This represents a classic trade-off in scaling inference systems.
The `replica_id` within the API response directly identifies the GPU instance that was assigned to process the specific inference request. This information is key for monitoring resource utilization and understanding how scaling impacts performance at a granular level. Loading time isn't provided; batching isn't mentioned; and `request_id` is simply an identifier.
What will I practise in "GPU & Inference Scaling Vocabulary"?
This module focuses on ML Model Serving — real workplace phrasing you'll use on the job. It contains 10 scenario-based multiple-choice questions with instant feedback.
Is this exercise free to use?
Yes. Every exercise on CoderSlingo, including this one, is free to use with no account or sign-up required.
How many questions does this exercise have?
This module includes 10 questions. Each one gives an immediate right/wrong result plus a full explanation of the correct phrasing.
What happens if I answer a question incorrectly?
You'll see the correct answer highlighted straight away, along with a plain-English explanation of why it's right and why the other options don't fit — mistakes are part of the learning here.
Can I retry the exercise if I want a better score?
Yes — use the 'Try again' button on the results screen to reset your score and go through the questions again. There's no limit on attempts.
Who is this ML Model Serving exercise for?
It's aimed at IT professionals with working English who want to sound more natural and precise around ml model serving — useful whether you're preparing for real conversations at work or just building confidence with the vocabulary.
Do I need an account to track my progress?
No account is needed. Your progress through the exercise is tracked locally in your browser for the current session, and you can replay the module at any time.
How is this different from reading a blog article?
This exercise is an interactive drill that tests and reinforces specific phrasing through multiple-choice questions with instant feedback, while blog articles explain concepts and vocabulary in prose. The two work well together.
Where can I find more ML Model Serving exercises?
See the ML Model Serving hub for more modules like this one, or browse the full Exercises page for other IT-English topics.
Can I complete this exercise on my phone?
Yes — every exercise on CoderSlingo is fully responsive and works on phones and tablets, so you can practise anywhere.