ML Model Serving & Inference Language Exercises
Serving architectures, model registries, inference optimisation, drift monitoring, shadow deployments, and explainability — production ML vocabulary.
- Model Serving Architectures — Vocabulary
- Model Registry & Versioning Language
- Inference Optimisation Vocabulary
- Model Monitoring & Drift Vocabulary
- Shadow & Canary Model Deployment Language
- Model Explainability Language
- ML Pipeline Orchestration Vocabulary
Frequently Asked Questions
What is ML model serving and how is it described by engineers?
ML model serving (also called model inference serving) is the process of deploying a trained machine learning model behind an API endpoint so that applications can send inputs and receive predictions in real time or in batch. Engineers discuss it in terms of "serving infrastructure", "prediction latency SLAs", "throughput requirements", and "model endpoint management".
What is the difference between online inference and batch inference?
Online (real-time) inference serves predictions synchronously in response to individual requests, typically with a latency requirement measured in milliseconds. Batch inference processes large volumes of inputs offline on a schedule, prioritising throughput over latency. Engineers choose between them based on "user-facing latency budgets", "data freshness requirements", and "infrastructure cost constraints".
How do ML engineers describe latency and throughput trade-offs?
Latency is the time taken to return a single prediction, while throughput is the number of predictions served per second. Engineers discuss "p50, p95, and p99 latency percentiles", "request batching to improve GPU utilisation", "horizontal scaling to increase throughput", and "the latency-throughput trade-off" when batching increases throughput at the cost of per-request latency.
What is a model registry and why is it important?
A model registry is a centralised catalogue that stores trained model artefacts alongside their metadata — version, training dataset, evaluation metrics, and approval status. Engineers "register a model", "promote a model from staging to production", "roll back to a previous version", and "compare model versions by metric" using registry tooling such as MLflow, Vertex AI Model Registry, or SageMaker Model Registry.
How is A/B testing of models described in production ML language?
A/B testing in ML serving means routing a fraction of live traffic to a challenger model while the incumbent handles the remainder, then comparing business and model metrics across cohorts. Engineers describe "traffic splitting", "shadow mode" (where the challenger runs without serving its predictions to users), "statistical significance of metric differences", and "promotion criteria" for the winning variant.
What does "model drift" mean and how is it monitored?
Model drift refers to degradation in model performance over time, caused by changes in the distribution of incoming data (data drift) or shifts in the relationship between inputs and targets (concept drift). Teams monitor for drift by "tracking prediction distribution", "comparing feature statistics against a training baseline", and "setting alert thresholds on performance metrics" using tools like Evidently, WhyLogs, or cloud-native monitoring suites.
What vocabulary surrounds inference optimisation techniques?
Common optimisation vocabulary includes "model quantisation" (reducing weight precision from FP32 to INT8), "pruning" (removing low-importance weights), "knowledge distillation" (training a smaller student model to mimic a larger teacher), "TensorRT compilation", "ONNX export", and "dynamic batching". Engineers balance "accuracy-latency trade-offs" when applying these techniques.
What is a shadow deployment in ML serving?
A shadow deployment (or shadow mode) sends a copy of live production traffic to a new model version without returning its predictions to end users, allowing the team to compare outputs and measure performance against the production model risk-free. Engineers "enable shadow mode", "compare shadow vs. production prediction distributions", and "validate latency under real traffic" before promoting the new model.
How do engineers discuss serving infrastructure choices?
Infrastructure discussions cover "REST vs. gRPC endpoints", "GPU vs. CPU serving costs", "Kubernetes-based serving with KServe or Triton Inference Server", "serverless inference for sporadic workloads", and "auto-scaling policies based on queue depth or GPU utilisation". Engineers weigh "cold start latency", "infrastructure overhead", and "cost per million predictions" when selecting a serving platform.
Why is ML serving vocabulary important for data scientists transitioning to production roles?
Data scientists who move into MLOps or production ML engineering roles must communicate clearly with platform engineers, DevOps teams, and product managers about deployment strategies, reliability requirements, and performance trade-offs. Mastering serving vocabulary — from canary deployments to inference optimisation — enables more effective collaboration and ensures that model quality translates into reliable, high-performance production systems.