Model Serving

Definition

Deploying a trained ML model as an API endpoint so applications can request predictions in real or batch time.

Example in context

"The recommendation model is wrapped in a FastAPI server — the product service calls /predict with user context."

Related terms

• Inference — The process of running new input data through a trained model to generate predictions.
• MLOps — Machine Learning Operations — combining ML development with DevOps practices for reliable model deployment and monitoring.
• Latency — The time delay between sending a request and receiving the first byte of a response, measured in milliseconds.