English for BentoML Model Serving Developers

BentoML provides a unified framework for packaging, serving, and deploying machine learning models at production scale. As MLOps practices mature, engineers and data scientists need precise English vocabulary to discuss BentoML’s architecture in design reviews, handoff documents, and cross-team technical discussions with platform and infrastructure colleagues.

Key Vocabulary

Service class — a Python class decorated with @bentoml.service that defines the serving logic for a model, including input/output types, resource requirements, and scaling configuration. “Define the inference logic inside the Service class and annotate each endpoint with @bentoml.api so BentoML can generate the OpenAPI specification and handle serialisation automatically.”

Runner — a BentoML abstraction (used in pre-2.0 versions) that encapsulates a model and its execution environment, enabling the service layer to call inference in an isolated, scalable process. “In legacy deployments, the runner handles model loading and batching independently of the HTTP layer; in BentoML 2.x this is replaced by the unified Service class.”

bentofile.yaml — the project manifest file that declares the service entry point, Python dependencies, Docker base image, and any additional files to include in the built Bento artifact. “Update bentofile.yaml to pin the transformers version to 4.38 and add the models directory to the include list before running bentoml build.”

containerize — the BentoML CLI command that builds a Docker image from a Bento artifact, embedding the model, dependencies, and serving code into a portable, deployable container. “Run bentoml containerize summariser:latest —platform linux/amd64 to produce a Docker image compatible with our Kubernetes cluster.”

Bento — the immutable, versioned artifact produced by bentoml build, containing the serialised model, service code, dependencies, and configuration in a single distributable unit. “Tag and push the Bento to the remote registry so the deployment pipeline can pull the exact version tested in staging rather than rebuilding from source.”

OpenLLM integration — BentoML’s native support for serving large language models through OpenLLM, providing an OpenAI-compatible API endpoint with configurable generation parameters. “Deploy Mistral 7B using OpenLLM integration so the frontend team can use the same client library they use with OpenAI’s API, with no code changes required.”

Adaptive batching — BentoML’s feature that automatically groups concurrent inference requests into a single batch to improve GPU utilisation without requiring the client to send batches explicitly. “Enable adaptive batching with a max batch size of 32 and a timeout of 10 milliseconds so we saturate the GPU during peak traffic without adding perceptible latency for individual requests.”

Resource annotation — configuration metadata on a BentoML service that specifies the CPU, memory, and GPU requirements for the serving environment, used by orchestration platforms to schedule correctly. “Add the resource annotation requesting one NVIDIA A10G GPU so BentoCloud and Kubernetes both know to schedule this service on an accelerated node.”

Common Phrases

• “Build the Bento, verify the artifact locally with bentoml serve, then containerize for the deployment pipeline.”
• “Pin all dependencies in bentofile.yaml — the Bento must be reproducible months after the original build.”
• “The Service class replaces the runner pattern in BentoML 2.x; don’t mix the two APIs.”
• “OpenLLM wraps the model loading and generation loop; you provide the model ID and the serving config.”
• “Use bentoml models list to inspect what is in the local store before building the artifact.”

Example Sentences

When presenting a serving architecture to the platform team: “The ML team will package each model as a versioned Bento artifact, which the platform team can pull and deploy to the GPU node pool using the containerized image. The bentofile.yaml declares all resource requirements, so the scheduler has the information it needs without additional configuration.”

When writing a handoff document for a new model: “To deploy the text classification model, run bentoml build in the repository root to produce a Bento, then bentoml containerize to create the Docker image. The bentofile.yaml specifies 4 CPU cores and 8 GB RAM; no GPU is required for inference on this model.”

When discussing LLM serving options in a technical review: “We evaluated vLLM and BentoML’s OpenLLM integration. We chose BentoML because the OpenAI-compatible endpoint allows the product team to switch models without changing the client code, and the Bento artifact model gives us traceable, reproducible deployments aligned with our MLOps maturity requirements.”

Professional Tips

• Use “artifact-based deployment” as a phrase when justifying BentoML to DevOps teams — it maps to familiar concepts like Docker images and Helm charts rather than treating models as a special case.
• When discussing adaptive batching, clarify both the max_batch_size and the timeout parameters — neither alone is sufficient to describe the trade-off between latency and throughput.
• Distinguish BentoML 1.x runner-based architecture from the BentoML 2.x Service class model when reading documentation or onboarding colleagues — the APIs are incompatible.
• Reference the OpenAI-compatible endpoint as a migration path when advocating for open-source LLM serving — it dramatically lowers the switching cost for product teams.

Practice Exercise

1. A data scientist has trained a model locally and wants to make it available to the backend team as an HTTP API. Describe the BentoML steps involved in three sentences.
2. Explain adaptive batching to a product manager who is asking why the inference server sometimes responds faster under high load than under low load.
3. A colleague asks what the difference is between a Bento and a Docker image. Write two sentences clarifying the relationship between the two.