English for SGLang Inference Developers
Learn the English vocabulary for SGLang: structured generation, radix attention caching, and serving LLMs with high-throughput constrained decoding.
SGLang conversations mix serving-infrastructure language with prompt-programming concepts — radix cache, constrained decoding, continuous batching — so engineers moving from a general LLM-serving background need a few precise terms to keep up with performance discussions.
Key Vocabulary
Radix attention — SGLang’s KV-cache reuse mechanism that shares common prefixes across requests in a radix tree, avoiding recomputation when prompts overlap. “Latency dropped once we restructured the prompts so the system prompt is a shared prefix — radix attention is caching that whole segment across requests now.”
Constrained decoding — forcing the model’s output to conform to a grammar, regex, or JSON schema at the token level during generation, rather than validating after the fact. “Instead of retrying on invalid JSON, we switched to constrained decoding so the model literally can’t emit a token that breaks the schema.”
Continuous batching — dynamically adding and removing requests from a running batch as they arrive and finish, instead of waiting for a fixed batch to complete. “Throughput doubled after we enabled continuous batching — short requests no longer wait behind one long-running generation.”
Frontend language (SGLang) — the Python DSL for describing multi-step, branching, or parallel LLM calls as a single program that the runtime can schedule and cache efficiently. “Write the retrieval-then-answer flow in the SGLang frontend language so the scheduler can overlap the two calls instead of running them as separate round-trips.”
Prefix cache hit rate — the percentage of incoming requests whose shared prompt prefix is already cached, a key metric for judging whether prompt structure is exploiting radix attention. “Our prefix cache hit rate is under 20% — we’re generating a slightly different system prompt per request, which defeats the caching entirely.”
Common Phrases
- “Is this a cache miss because the prefix changed, or is the radix tree just cold after a restart?”
- “Can we express this as constrained decoding instead of a post-hoc JSON validator with retries?”
- “What’s our continuous batching window looking like under this concurrency — are short requests getting starved?”
- “Should this multi-step flow live in the SGLang frontend language so the scheduler can parallelize it?”
- “What’s the prefix cache hit rate telling us — is prompt templating actually helping here?”
Example Sentences
Debugging a latency regression: “The p99 spike lines up with a drop in prefix cache hit rate — someone added a timestamp to the system prompt, so every request is now a cache miss.”
Explaining an architecture choice: “We chose constrained decoding over a validate-and-retry loop because it guarantees valid output on the first pass instead of burning tokens on failed attempts.”
Reviewing a pull request: “Move the shared instructions to the start of the prompt and the per-request data to the end — that ordering is what lets radix attention actually cache anything.”
Professional Tips
- Say radix attention, not “the caching thing” — it signals you understand SGLang’s specific mechanism versus generic KV-caching.
- Distinguish constrained decoding from output validation in design discussions — one prevents invalid tokens, the other catches them after generation.
- Reference prefix cache hit rate as a concrete metric when arguing for prompt restructuring, not just “this should be faster.”
- Use frontend language precisely when describing SGLang’s Python DSL, not “the SDK” — it’s a program the scheduler compiles and optimizes.
Practice Exercise
- Explain why prompt structure affects the prefix cache hit rate, using the term “radix attention.”
- Describe the difference between constrained decoding and validating output after generation.
- Write a sentence justifying a move to continuous batching for a mixed-length workload.