Inference / 10 chapters
Put the model into production
Ten chapters follow the full serving path from service-level objectives to realistic load tests and reliable overload behavior.
- 01Read ↗
Inference Begins With a Service Contract
Define time to first token, inter-token latency, end-to-end latency, throughput, availability, workload shape, and cost before selecting hardware.
- 02Read ↗
Anatomy of an Inference Request
Trace admission, tokenization, queueing, batching, model execution, sampling, streaming, cancellation, and observability across explicit boundaries.
- 03Read ↗
Prefill and Decode Are Different Workloads
Separate compute-heavy prompt processing from memory-bound token generation and measure each phase with the metric it controls.
- 04Read ↗
KV-Cache Arithmetic and Memory
Calculate cache bytes per token and request, then connect context length, concurrency, paging, eviction, and admission control.
- 05Read ↗
Continuous Batching and Scheduling
Schedule token work rather than whole requests, control queue delay, handle preemption and cancellation, and prevent long prompts from dominating.
- 06Read ↗
Parallel Inference Across Accelerators
Choose tensor, pipeline, data, or expert parallelism by balancing memory fit, communication volume, latency, throughput, and failure domains.
- 07Read ↗
Quantization and Numerical Formats
Compare weight, activation, and cache precision; measure memory traffic and kernel support while guarding output quality and stability.
- 08Read ↗
Speculative Decoding and Latency Levers
Use fused kernels, prefix reuse, speculative decoding, and workload-aware tuning only after identifying the phase that dominates latency.
- 09Read ↗
Capacity Planning and Reliable Overload
Turn memory, token throughput, queue limits, timeouts, retries, backpressure, and degraded modes into an explicit capacity envelope.
- 10Read ↗
Load Testing and the Evidence Bundle
Benchmark realistic prompt and output distributions, concurrency, streaming, cache pressure, failures, tail latency, quality, and cost together.