Inference Engineering 101 - with Philip Kiely

TL;DR

• 1

  Production viability trumps benchmark scores: The real question isn't whether a model scores 72 or 74 on a benchmark, but whether it's sufficiently intelligent for your specific task, which is a binary outcome.

• 2

  Quantization is essential, not optional: Models trained in BF-16 (2 bytes per parameter) become impractical at scale; NVFP4 cuts memory requirements in half while preserving model quality through dual scale factors that protect outlier dynamics.

• 3

  Mixture of experts architecture benefits everyone differently: Local users benefit from only activating a fraction of parameters per request, while data centers use expert parallelism to optimize throughput across batched requests.

• 4

  Prefill and decode are fundamentally different workloads: Prefill is compute-bound (time to first token), decode is memory-bound (tokens per second), which is why disaggregated inference separates them onto dedicated workers for 2x performance gains.

• 5

  The inference field is remarkably young: Optimizations routinely deliver 20-50% improvements or even 2x speedups, unlike mature fields like quantitative finance where 2-5% gains are considered significant.

• 6

  Framework choice depends heavily on use case: vLLM runs everywhere on everything, SGLang excels at multimodal especially image, and TensorRT-LLM delivers best performance but has steep learning curve and limited model support.