Arun Prabhakaran measures Qwen3-8B API call on NVIDIA A100

The author examines what happens inside a single LLM API call on an NVIDIA A100 GPU using the Qwen3-8B model, highlighting stateless memory, prefix KV caching, chat templates, prefill-to-decode throughput gaps, and the limitations of prompt-based JSON generation. Through empirical measurement, the article demonstrates how prefix caching saves computation across conversation turns, why output generation is vastly slower than input prefilling due to memory bandwidth, and how constrained grammar masks ensure valid JSON outputs where naive prompts fail. Open article →