Fast and Memory-Efficient Exact Sampling
AI-generated summary Researchers present FlashSampling, an exact sampling primitive that fuses sampling directly into the LM-head matmul to avoid materializing the logits tensor in HBM. The method demonstrates kernel-level speedups on decode workloads across H100, H200, B200, and B300 GPUs while reducing time per output token by up to ten percent in vLLM experiments.