Inference Optimization
Inference optimization encompasses the techniques and software used to reduce the computational cost and latency of running artificial intelligence models in production. Truncated covers hardware acceleration methods, quantization strategies, compiler frameworks and performance benchmarks.
Hacking on XLA: Building It, and Teaching It a New Pass
Aditya Kumar — The XLA review said that when you disagree with XLA’s fusion decisions, your recourse is to restructure your source or leave through custom_call. That is true of fusion. It isn’t true of the pipeline as a whole. XLA is a C++ compiler with an ordinary pass manager, and you can build it from source and add a pass. The pass itself isn’t the interesting part. Writing one forces you to learn how the pipeline decides it is done, and that matters more than any single rewrite.
On Big Chip Club, @MilksandMatcha talks with @cdngdev, the former @OpenAI robotics intern behind this summer’s viral As…
A Request’s Life in vLLM V1
Aditya Kumar — Part 2 of a ten-part series on serving engines from the inside. Pinned to vLLM 0.32.0 (5df63a7). Part 1 named the five layers and where vLLM and SGLang cut them into processes. This part follows one streamed chat request through vLLM’s cut.
GLM-5.2 AgentX on AMD Instinct MI355X: Boosting Throughput and Interactivity
ROCm Blogs — Learn how cache and kernel optimizations raised GLM-5.2 AgentX throughput and interactivity on AMD Instinct MI355X.
AMD at PyTorch Conference 2026: Advancing the Open AI Ecosystem
AMD — AMD highlights PyTorch advancements across training, inference, and performance optimization at PyTorch Conference 2026.
Together Brings Recursive Self-Improvement To Open Models
War Thunder gets DLSS 5 this month, NVIDIA claims up to 400 FPS at 4K
How to Scale Diffusion Inference Across AMD RDNA 4 GPUs
ROCm Blogs — Run xDiT image and video diffusion workloads on multiple RDNA 4 GPUs with practical guidance for latency, memory, and scaling.
Dynamic LoRA on AMD Ryzen™ AI: One Base Model, Many Specialists, Switched at Runtime
AMD — Explore dynamic LoRA on AMD Ryzen™ AI to switch adapters at inference time, reduce memory overhead, and efficiently deploy specialized LLMs on AI PCs.
Faster Stable Diffusion on AMD Ryzen™ AI with TeaCache
ROCm Blogs — In this blog, you will learn how to speed up Stable Diffusion 3 and 3.5 on AMD Ryzen AI NPUs with TeaCache, without retraining.
Progress on provable inference for large language models
anthropic.com — Sharing progress on a powerful type of cryptographic proof known as a zkSNARK, which is uniquely fit for provable inference.
Why Billion-Dollar AI Systems Are Only As Fast As The Data Feeding Them
From capacity, to the cost of intelligence
datacenterdynamics.com — The next shift in AI infrastructure
Qualcomm Demonstrates HBC Memory With 18X Better Bandwidth
Modders create automated tool to unlock AI performance on NVIDIA GeForce RTX 20, 30 and 40 series GPUs
Nvidia quietly gave RTX 40-series owners a new frame generation mode, and I measured it on my 4070 Ti Super
XDA — Does the frame generation "performance" mode live up to its promises?
The fastest delta weight syncs in the world
Baseten — 770GB of weights previously took ~36 seconds to sync on average; now they take 15, with a sub-second request pause, 7.5x faster than previous SOTA.
Bringing the MONAI Model Zoo to AMD Instinct™ GPUs: Validated Inference on ROCm
ROCm Blogs — Learn how to run validated inference for five MONAI Model Zoo bundles on AMD Instinct™ GPUs using a ROCm overlay.
How OpenRouter rebuilt provider routing for better, faster inference
OpenRouter Blog — OpenRouter's new default provider routing scores each provider on effective price, throughput, reliability, availability, and output quality for every request instead of ranking mostly by prompt token price. Measured with a switchback experiment against the old router.
Dismiss email signup
Subscribe to 
The top AI news in your email delivered each morning. Choose categories