Online Quantization: Smooth Path from One Checkpoint to Any Desired Serving Scenario
ROCm Blogs — Learn how AMD Quark's online quantization quantizes LLM weights at load time across vLLM, ATOM, and SGLang, with per-layer mixed FP8/MXFP4 recipes.
Model Serving
Model serving encompasses the operational systems and infrastructure used to deliver artificial intelligence inference requests in production environments. Truncated covers inference hardware, serving frameworks, optimization techniques and deployment architectures.
Five Ways NVIDIA Co-Design Maximizes the Value of Every Byte in the AI Factory
NVIDIA Technical Blog — In an AI factory, high bandwidth memory (HBM)is the fastest, most expensive, and most contested memory tier. Every byte spent on a redundant weight copy, an idle session’s KV cache…
DeepSeek V4.1 Flash 1017 matches GLM-5.3 intelligence at ~⅓ the cost and 4x the speed.
How Figma Manages Sandboxes at Scale
Figma — We built Nimbus: a platform that makes sandboxes fast, reliable, and easy for Figma product teams and operators to manage.
FlyDSL Mono Kernels for High-Interactivity LLM Serving in ATOM
ROCm Blogs — FlyDSL mono kernels cut decode latency in ATOM, with MiniMax-M3 and DeepSeek-V4.1 results.
F3 Lightning: How Meta Accelerates Ads ML Features to Production at Scale
Engineering at Meta — Facebook Feature Framework (F3) is Meta’s framework for authoring, compiling, and serving the machine learning features underlying ads ranking models, where a single request can require more …
Announcing the next version of Artificial Analysis API Performance Benchmarking: Time per Task is now our headline perf…
A Request’s Life in vLLM V1
Aditya Kumar — Part 2 of a ten-part series on serving engines from the inside. Pinned to vLLM 0.32.0 (5df63a7). Part 1 named the five layers and where vLLM and SGLang cut them into processes. This part follows one streamed chat request through vLLM’s cut.
SambaNova partners with Supermicro for SN50 RDU deployments
datacenterdynamics.com — Server maker will initially purchase 10,000 SN50 chips from SambaNova
We launched our sovereign inference platform Model Vault in January so you could choose the right Cohere model for the…
AI Inference Startup Fal Raises Capital at $16 Billion Valuation
AI coding needs smarter routing over bigger models
The Deep View — Coding agents can drastically accelerate how fast developers build, but defaulting to the most powerful models gets expensive fast. But what if your coding ass...
Does your model need a Rolls Royce or a bicycle?
The Stack — Nutanix CTO: “When you pay for infrastructure, it is much more predictable than paying per token”
The fastest delta weight syncs in the world
Baseten — 770GB of weights previously took ~36 seconds to sync on average; now they take 15, with a sub-second request pause, 7.5x faster than previous SOTA.
How OpenRouter rebuilt provider routing for better, faster inference
OpenRouter Blog — OpenRouter's new default provider routing scores each provider on effective price, throughput, reliability, availability, and output quality for every request instead of ranking mostly by prompt token price. Measured with a switchback experiment against the old router.
The fastest, lowest-cost batch transcription with Qwen3-ASR
Baseten — Transcribe an hour of audio with Qwen3-ASR for 1/10th of a cent, with 7x higher throughput, 6.9x lower latency, and no meaningful effect on quality.
Akamai launches AI inference service
datacenterdynamics.com — Company adds new GPUs to cloud fleet, targets next year for Vera Rubin hardware
Dismiss email signup
Subscribe to 
The top AI news in your email delivered each morning. Choose categories