Inference Optimization
Inference optimization encompasses the techniques and software used to reduce the computational cost and latency of running artificial intelligence models in production. Truncated covers hardware acceleration methods, quantization strategies, compiler frameworks and performance benchmarks.
MONAILabel 0.8.5 on AMD Instinct™ GPUs: Interactive Medical-Image Annotation
rocm.blogs.amd.com — Learn how to run MONAILabel 0.8.5 for interactive medical-image annotation on AMD Instinct™ GPUs.
AI’s next infrastructure constraint
www.datacenterdynamics.com — Data center expansions have long been represented in gigawatts, but measurements of terabits-second and microseconds are now entering the constraint conversation
Expedera Demonstrates Edge AI Subsystem Pairing with Cadence
www.unite.ai — Expedera announced on November 9, 2026, that it had paired its Origin Evolution neural processing unit (NPU) IP with Cadence’s Tensilica NeuroEdge 130 AI Co-Processor in a pre-silicon simulation, demonstrating how the tw...
Lawyer builds private AI server with eight used Nvidia V100 GPUs — up to 215 tokens per second to keep legal files offline, costing about $2,400 per four-GPU board
GKE dynamic slicing on TPUs
cloud.google.com — Compared to creating a static nodepool for each TPU slice, GKE dynamic slicing reduces AI workload startup and workload recovery times.
Distributed LLM Inference on Kubernetes with llm-d, Wasm, & TornadoVM
www.infoq.com — In this episode, Edoardo Vacchi discusses the evolving landscape of WebAssembly on the backend, running Java workloads on GPUs using TornadoVM, and optimizing distributed AI inference using llm-d and vLLM on Kubernetes. Edoardo explains how llm-d manages cache locality (KV and prefix caches), request routing, and batch inference across Kubernetes clusters to maximize expensive GPU utilization.
AWS Launches TorchNeuron to Run PyTorch Models on AWS Trainium Without Modifications
www.infoq.com — At the recent PyTorch Conference North America 2026, Amazon introduced TorchNeuron, a native PyTorch backend that enables developers to run existing PyTorch code on AWS Trainium AI chips with virtually no code modifications.
NVIDIA Dynamo Recipes Bring Korea’s Leading Open Models to Production
blogs.nvidia.com — NVIDIA is releasing Dynamo-optimized deployment recipes for Korea’s leading open models.
Capability Gain from ISA to Shared Execution on AMD MI300A and MI250X
hgpu.org — Each new GPU generation is showcased by its peak instruction rates, yet users hardly ever run instructions directly: they reach the hardware's performance through vendor libraries, application…
Deep Learning Compilers: Architecture, Optimization, and Empirical Characterization
hgpu.org — As deep learning (DL) models become increasingly complex and hardware architectures grow more heterogeneous, DL compilers play a critical role in optimizing computational efficiency and enabling sc…
Characterizing CUDA Locality Domains on Blackwell GPUs
hgpu.org — Low latency is critical to user experience. As NVIDIA Blackwell and Rubin GPUs advance multi-die scaling, data placement and cross-die communication become increasingly important. However, the late…
HPC-Bench: A Comprehensive Benchmark for High Performance Computing Codes
hgpu.org — Large Language Models (LLMs) have shown strong capabilities in code generation and reasoning, motivating extensive benchmarks for evaluating functional correctness. However, performance optimizatio…
I played Android’s first AI frame-gen game: Mobile just had its DLSS moment
Ollama was holding back my "weak" Intel GPU, and one swap turned it into a solid local AI setup
XDA — My Intel Arc GPU finally became a useful local AI machine
Fireworks says you’re wasting money on AI unless you’re using all the models
theregister — The case for managed inference, or automatically routing queries to the right model for the right job
My 12GB RTX 4070 Ti runs multiple DLSS 5 layers thanks to one OptiScaler setting
XDA — Nothing like an open-source tool to squeeze more power
Speed Up Image Generation with FLUX 3 Image Klein on NVIDIA Blackwell GPUs
NVIDIA Technical Blog — When deploying diffusion models in design tools, image editors, and visual production workflows, developers must account for inference latency. Generating an image requires multiple sequential passes…
DeepSeek V4.1 Flash speeds on GB300 NVL72 have increased by up to 1.5x tok/s per agent over the past month on AA-AgentP…
AWS Adds NVIDIA GPU Partitioning to SageMaker Inference Endpoints
Unite.AI — Amazon Web Services (AWS) announced NVIDIA Multi-Instance GPU support for Amazon SageMaker Inference on November 6, 2026, letting multiple hardware-isolated models share one accelerator on real-time endpoints instead of...
Dismiss email signup
Subscribe to 
The top AI news in your email delivered each morning. Choose categories