Apple has new Mac app coming to make running local AI models easy
9to5mac.com — Leaker Aaron Perris has discovered evidence of a new Mac app in the works at Apple that makes it easy to run local AI models.
Model Serving
Model serving encompasses the operational systems and infrastructure used to deliver artificial intelligence inference requests in production environments. Truncated covers inference hardware, serving frameworks, optimization techniques and deployment architectures.
Start and Restore GPU Workloads in Seconds with NVIDIA Snapshot
developer.nvidia.com — NVIDIA Snapshot, a new open-source component of the NVIDIA DSX AI Factory Platform, makes initialized GPU workloads reusable across inference replicas—speeding startup and recovery.
Google Cloud Makes Pathways Runtime GA for Large-Scale TPU Workloads
www.unite.ai — Google Cloud on November 9, 2026, announced that Cloud Pathways is generally available on Google Kubernetes Engine (GKE), bringing a single-controller runtime for large-scale training and inference to customers. Google s...
Dynamic model routing: The new SD-WAN for the AI era
blogs.cisco.com — Stop overspending on monolithic LLMs. Learn how dynamic model routing can intelligently steer queries based on performance, cost, and security requirements to help maximize your AI ROI and optimize token efficiency.
Google claims its new AI software cuts inference latency by over 90% — but there's a catch
thenewstack.io — Google’s new open-source TPU Sync aims to speed AI inference by moving KV-cache data between memory tiers, cutting time to first token by up to 90%.
How to run open models in production without owning the serving stack
Agentic AI Will Hit an Infrastructure Wall Before It Hits an Intelligence Wall
www.unite.ai — The Models Are Improving Faster Than Our Ability to Serve What They’re Being Asked to Do. Most of the argument about where AI agents are headed is an argument about capability. Can the model plan reliably? Can it recover...
Fearless Engineering: The CS-4 is our first engineering documentary.
Arcee is a proud sponsor of the @lmsysorg SGLang Summit at Fort Mason Center kicking off this Thursday.
Rafay’s playbook on helping neoclouds graduate beyond selling just infrastructure
www.sdxcentral.com — CPO explains how token routing and software nous keeps GPUs busy
Distributed LLM Inference on Kubernetes with llm-d, Wasm, & TornadoVM
www.infoq.com — In this episode, Edoardo Vacchi discusses the evolving landscape of WebAssembly on the backend, running Java workloads on GPUs using TornadoVM, and optimizing distributed AI inference using llm-d and vLLM on Kubernetes. Edoardo explains how llm-d manages cache locality (KV and prefix caches), request routing, and batch inference across Kubernetes clusters to maximize expensive GPU utilization.
NVIDIA Dynamo Recipes Bring Korea’s Leading Open Models to Production
blogs.nvidia.com — NVIDIA is releasing Dynamo-optimized deployment recipes for Korea’s leading open models.
Fireworks says you’re wasting money on AI unless you’re using all the models
theregister — The case for managed inference, or automatically routing queries to the right model for the right job
Voice AI infrastructure: The 2026 stack and architecture
ElevenLabs — Explore every layer of voice AI infrastructure and AI agent architecture, from speech recognition to guardrails, with ElevenAgents examples.
AWS Adds NVIDIA GPU Partitioning to SageMaker Inference Endpoints
Unite.AI — Amazon Web Services (AWS) announced NVIDIA Multi-Instance GPU support for Amazon SageMaker Inference on November 6, 2026, letting multiple hardware-isolated models share one accelerator on real-time endpoints instead of...
Introducing the New Auto and Auto Code Routers on OpenRouter
OpenRouter Blog — OpenRouter's next generation Auto Router and the new Auto Code router pick a model and reasoning effort for each request from the model pool your guardrails, ZDR, provider restrictions, and BYOK policies allow. Auto Router for general agents and apps, Auto Code for coding agents and harnesses.
ROCm InferaSim: Cut LLM Serving Trial Costs and Deploy the Best Configuration
ROCm Blogs — Learn how ROCm InferaSim predicts LLM serving performance on AMD Instinct GPUs from a few short runs, so you can deploy the best configuration.
Why edge-cloud AI inference pipelines are key
InfoWorld — Splitting AI work across devices, edge platforms, and central cloud is how enterprises will make latency, cost, and control work at scale.
My gaming GPU sat idle most of the day, so I turned it into a local AI server the whole house can use
XDA — One Llama.cpp command can turn your PC into an AI server, and it really is that easy
Online Quantization: Smooth Path from One Checkpoint to Any Desired Serving Scenario
ROCm Blogs — Learn how AMD Quark's online quantization quantizes LLM weights at load time across vLLM, ATOM, and SGLang, with per-layer mixed FP8/MXFP4 recipes.
Dismiss email signup
Subscribe to 
The top AI news in your email delivered each morning. Choose categories