Model Serving

AIAI-generated summary NVIDIA says its Vera Rubin NVL72 delivers up to thirty times the throughput per megawatt of GB300 NVL72 at one hundred sixty tokens per second per user on SemiAnalysis AgentX. NVIDIA says NVFP4 reduces the model footprint by approximately three point five times relative to FP16 and one point eight times relative to FP8.

AIAI-generated summary DeepSeek V4.1 Flash 1017 matches GLM-5.3 intelligence at about one third the cost and four times the speed according to Baseten. Baseten details three optimizations used to make Flash models faster including kernel fusion Programmatic Dependent Launch and host loop minimization.

AIAI-generated summary Figma says its users are moving faster with AI tools like the agent and Figma Make, which require secure sandboxes with stable infrastructure to run AI-generated code quickly. Figma says it built a centralized platform called Nimbus to manage sandboxes at scale across multiple cloud providers and handle resource allocation and network security.

AIAI-generated summary FlyDSL allows developers to write GPU programs in Python and compile specialized kernels on demand for LLM serving. ATOM integrates FlyDSL mono decode for models including MiniMax-M3, GLM-5.2, Kimi-K3, and DeepSeek-V4.1.

AIAI-generated summary Engineering at Meta explains that F3 Lightning was developed to allow every feature to be released on its own schedule while preserving extraction efficiency. Engineering at Meta notes that the system compiles each feature independently and consolidates shared work at startup using identifiers based on the computations themselves.

AIAI-generated summary Artificial Analysis has launched the next version of its API Performance Benchmarking with Time per Task as the headline performance metric for serverless endpoints. Artificial Analysis says the update reflects how agentic workloads make the time required to complete a task more important than streaming speed.

AIAI-generated summary Aditya Kumar released a detailed technical post pinned to vLLM 0.32.0 explaining the internal architecture and request lifecycle of vLLM V1. Every per-token cost that is not the forward pass is paid in the API server on a single asyncio event loop per process.

AIAI-generated summary SambaNova announced a partnership with Supermicro to deploy SN50 RDU chips for AI disaggregated inference systems. Supermicro will initially purchase 10,000 SN50 chips from SambaNova and integrate them alongside Nvidia GPUs for customers.

AIAI-generated summary Cohere rolls out open models inside its Model Vault sovereign inference platform to provide increased accessibility. Cohere says this platform allows users to choose the right model for the job while keeping their data safe.

AIAI-generated summary AI inference startup Fal announced a new $1.3 billion financing round led by Bessemer Venture Partners that doubles its valuation to $16 billion. Chief executive officer and co-founder Burkay Gur says the startup raised the large amount in part to secure more computing power after a surge of demand led to customer rate limits.

AIAI-generated summary GitHub launches Copilot Fusion to route between models and reduce token costs by routing tasks intelligently. GitHub states that teams and early enterprise customers are seeing 50% to 60% lower costs on complex work while matching high-end model results.

AIAI-generated summary The Nutanix Agent Gateway is available now in Nutanix Enterprise AI 2.7 to help organizations manage and control artificial intelligence workloads. Nutanix says its own pivot to open-weight models running on local GPUs has cut costs by 80 percent.

AIAI-generated summary Baseten released an updated delta weight sync design for models like GLM-5.3 that reduces synchronization time to 15 seconds. Baseten says the new design enables throughput improvements by as much as 3.2x through optimized GPU memory utilization.

AIAI-generated summary OpenRouter launched a new provider routing algorithm designed to reward providers for delivering fast, high-throughput, and reliable inference. OpenRouter says the algorithm has unlocked improvements in time to first token and throughput, along with a reduction in capacity errors.

AIAI-generated summary Baseten makes available an optimized implementation of Qwen3-ASR 1.7B for batch transcription. Baseten says the optimized stack transcribes 7x more audio per second at high concurrency compared with out-of-the-box vLLM.

AIAI-generated summary Akamai has introduced Akamai Inference in beta to provide enterprises with managed access to open-weight and custom models through OpenAI-compatible APIs. The new cloud-based service includes serverless deployment options and a selection of twelve open-weight models ranging from eight billion to 2.8 trillion parameters.