SageMaker AI Routes Requests Using Container-Reported GPU Cache State
AI-generated summary Amazon Web Services introduced KV cache-aware routing for Amazon SageMaker AI real-time endpoints on November 17, 2026, using container-reported cache state to place requests. AWS reports up to 97% lower median time to first token and 234% higher throughput than random routing in long-prompt tests under load.