LLM Serving Adapted to Agentic Workloads
AI-generated summary MIT Han Lab, Georgia Tech, UIUC, UC Berkeley, NVIDIA, and OpenAI published Evolving Infra on arXiv, an LLM serving framework designed to adapt online to agentic workloads. The framework features nodes that switch between prefill and decode in seconds alongside a KV-manager that utilizes agent signals to manage cache.