Google claims its new AI software cuts inference latency by over 90% — but there's a catch
AI-generated summary Google announced an open-source system called TPU Sync designed to speed up AI inference by addressing data movement between memory types. Google claims the software can cut key-value cache transfer latency and time to first token by more than 90 percent in certain tests.