Scaling LLM-Based Ranking Systems for Latency-Critical Search & Recommendation Workloads
AI-generated summary Sundara Raman Ramachandran details how LinkedIn productionizes large language model ranking under strict latency service level agreements for search and recommendation workloads. The approach involves distilling a 7B model down to 400M parameters, compressing job contexts into single-token embeddings, and removing system bottlenecks.