LazyWeights: Faster Quantized Edge LLM Inference
AI-generated summary Researchers introduced LazyWeights, an edge inference architecture that pairs entropy-aware quantization with an on-demand hardware decompressor situated between DRAM and the processor L2 cache. Tested on hardware including the NVIDIA Jetson Orin Nano Super and Jetson Nano, the architecture reduced per-token latency and energy consumption during model evaluation.