Approximating self-attention in linear time and memory via the Nyström method
AI-generated summary Transformers utilize the self-attention mechanism to capture pairwise interactions between input tokens across various natural language processing and computer vision tasks. The Nyströmformer model approximates standard self-attention with linear time and memory complexity by applying the Nyström method to the softmax matrix.