From Tokens to Bytes II: Learning from Raw Data
AI-generated summary Cartesia releases a blog post exploring raw data processing methods and tokenization tradeoffs in language models. Cartesia says BPE tokenization noticeably improves compute-to-loss efficiency compared to byte-level models.