From Tokens to Bytes I: Scaling Hierarchical Networks
AI-generated summary Cartesia explores scaling hierarchical networks to replace hand-crafted tokenization steps in language models and learn directly from raw bytes. Cartesia says that hierarchical networks improve validation loss by 1.3x and yield greater gains on code and out-of-distribution evaluation.