Baseten publishes DeepSeek engine, reports 2,060 tokens per second on eight B300s
AI-generated summary Baseten published an inference engine for the DeepSeek V4.1 Flash model, reporting a speed of 2,060 output tokens per second on eight Nvidia B300 GPUs. Teams with compatible hardware can run the DeepSeek weights on the engine, which was written in Rust and CUDA and released under the Elastic License 2.0.
- Lawyer upgrades eight-V100 private AI server to run 80% of DeepSeek V4.1 Flash at 4-bit NVFP4 precision — adds 1TB of RAM, an Nvidia RTX 3090, and fixes crashes caused by a bad PSU
- Lawyer builds private AI server with eight used Nvidia V100 GPUs — up to 215 tokens per second to keep legal files offline, costing about $2,400 per four-GPU board