HRM-Text

HRM-Text is a pretraining framework for building your own small language model from scratch, cheaply. The claim is foundation-model pretraining with 130 to 600 times less compute and far less data than usual, using a Hierarchical Reasoning Model architecture instead of a standard transformer. Apache-2.0 and free, with a pre-trained 1B checkpoint on Hugging Face if you'd rather not train.

This is research infrastructure, not a product. It's a full training pipeline: prepare tokenized data, launch distributed training, evaluate on benchmarks like MATH and MMLU, export to Hugging Face format. The architecture leans on recurrent reasoning layers, sequence packing, and FlashAttention 3 kernels to squeeze efficiency out of the run.

The catch: cheap is relative. You still need a cluster of 8 to 16 H100 GPUs and roughly 800 to 1,500 dollars in compute for a full training run. This is for researchers and teams exploring efficient architectures, not for anyone who just wants to use a model. If that's you, grab the checkpoint and skip the training code.