# Pretraining Articles associés

Tsinghua '00s Alumnus Wang Guan's New Work: Disrupting Transformer Pretraining Models with 1/900 Tokens, 1/432 Compute Power

Tsinghua alumnus Wang Guan's team proposes HRM-Text, a novel pre-training paradigm using a Hierarchical Recurrent Model to replace standard Transformers. With just 1B parameters and 40B unique tokens trained at a cost of ~$1500, HRM-Text achieves performance comparable to 2B-7B open-source models, using up to 900x fewer tokens and 432x less estimated compute. Key innovations include a dual-timescale recurrent architecture for greater effective depth, a task-completion objective training only on answer tokens with PrefixLM masking, and techniques like MagicNorm and Warmup Deep Credit Assignment for stability. Evaluations show strong results on benchmarks like MMLU (60.7%) and GSM8K (84.5%). The work highlights how architectural priors and targeted objectives can lower pre-training barriers, though limitations include knowledge-reasoning coupling, fixed compute per token, and scalability beyond 3B parameters.

marsbit05/26 03:16