Deforming the Transformer, LLMs Become Smarter
A new research paper proposes "Tapered Language Models (TLMs)," a method that improves large language model performance without adding any parameters. It challenges the standard Transformer design where each layer has the same number of parameters ("feed-forward network" width). Building on evidence that layers are not equally important—earlier layers handle foundational information like grammar, while later layers often reinforce existing judgments—the researchers suggest reallocating model capacity from later to earlier layers.
The core idea is to make the layer width taper off monotonically from start to end, keeping total parameters and compute constant. Experiments compared linear, cosine, and sigmoid tapering curves on a 440M parameter model. The cosine curve (e.g., starting width 1.5x baseline, ending 0.5x) achieved the best result, reducing perplexity by 1.84 points compared to the uniform baseline—a significant gain at zero cost.
This finding proved robust across four different model architectures (including gated attention and memory-augmented models) and at larger scales (760M and 1.3B parameters), consistently improving performance on commonsense reasoning and language modeling tasks without harming long-context retrieval ability.
The work highlights a long-overlooked design dimension: optimal parameter allocation across depth. It offers a "free lever" for efficiency, potentially applicable beyond language models to vision Transformers and diffusion models. The study was conducted by researchers from Mila, Cornell University, and the University of Montreal.
marsbit7h ago
