Many key techniques in Transformer architecture collectively crumbled at a conference?!
At COLM 2026, multiple papers simultaneously targeted the same issue—
Those unquestioned design choices in Transformers that nobody doubts anymore, actually don't hold up.
Transformer is almost the default architecture for all mainstream models, and its settings are largely regarded as sacred, with few looking back to verify.
As a result, these papers recalculated the costs of these default choices from different angles.
And the conclusions reached by different researchers were strikingly consistent: every standard design choice in Transformer has paid a real cost in some dimension not measured by existing evaluation systems.
Cracks in Long Context
The first paper, titled 'Cracks in the Foundation', from institutions including Ai2, targeted the architectural choices often treated as 'standard'—QK normalization, GQA, sliding window attention, and pretraining context length.
The researchers selected the values used in real models like Llama 2, Llama 3, Qwen 3, and Olmo 3, created permutations and combinations, and trained 26 models for comparison.
These models had parameters ranging from 7B to 8B, with data, tokenizer, and context extension schemes kept identical—only the architecture differed.
The team called these models 'OlmPool', pretrained them all on 140B tokens, then performed long-context finetuning with an additional 10B tokens, burning over 170,000 GPU hours in total.
The result: on the HELMET 32K evaluation, the 26 models scored a high of 56.4 and a low of 29.9—a difference of 26.5 points, translating to a relative gap of 47%.
With the same data and largely similar architecture, just flipping four different switches could cause such a wide score gap.
Flipping a single switch had little impact. Using a context length of 4096 vs 8192 during pretraining, or adding sliding window attention, only dropped the score by one or two points on average.
The real trouble came when several choices were combined. Once SWA and GQA were placed in the same model, the average score dropped by 9 points—far more than the sum of their individual effects.
The worst combination identified by the team was using GQA, sliding window attention, and per-head QK normalization together, with a drop in performance also significantly greater than the sum of individual effects.
They later found that simply counting how many 'harmful design' items a model had could accurately predict its long-context performance, even more reliably than analyzing the architecture item by item.
QK normalization was the most counterintuitive. It was originally added to models for training stability and is largely considered a good thing in the industry.
But the paper found that removing Olmo 3's original QK normalization and post-norm, and replacing them with pre-norm, actually increased the HELMET score by 6 points.
Applying the same change to Llama 3, however, had the opposite effect, dropping the score by 3.8 points.
The same 'standard' choice can yield completely opposite results when applied to a different base model.
This section also revealed a phenomenon contrary to many people's intuition—'attention concentration', referring to the model allocating a large portion of its attention to the first few tokens of the context.
This phenomenon has long been considered a bad habit that wastes compute, and many new methods specifically aim to eliminate it. However, in the OlmPool models, the more pronounced the attention concentration phenomenon, the better the long-context performance.
QK normalization happens to weaken this concentration, which might be the reason it hampers long-context ability.
Lost Gradients
The second paper, titled 'Lost in Backpropagation', from Cornell University, targeted the last layer used by almost all language models—the output projection layer.
At each processing step, a language model calculates a series of numbers representing its understanding of the current context; this series is called the hidden state D.
To predict the next word, the model must convert the hidden state into scores for each candidate word in the vocabulary, outputting as many scores as there are words in the vocabulary.
These scores are called logits; the higher the score, the more likely the model considers that word to be the next one.
The component performing this conversion is the output layer, essentially a matrix operation.
The size of the vocabulary is typically much larger than the length of the hidden state—one is tens of thousands, the other may only be a few thousand.
This size disparity was previously only viewed as one problem, called the 'softmax bottleneck', meaning a vocabulary too large compared to the hidden state limits the variety of next-word distributions the model can express.
This paper argues that the same disparity has another effect.
During backpropagation, the error signal must pass through the output layer to guide the model's parameter adjustments; this stage is also affected by the disparity.
Theoretically, the error signal contains as much information as the vocabulary before it is passed back.
However, the amount of information the output layer can transmit back is limited by its own structure, with an upper bound roughly the size of the hidden state, which is much smaller than the vocabulary.
The paper measured this information loss step on already-trained models like GPT-2, Pythia, Llama 3, OLMo 2, and Qwen 3, by projecting the error signal into the null space of the output layer weights and observing what remained.
The result: on models like GPT-2, Pythia, Llama 3, OLMo 2, and Qwen 3, 95% to 99% of the gradient norm was clipped at this step.
The remaining signal had a cosine similarity with the original gradient of only 0.1 to 0.2.
Over 90% of the signal meant to be passed back was 'eaten' at this step, and the remaining bit pointed in a direction misaligned with the intended one.
This output layer is a necessary passage almost every language model must traverse, never viewed as something needing scrutiny. Yet, during every backpropagation, it quietly erases most of the training signal that should have been transmitted back.
Cutting a Layer Breaks the Reasoning Chain
Another paper, titled 'When Fewer Layers Break More Chains', from the University of Tübingen, discusses a widely used model compression technique—layer pruning, which involves directly deleting certain layers from the Transformer to save compute by reducing model depth.
Layer pruning is viable because prior research found that some layers contribute little to overall performance; deleting them results in minimal score drops on conventional knowledge-based evaluations.
Pruning a quarter of the layers can retain over 80% of the original accuracy. Results like these have made layer pruning a generally accepted efficiency method.
But these evaluations test knowledge-based tasks with short answers, which can be answered correctly based on what the model has memorized.
This paper tests another capability—long-chain reasoning. The method used is 'test-time scaling', essentially letting the model think longer—giving it more tokens for reasoning, or having it generate answers multiple times and pick the correct one. Normally, the longer it thinks or the more attempts it makes, the higher the accuracy should be.
The paper selected two models inherently capable of this, s1.1-7B and Qwen3-8B, pruned one or two layers using three mainstream layer pruning methods, and observed whether these scaling methods still worked.
On knowledge-based evaluations, scores indeed dropped gradually. But on AIME24, with math competition difficulty, for s1.1-7B, pruning just one layer caused a significant drop in accuracy; pruning two layers dropped it close to zero.
Giving the model more thinking tokens should help it reason better. Instead, the pruned models showed no improvement with longer thinking time; test-time scaling essentially failed.
The authors then tested if finetuning could recover performance, using both LoRA finetuning and full-parameter finetuning on the pruned models.
For models pruned by one layer, finetuning had almost no effect. For models pruned by two layers, finetuning recovered some scores but remained far from pre-pruning levels.
In short, this paper aims to warn that layer pruning is not as safe as it appears.
The three papers study completely different subjects: one looks at long context, one at training gradients, and one at reasoning chains.
But viewed together, a common pattern emerges—all three papers focus on conventional designs already widely adopted by mainstream models, with few looking back to question them.
Now, these 'conventional' practices are also being re-examined.
This article is from the WeChat public account 'QbitAI', author: Follow Frontier Technology
