For a long time, in the Transformer architecture, the Attention mechanism and the Feed-Forward Network (FFN) have been regarded as two distinct and complementary modules:
The former facilitates interaction between tokens, while the latter applies non-linear transformations to each token.
However, expanding the formula of the attention mechanism reveals an often-overlooked connection: Standard attention can also be written as an FFN—it's just that its weights are generated from the current input, and its hidden layer width grows with the sequence length.
Thus, the cost of long contexts gets another explanation: we are not only processing more tokens but also running an ever-widening network. So, can seemingly different technical approaches like linear attention, mixture-of-experts attention, and Top-k attention be re-understood from the perspective of 'how to organize this network'?
Addressing this question, the team from Beijing University of Posts and Telecommunications proposed Mixture-of-Top-k Attention (MiTA), which develops the existing Fast Weights perspective into a five-dimensional classification framework. Based on this, they designed deformable and reusable Top-k experts that both retain global information and read raw details on demand. This work has been accepted by NeurIPS 2026, and the code is open-sourced.

Paper:https://arxiv.org/pdf/2602.01219
Code:https://github.com/QishuaiWen/MiTA
Attention is Also FFN, The Key Lies in Where the Weights Come From
Let query

, keys and values form

column-wise.
Comparing the common two-layer Multi-Layer Perceptron (MLP) used in FFNs with standard attention, where the hidden layer width of the former is denoted as D:

Expanding the Softmax, absorbing the scaling and query-dependent normalization into the weights:

The correspondence emerges:

The query corresponds to the input, the transformed keys and values correspond to the two-layer weights, the exponential function corresponds to the activation function, the bias is zero, and the hidden layer width changes from D to the sequence length N.
The difference lies in the source of the weights: ordinary FFNs use trained Slow Weights, while attention uses K, V generated from the current input as context-dependent Fast Weights.
A more important difference lies in width: each key-value pair corresponds to a hidden unit; the longer the context, the wider this fast-weight MLP becomes. Each query accesses N hidden units, processing N queries results in an overall cost of

.
The design of efficient attention thus transforms into a unified question: How to keep the ever-expanding fast-weight network scalable?
Understanding Compression and Routing from an Ever-Widening MLP
Facing this ever-widening network, the main ideas of efficient attention can be summarized into two approaches.
Compression: Turn the wide network into compact modules (Scaling by Compression). Linear Attention compresses key-value information into shared linear layers, Linformer maps them into a narrower MLP, and Test-Time Training (TTT) learns compact fast-weight modules. This approach provides global summarization but struggles to directly read the details of raw key-value pairs.
Routing: Retain raw content, only access needed parts (Scaling by Routing). Organize subsets of key-value pairs into fast-weight experts and let queries select on demand. These experts are constructed from key-value pairs in the current context: they retain the details of selected content but lack summarization of the full context.

Figure 1: Context growth widens the fast-weight MLP. Routing selects a part of it, while compression constructs a more compact representation.
These two approaches are not mutually exclusive. Further considering Scaling Strategy, Expert Count, Expert Type, Expert Construction, and Routing Topology, a five-dimensional classification framework is derived.

Table 1: Five-dimensional classification framework for efficient attention from the fast-weight perspective.
This framework not only organizes existing methods but also raises specific questions for new designs: What information constructs the experts? How many are needed? Which queries can reuse the same expert?
MiTA's entry point: Must Top-k selections be individual for each query?
Taking block-based Mixture-of-Experts Attention (MoBA) as an example, the common process is: first, cut the sequence into consecutive, non-overlapping blocks; then, generate routing vectors for each block, letting queries select as needed.
Routing can vary per query, but expert boundaries are predetermined. In images, information relevant to the same target may not fall within a single rectangular block; to read a small amount of related content, a query might need to access the entire block.
Top-k attention goes in another direction: each query individually selects the k most relevant key-value pairs, free from fixed block shapes. But correspondingly, N queries also correspond to N 'private' sets. Different queries, even focusing on similar content, must still perform selection and organization separately.
Can we retain the flexibility of Top-k selection while letting the selection results be reused by multiple queries?
MiTA reverses this logic: instead of predefining expert content, first extract a small number of representative queries, which then select content from the full context to form adjustable-number experts. Original queries only need to choose which expert to access, rather than constructing their own global Top-k sets.

Figure 2: Full attention shares all key-value pairs, block-based MoE shares fixed blocks, per-query Top-k constructs private sets, while MiTA constructs reusable deformable experts.
Using a Few Landmark Queries to Simultaneously Construct Global Summaries and Detail Experts
MiTA extracts m Landmark Queries through local average pooling. Each landmark probes the full sequence, simultaneously accomplishing global information summarization and raw detail retrieval.

Figure 3: Each query accesses both compressed global key-value pairs and routed original key-value pairs.
First, construct reusable Top-k experts. The i-th landmark query

matches with all keys, selecting the k positions with the highest relevance:

These key-value pairs form the i-th routed expert, which can span different spatial positions, change shape with input content, and different experts are allowed to overlap.
Subsequently, a query selects the expert corresponding to the landmark that best matches it:

Different queries share candidate key-value sets but still independently compute their own attention weights.
Then, use the same set of landmarks to construct a global shared expert. Landmark queries read the full context via cross-attention, obtaining corresponding landmark values:

Using landmark queries as keys and landmark values as values yields globally shared key-value pairs

.
MiTA thus contains m routed experts and 1 shared expert: the former provides raw details, the latter provides global summarization.
Finally, let summarization and details interact within the same attention step. Under the configuration where each query selects one routed expert, i.e., s=1:

The two sets of key-value pairs are concatenated, and a unified Softmax normalization is applied. Top-k selection and global summarization share landmark scores, and query-to-landmark scores can also be reused for routing. Compression and routing not only complement each other functionally but also reuse computations.
Taking ImageNet-1K with m=k=25 as an example, each query ultimately accesses 25 compressed key-value pairs and 25 original key-value pairs, totaling 50. The work of scanning the full sequence and constructing Top-k sets is centralized by a few landmarks.

Algorithm 1: First construct experts, then compute by grouping queries according to routing results, merging results via Online Softmax.
The final query-expert attention cost is reduced from

to

, and is accelerated via FlashAttention and grouped computation by expert. Let's further examine task performance and actual runtime.
Experimental Results
Image Classification: A Base Mechanism Close to Full Attention
On ImageNet-1K, first using the same DeiT training configuration, attention mechanisms without additional components are compared.

Table 2: Classification results on DeiT-T and DeiT-S. DWC denotes depth-wise convolution, Gate denotes gating mechanism.
On DeiT-S, the base MiTA achieves a Top-1 accuracy of 79.8%, differing from full attention's 79.9% by 0.1 percentage point; parameter count remains unchanged, and FLOPs drop from 4.6G to 4.4G. Under the same setup, Agent Attention scores 78.5%.
After adding depth-wise convolution (DWC), MiTA reaches 80.6%; further adding gating brings it to 81.2%. These extra accuracy-boosting techniques commonly used in visual efficient attention are transparently demonstrated by MiTA.

Table 3: Results combined with the ViT-5 architecture design, and comparison with other efficient vision Transformers.
Combined with the ViT-5 architecture design and DWC, MiTA-ViT-5-S achieves 81.7%, differing from ViT-5-S's 82.2% by 0.5 percentage points, while FLOPs drop from 4.7G to 4.5G.
Semantic Segmentation: Reducing Computation on Longer Visual Sequences
In ADE20K semantic segmentation, the attention in pre-trained ViT is replaced with MiTA, and then the segmentation model is fine-tuned. With m=k=49, each query accesses only 98 key-value pairs, while full attention accesses 1,024 or 1,600 key-value pairs under the two input resolutions respectively.

Table 4: ADE20K segmentation results after replacing ViTs of different scales with MiTA.
Taking ViT-S+CBT as an example, FLOPs drop from 33G to 25G, a reduction of about 24%, and mean Intersection over Union (mIoU) changes from 45.8% to 44.5%. For ViT-T configuration, FLOPs are reduced by about 42%, with mIoU dropping by 2.6 percentage points.
Long Sequence Modeling: Training Time Reduced from 10.6 Hours to 2.4 Hours
On the five tasks of the Long Range Arena (LRA) benchmark, MiTA achieves an average accuracy of 58.91%, while full attention scores 58.19%; total training time decreases from 10.6 hours to 2.4 hours, a reduction of about 77%.

Table 5: Accuracy reported per task and training throughput (steps/s). The last column shows average accuracy and total training time (hours).
Testing was conducted on a single 24GB RTX 4090. The full attention baseline uses FlashAttention.
The compression branch also reduces the gather overhead of original key-value pairs: in an ablation study where the number of accessed entries per query is kept the same, adding the compression branch reduces training time from 5.5 hours to 2.4 hours compared to the routing-only version, and improves average accuracy from 58.81% to 58.91%.

Figure 4: Inference throughput (Throughput) on RTX 4090 for a Transformer with three layers and embedding dimension 128. MiTA uses m*k≈N/4; batch size is adjusted per configuration to maximize throughput.
As sequence length increases, the inference throughput advantage further expands: at lengths of 64K, 256K, and 1024K, the speedup compared to full attention is approximately 19x, 56x, and 159x respectively.
Long Document Classification: Balancing Overall and Detailed Understanding for 16K—64K Texts
arXiv Long Document Classification requires identifying the category of lengthy papers, needing both overall understanding and capturing fine-grained clues scattered throughout the text, perfectly testing the combination of global summarization and detail retrieval.

Table 6: arXiv long document classification results (%).
At lengths of 16K, 32K, and 64K, MiTA achieves accuracies of 88.0%, 92.9%, and 98.7% respectively, leading over other methods in the table; compared to standard attention, this is an improvement of 18.2, 3.2, and 1.7 percentage points respectively.
Two Other Noteworthy Phenomena
First phenomenon: Without extra pruning modules, it gradually focuses on the target. As the network deepens, the key-value positions selected into routed experts gradually concentrate on category-relevant regions, with background positions being selected less and less, showing an emergent token-pruning effect.

Figure 5: Far left is the input image, other columns show key-value positions selected into experts per layer, aggregated across attention heads. Later layers gradually focus on category-relevant regions.
Background tokens not selected can still participate in global summarization via the compression branch.
Second phenomenon: Weights can be reused, and inference budget can be adjusted. After training, freezing model parameters and only changing the attention mechanism during inference, MiTA and full attention show good mutual compatibility.

Figure 6: Classification accuracy (%) when using different attention mechanisms during training and inference with fixed model parameters. Left and right are DeiT-T and DeiT-S respectively.
Furthermore, replacing the attention in ViT pre-trained with full attention with MiTA and fine-tuning on ImageNet-1K, ViT-S and ViT-L achieve 80.9% and 85.5% respectively, only 0.3 and 0.4 percentage points behind full attention.
Additionally, keeping model parameters unchanged and only adjusting the number of experts m and expert width k during inference, suitable configurations can further improve accuracy. This Algorithmic Generalization offers the possibility of controlling costs during training and adjusting budgets on demand during inference.

Figure 7: MiTA accuracy under different m, k. The red box marks the training configuration, orange borders circle inference settings with accuracy higher than the original configuration.
From 'What is Attention' to 'How Can Attention Be Designed Further'
Returning to the initial question: The value of 'Attention is also FFN' is not merely that two formulas can correspond, but that the efficiency problem of attention thereby gains a new set of design dimensions.
Compression decides how to retain global information, expert construction decides which raw content is worth organizing together, and routing decides which queries can reuse this content. These problems studied, designs attempted, and experiences accumulated in FFNs can all feed back into the further development of attention mechanisms, perhaps one day even breaking the absolute boundary between Attention and FFN.
This work provides not just an attention variant, but also a vision pointing to the underlying design beliefs of the architecture.
References:
https://arxiv.org/pdf/2602.01219
This article is from the WeChat public account "New Zhiyuan", author: New Zhiyuan





