NeurIPS'26: Is Attention Also FFN? Understanding Efficient Attention from the Perspective of Fast Weights

marsbitPublished on 2026-10-12Last updated on 2026-10-12

Abstract

NeurIPS '26: Attention as a Fast-Weight FFN? Revisiting Efficient Attention through Fast Weights. This work challenges the traditional separation between the Attention and Feed-Forward Network (FFN) modules in Transformers. It demonstrates that standard self-attention can be reformulated as a two-layer FFN whose weights (Fast Weights) are dynamically generated from the input, with a hidden layer width that grows linearly with sequence length. This perspective reframes the quadratic complexity of attention as operating an increasingly wide network. From this "Fast-Weight" viewpoint, the paper proposes a five-dimensional taxonomy (Scaling Strategy, Expert Count, Expert Type, Expert Construction, Routing Topology) to unify and analyze efficient attention methods. These methods are seen as either compressing the wide network (e.g., linear attention) or routing queries to relevant subsets of it (e.g., top-k attention). Building on this framework, the authors introduce Mixture-of-Top-k Attention (MiTA). MiTA first extracts a small set of Landmark Queries via pooling. Each landmark constructs one reusable "Top-k expert" by selecting the most relevant K/V pairs from the *full* context, and also contributes to a single, shared "compressed expert" via cross-attention for global context. All original queries then route to one of these experts. This design allows queries to access both precise local details (via routed experts) and global summaries (via the shared expert) within a sin...

For a long time, in the Transformer architecture, the Attention mechanism and the Feed-Forward Network (FFN) have been regarded as two distinct and complementary modules:

The former facilitates interaction between tokens, while the latter applies non-linear transformations to each token.

However, expanding the formula of the attention mechanism reveals an often-overlooked connection: Standard attention can also be written as an FFN—it's just that its weights are generated from the current input, and its hidden layer width grows with the sequence length.

Thus, the cost of long contexts gets another explanation: we are not only processing more tokens but also running an ever-widening network. So, can seemingly different technical approaches like linear attention, mixture-of-experts attention, and Top-k attention be re-understood from the perspective of 'how to organize this network'?

Addressing this question, the team from Beijing University of Posts and Telecommunications proposed Mixture-of-Top-k Attention (MiTA), which develops the existing Fast Weights perspective into a five-dimensional classification framework. Based on this, they designed deformable and reusable Top-k experts that both retain global information and read raw details on demand. This work has been accepted by NeurIPS 2026, and the code is open-sourced.

Paper:https://arxiv.org/pdf/2602.01219

Code:https://github.com/QishuaiWen/MiTA

Attention is Also FFN, The Key Lies in Where the Weights Come From

Let query

, keys and values form

column-wise.

Comparing the common two-layer Multi-Layer Perceptron (MLP) used in FFNs with standard attention, where the hidden layer width of the former is denoted as D:

Expanding the Softmax, absorbing the scaling and query-dependent normalization into the weights:

The correspondence emerges:

The query corresponds to the input, the transformed keys and values correspond to the two-layer weights, the exponential function corresponds to the activation function, the bias is zero, and the hidden layer width changes from D to the sequence length N.

The difference lies in the source of the weights: ordinary FFNs use trained Slow Weights, while attention uses K, V generated from the current input as context-dependent Fast Weights.

A more important difference lies in width: each key-value pair corresponds to a hidden unit; the longer the context, the wider this fast-weight MLP becomes. Each query accesses N hidden units, processing N queries results in an overall cost of

.

The design of efficient attention thus transforms into a unified question: How to keep the ever-expanding fast-weight network scalable?

Understanding Compression and Routing from an Ever-Widening MLP

Facing this ever-widening network, the main ideas of efficient attention can be summarized into two approaches.

Compression: Turn the wide network into compact modules (Scaling by Compression). Linear Attention compresses key-value information into shared linear layers, Linformer maps them into a narrower MLP, and Test-Time Training (TTT) learns compact fast-weight modules. This approach provides global summarization but struggles to directly read the details of raw key-value pairs.

Routing: Retain raw content, only access needed parts (Scaling by Routing). Organize subsets of key-value pairs into fast-weight experts and let queries select on demand. These experts are constructed from key-value pairs in the current context: they retain the details of selected content but lack summarization of the full context.

Figure 1: Context growth widens the fast-weight MLP. Routing selects a part of it, while compression constructs a more compact representation.

These two approaches are not mutually exclusive. Further considering Scaling Strategy, Expert Count, Expert Type, Expert Construction, and Routing Topology, a five-dimensional classification framework is derived.

Table 1: Five-dimensional classification framework for efficient attention from the fast-weight perspective.

This framework not only organizes existing methods but also raises specific questions for new designs: What information constructs the experts? How many are needed? Which queries can reuse the same expert?

MiTA's entry point: Must Top-k selections be individual for each query?

Taking block-based Mixture-of-Experts Attention (MoBA) as an example, the common process is: first, cut the sequence into consecutive, non-overlapping blocks; then, generate routing vectors for each block, letting queries select as needed.

Routing can vary per query, but expert boundaries are predetermined. In images, information relevant to the same target may not fall within a single rectangular block; to read a small amount of related content, a query might need to access the entire block.

Top-k attention goes in another direction: each query individually selects the k most relevant key-value pairs, free from fixed block shapes. But correspondingly, N queries also correspond to N 'private' sets. Different queries, even focusing on similar content, must still perform selection and organization separately.

Can we retain the flexibility of Top-k selection while letting the selection results be reused by multiple queries?

MiTA reverses this logic: instead of predefining expert content, first extract a small number of representative queries, which then select content from the full context to form adjustable-number experts. Original queries only need to choose which expert to access, rather than constructing their own global Top-k sets.

Figure 2: Full attention shares all key-value pairs, block-based MoE shares fixed blocks, per-query Top-k constructs private sets, while MiTA constructs reusable deformable experts.

Using a Few Landmark Queries to Simultaneously Construct Global Summaries and Detail Experts

MiTA extracts m Landmark Queries through local average pooling. Each landmark probes the full sequence, simultaneously accomplishing global information summarization and raw detail retrieval.

Figure 3: Each query accesses both compressed global key-value pairs and routed original key-value pairs.

First, construct reusable Top-k experts. The i-th landmark query

matches with all keys, selecting the k positions with the highest relevance:

These key-value pairs form the i-th routed expert, which can span different spatial positions, change shape with input content, and different experts are allowed to overlap.

Subsequently, a query selects the expert corresponding to the landmark that best matches it:

Different queries share candidate key-value sets but still independently compute their own attention weights.

Then, use the same set of landmarks to construct a global shared expert. Landmark queries read the full context via cross-attention, obtaining corresponding landmark values:

Using landmark queries as keys and landmark values as values yields globally shared key-value pairs

.

MiTA thus contains m routed experts and 1 shared expert: the former provides raw details, the latter provides global summarization.

Finally, let summarization and details interact within the same attention step. Under the configuration where each query selects one routed expert, i.e., s=1:

The two sets of key-value pairs are concatenated, and a unified Softmax normalization is applied. Top-k selection and global summarization share landmark scores, and query-to-landmark scores can also be reused for routing. Compression and routing not only complement each other functionally but also reuse computations.

Taking ImageNet-1K with m=k=25 as an example, each query ultimately accesses 25 compressed key-value pairs and 25 original key-value pairs, totaling 50. The work of scanning the full sequence and constructing Top-k sets is centralized by a few landmarks.

Algorithm 1: First construct experts, then compute by grouping queries according to routing results, merging results via Online Softmax.

The final query-expert attention cost is reduced from

to

, and is accelerated via FlashAttention and grouped computation by expert. Let's further examine task performance and actual runtime.

Experimental Results

Image Classification: A Base Mechanism Close to Full Attention

On ImageNet-1K, first using the same DeiT training configuration, attention mechanisms without additional components are compared.

Table 2: Classification results on DeiT-T and DeiT-S. DWC denotes depth-wise convolution, Gate denotes gating mechanism.

On DeiT-S, the base MiTA achieves a Top-1 accuracy of 79.8%, differing from full attention's 79.9% by 0.1 percentage point; parameter count remains unchanged, and FLOPs drop from 4.6G to 4.4G. Under the same setup, Agent Attention scores 78.5%.

After adding depth-wise convolution (DWC), MiTA reaches 80.6%; further adding gating brings it to 81.2%. These extra accuracy-boosting techniques commonly used in visual efficient attention are transparently demonstrated by MiTA.

Table 3: Results combined with the ViT-5 architecture design, and comparison with other efficient vision Transformers.

Combined with the ViT-5 architecture design and DWC, MiTA-ViT-5-S achieves 81.7%, differing from ViT-5-S's 82.2% by 0.5 percentage points, while FLOPs drop from 4.7G to 4.5G.

Semantic Segmentation: Reducing Computation on Longer Visual Sequences

In ADE20K semantic segmentation, the attention in pre-trained ViT is replaced with MiTA, and then the segmentation model is fine-tuned. With m=k=49, each query accesses only 98 key-value pairs, while full attention accesses 1,024 or 1,600 key-value pairs under the two input resolutions respectively.

Table 4: ADE20K segmentation results after replacing ViTs of different scales with MiTA.

Taking ViT-S+CBT as an example, FLOPs drop from 33G to 25G, a reduction of about 24%, and mean Intersection over Union (mIoU) changes from 45.8% to 44.5%. For ViT-T configuration, FLOPs are reduced by about 42%, with mIoU dropping by 2.6 percentage points.

Long Sequence Modeling: Training Time Reduced from 10.6 Hours to 2.4 Hours

On the five tasks of the Long Range Arena (LRA) benchmark, MiTA achieves an average accuracy of 58.91%, while full attention scores 58.19%; total training time decreases from 10.6 hours to 2.4 hours, a reduction of about 77%.

Table 5: Accuracy reported per task and training throughput (steps/s). The last column shows average accuracy and total training time (hours).

Testing was conducted on a single 24GB RTX 4090. The full attention baseline uses FlashAttention.

The compression branch also reduces the gather overhead of original key-value pairs: in an ablation study where the number of accessed entries per query is kept the same, adding the compression branch reduces training time from 5.5 hours to 2.4 hours compared to the routing-only version, and improves average accuracy from 58.81% to 58.91%.

Figure 4: Inference throughput (Throughput) on RTX 4090 for a Transformer with three layers and embedding dimension 128. MiTA uses m*k≈N/4; batch size is adjusted per configuration to maximize throughput.

As sequence length increases, the inference throughput advantage further expands: at lengths of 64K, 256K, and 1024K, the speedup compared to full attention is approximately 19x, 56x, and 159x respectively.

Long Document Classification: Balancing Overall and Detailed Understanding for 16K—64K Texts

arXiv Long Document Classification requires identifying the category of lengthy papers, needing both overall understanding and capturing fine-grained clues scattered throughout the text, perfectly testing the combination of global summarization and detail retrieval.

Table 6: arXiv long document classification results (%).

At lengths of 16K, 32K, and 64K, MiTA achieves accuracies of 88.0%, 92.9%, and 98.7% respectively, leading over other methods in the table; compared to standard attention, this is an improvement of 18.2, 3.2, and 1.7 percentage points respectively.

Two Other Noteworthy Phenomena

First phenomenon: Without extra pruning modules, it gradually focuses on the target. As the network deepens, the key-value positions selected into routed experts gradually concentrate on category-relevant regions, with background positions being selected less and less, showing an emergent token-pruning effect.

Figure 5: Far left is the input image, other columns show key-value positions selected into experts per layer, aggregated across attention heads. Later layers gradually focus on category-relevant regions.

Background tokens not selected can still participate in global summarization via the compression branch.

Second phenomenon: Weights can be reused, and inference budget can be adjusted. After training, freezing model parameters and only changing the attention mechanism during inference, MiTA and full attention show good mutual compatibility.

Figure 6: Classification accuracy (%) when using different attention mechanisms during training and inference with fixed model parameters. Left and right are DeiT-T and DeiT-S respectively.

Furthermore, replacing the attention in ViT pre-trained with full attention with MiTA and fine-tuning on ImageNet-1K, ViT-S and ViT-L achieve 80.9% and 85.5% respectively, only 0.3 and 0.4 percentage points behind full attention.

Additionally, keeping model parameters unchanged and only adjusting the number of experts m and expert width k during inference, suitable configurations can further improve accuracy. This Algorithmic Generalization offers the possibility of controlling costs during training and adjusting budgets on demand during inference.

Figure 7: MiTA accuracy under different m, k. The red box marks the training configuration, orange borders circle inference settings with accuracy higher than the original configuration.

From 'What is Attention' to 'How Can Attention Be Designed Further'

Returning to the initial question: The value of 'Attention is also FFN' is not merely that two formulas can correspond, but that the efficiency problem of attention thereby gains a new set of design dimensions.

Compression decides how to retain global information, expert construction decides which raw content is worth organizing together, and routing decides which queries can reuse this content. These problems studied, designs attempted, and experiences accumulated in FFNs can all feed back into the further development of attention mechanisms, perhaps one day even breaking the absolute boundary between Attention and FFN.

This work provides not just an attention variant, but also a vision pointing to the underlying design beliefs of the architecture.

References:

https://arxiv.org/pdf/2602.01219

This article is from the WeChat public account "New Zhiyuan", author: New Zhiyuan

Related Questions

QAccording to the article, how can standard attention be reformulated from the perspective of feed-forward networks (FFNs)?

AStandard attention can be reformulated as a feed-forward network (FFN) where the queries act as the input, the transformed keys and values act as the two-layer weights, the exponential function acts as the activation function, and the hidden layer width grows with the sequence length N, instead of being a fixed value D.

QWhat is the key difference between 'Slow Weights' and 'Fast Weights' in the context of this article's proposed framework?

AThe key difference is their source. 'Slow Weights' are the static, trainable parameters used by a standard FFN. 'Fast Weights' are dynamically generated from the current input (keys and values) and change with the context, as in an attention mechanism.

QWhat are the two main scaling strategies identified for managing the increasingly wide fast-weight network in efficient attention mechanisms?

AThe two main scaling strategies are 'Scaling by Compression', which compresses the wide network into a compact module, and 'Scaling by Routing', which retains original content but only accesses a needed subset.

QWhat is the core innovation of the Mixture-of-Top-k Attention (MiTA) method proposed in the work?

AMiTA's core innovation is constructing a small number of 'Landmark Queries' that simultaneously retrieve a global summary (via a compressed shared expert) and select specific details (via content-based, deformable Top-k experts) from the full context. These experts are then reused by many original queries, combining the flexibility of Top-k selection with computational efficiency.

QWhat practical benefit does MiTA's algorithmic generalization (adjusting m and k at inference) potentially offer, as shown in the results?

AAlgorithmic generalization allows for 'training with a controlled cost and adjusting the budget on-demand during inference'. By keeping model parameters fixed and only adjusting the expert count (m) and expert width (k) at inference time, performance can sometimes be improved, offering a way to trade compute resources for accuracy post-training.

Related Reads

OpenAI and Meta Join Forces, Bridging the "Last Mile" of Agent Commerce

On November 13, 2026, two major announcements landed the same day, together addressing the "last mile" of agent commerce: how AI assistants get through the door, and how they pay. Sierra and Meta released Poppy v1.0, a Personal Agent Protocol built on OAuth, HTTPS, OpenAPI, and the Model Context Protocol. Poppy lets businesses cryptographically verify three things: which assistant is making a request, which user it represents, and what permissions it holds. Companies publish a discovery file at /.well-known/poppy describing endpoints and allowed actions, while both user and business must consent to each interaction via two-way authorization. Shopify automatically enabled Poppy discovery files for merchants in 175+ countries, giving the protocol immediate scale, with over 100 design partners including Delta, 1Password, Okta, and Cloudflare. Separately, OpenAI launched ChatGPT Wallet for its personal assistant dots. The assistant can search, compare, and fill a cart, but must stop at checkout—users confirm via biometric authentication. Neither the assistant nor the merchant ever sees the real card number; card networks issue a one-time payment credential instead. Built with Amex, Mastercard, and Visa, the wallet runs entirely on traditional fiat rails—explicitly not crypto or stablecoins. It initially opens only to US Pro subscribers aged 18+. Notably, Visa and Mastercard span both initiatives, and OpenAI itself appears among Poppy's design partners—despite dots and Meta's Muse competing for the consumer entry point. The shared logic: consumer-facing competition, but shared infrastructure on the business side. The common bottom line is "human in the loop"—agent autonomy stops at the shopping cart. Open questions remain: liability when an assistant errs or is manipulated by malicious prompts, coverage beyond Shopify and US Pro users, and whether mandatory biometric confirmation for every small purchase will become friction that undermines the convenience promise.

marsbit9h ago

OpenAI and Meta Join Forces, Bridging the "Last Mile" of Agent Commerce

marsbit9h ago

Has the Time Come to Buy the Dip on Nvidia?

Nvidia announced on November 12 that Jensen Huang resigned as President, CEO, and Director, effective immediately. Shares fell as much as 14% intraday to $148.8, down nearly 40% from October highs, with over $600 billion in market value briefly erased. The founder's departure triggered the selloff, but the market is repricing the entire company rather than reacting to a single risk. Nvidia now trades at roughly 12x forward PE, its lowest in 15 years and below the S&P 500's ~18x, despite strong fundamentals: FY2026 revenue of $215.9 billion (+65%), data center revenue of $62.3 billion in a single quarter, ~75% gross margins, and Q2 FY2027 revenue of $96.2 billion with $21.3 billion free cash flow. Most investment banks cut targets but maintained Buy/Overweight ratings (Barclays $210, Goldman $220, Citi $225, RBC $230), implying ~20% upside, while Truist downgraded to Hold at $174. The consensus: the business is intact, but its valuation multiple needed resetting. Supporting demand, Huang forecast at GTC 2026 that Blackwell and Rubin AI chip demand would reach at least $1 trillion by 2027, double the prior 2026 estimate. However, risks remain beyond CEO succession. CUDA's lock-in effect is eroding on the inference side—Anthropic disclosed that its Claude Mythos Preview autonomously ported core GPU kernels to AMD flagship GPUs in three days with minimal human intervention, prompting investor Marcelo P. Lima's widely cited claim that "Nvidia's CUDA moat no longer exists." Supply bottlenecks lie in HBM and packaging, not wafers: TSMC can cut over 400 million GPU dies annually, but Nvidia ships only ~10 million GPUs yearly, constrained by HBM capacity (~3.75EB globally, ~70% allocated to Nvidia, ~260GB per GPU) and tight packaging, with HBM4 adjustments possibly delaying Rubin. Huang also leaves behind highly personalized assets—strategic technology judgment, supply chain negotiation with TSMC and SK Hynix, and direct channels to customers and governments—that interim CEO Colette Kress, the long-time CFO, may not fully replace. In short, the business remains, but the "faith premium" is being systematically stripped away. At 12x forward PE, much pessimism is priced in, offering a favorable risk-reward window if AI capex holds and demand materializes. Yet cheapness alone isn't sufficient: CUDA's erosion is slow but irreversible, HBM and packaging limit revenue upside, and succession uncertainty means the founder premium may not be fully removed. As Bernstein noted, "if you see two ants in the kitchen, there are more than two," warning that chains from scandal to governance vacuum to government probes could trigger further selloffs. Until the dust settles, what a Jensen-less Nvidia is worth remains an open question.

marsbit10h ago

Has the Time Come to Buy the Dip on Nvidia?

marsbit10h ago

AI Bull Market Trap? The Big Short Seller Strikes Again

Recent signals from Goldman Sachs, the Federal Reserve, and prominent short-seller Michael Burry are converging on concerns about the AI trade. Goldman warns that AI capital expenditure growth will likely peak in early 2027, with the boost to S&P 500 earnings falling from a 12-point peak in 2026 to a 5-point drag by 2028, shifting the market from a one-way rally to two-way volatility. The Fed has flagged AI as a potential threat to asset valuations and financial stability. Burry has doubled his Nasdaq 100 put options and rebuilt short positions in a semiconductor ETF, calling this a "bull market trap" analogous to the 2000 dot-com bubble, with Nvidia's influence a key worry. Four pressures weigh on the AI narrative: peaking capex growth, high interest rates that hit long-duration AI assets and leveraged GPU cloud providers like CoreWeave and Nebius, circular financing risks among interconnected tech firms, and tighter regulation under Democrats affecting data center energy, land, and permitting. Nvidia-related turmoil has shaken the pricing anchor for the entire supply chain. However, a valuation bubble bursting does not mean the AI industry is declining. Goldman notes US cloud giants trade at just 19.7x forward P/E, below their 20-year historical range and even below defensive staples, suggesting much pessimism is already priced in. Opportunities are shifting from upstream hardware "shovel sellers" toward fairly valued cloud giants with improving free cash flow, AI application companies delivering real ROI, and deeply discounted cyclical names like Korean memory. Conversely, GPU cloud providers dependent on Nvidia orders and cash-burning startups without products are most vulnerable. Investors should diversify, hedge, and focus on assets that convert spending into stable returns.

marsbit10h ago

AI Bull Market Trap? The Big Short Seller Strikes Again

marsbit10h ago

Just Offered 3.3 Million Yuan Salaries to Poach Talent, OpenAI Suddenly Dissolves Robotics Team—Is the Industry About to Change!

OpenAI has disbanded its standalone robotics research team, confirmed shortly after booking booth 208 at the CoRL robotics conference in Austin went empty on November 13. The move surprised many, given that just months earlier OpenAI was aggressively hiring robotics talent—some roles paying up to $500,000 base salary—and had expanded its San Francisco data-collection lab to about 100 workers across three shifts, with a second lab planned in Richmond, California. Researchers from the robotics team are now folded into Chief Research Officer Mark Chen's core research division, while robotics applications and hardware staff move to President Greg Brockman's Applied division. OpenAI says robotics remains part of its long-term vision and the reshuffle is meant to accelerate progress. This is not OpenAI's first robotics retreat: it ended its first independent robotics effort in 2020 to focus on large language models, which produced ChatGPT, and only rebuilt a team in 2024. OpenAI explains the latest pivot technically—researchers initially believed video-generation models like Sora could teach robots physics, but general models such as GPT-6 Astra proved capable of directly controlling robots, so robotics work is being merged into the general foundation-model team. The broader industry context supports this shift. Unitree's Wang Xingxing said his biggest mistake over the past decade was not trusting AI enough, noting GPT-6 integration exceeded expectations and that humanoid intelligence's tipping point may arrive within one to two years. Anthropic's Claude Fable 5.2 scored 64.9% on the RoboDojo Sim benchmark versus 29.8% for GPT-6 Astra, and Nvidia's Astranomy study showed GPT-6 Astra controlling simulated robots with 74.8% success using an agentic approach, beating the prior specialized-model best of 51.0%. Yet challenges remain: general models are costly and slow—GPT-6 Astra averaged 18.56 seconds per inference and about 30.9 million tokens (roughly $36.40 in API fees) per successful task—problematic for real factories where conveyor belts don't pause. Google DeepMind, Nvidia, Unitree and AgiBot continue training specialized robotics models; Google's Apollo 2 achieved 96% success sorting parts in Mercedes factories using paired reasoning and action models, while Nvidia's GR00T 2 reached 95% success after brief fine-tuning. Analysts suggest OpenAI now treats robotics as a downstream application of general intelligence, leveraging core model advances while using its data-collection and hardware capabilities to validate models on real robots.

marsbit10h ago

Just Offered 3.3 Million Yuan Salaries to Poach Talent, OpenAI Suddenly Dissolves Robotics Team—Is the Industry About to Change!

marsbit10h ago

Trading

Spot
活动图片