ByteDance Discovers Why DeepSeek's Performance Fluctuates

marsbitPublished on 2026-10-09Last updated on 2026-10-09

Abstract

ByteDance's Seed team has discovered a significant cause of the seemingly erratic performance of DeepSeek-V4 models. They identified a phenomenon called "Phase Sensitivity," where the model's performance on a task varies cyclically based on the position of key information relative to a compression window boundary, even if the input context remains semantically identical. The core issue is linked to DeepSeek-V4's long-context optimization technique: Chunked KV Cache Compression. This method groups tokens into chunks (e.g., every 4 tokens for V4, 2 for V4.1) to compress Key-Value memory for efficiency. However, researchers found the model develops systematic biases—called Phase Specialization—in how it encodes and retrieves information from different positions (phases) within these chunks. For example, in a code completion task, padding the input with a few extra irrelevant tokens could cause the model's answer to flip between correct and incorrect every 4 tokens. In a 128K-token needle-in-a-haystack retrieval test, accuracy gaps between different phases reached up to 40.2 percentage points. Controlled experiments training models with various compression schemes confirmed this cyclical performance fluctuation is intrinsically tied to the chunk compression step size, not to factors like positional encoding. While post-training and newer model versions (e.g., DeepSeek-V4.1) have reduced the performance gap, the underlying cyclical bias persists. The findings suggest that evalua...

DeepSeek's "Jekyll and Hyde" duality has been caught by ByteDance's Seed team.

The same problem, with nothing changed but a few irrelevant characters added in front, and the model suddenly fails???

And it's not just an occasional hiccup.

Seed researchers found that DeepSeek-V4's performance actually changes periodically every 4 tokens depending on the input position of the information.

What does this mean? Whether the model remembers something actually depends on *where* that information is located in the input??

Add two tokens in front, the answer might change from wrong to right; add two more, it flips back again.

Great, now models need to worry about positioning too.

In 128K long-context retrieval tests, researchers found that simply moving the same piece of information to a different position could cause the retrieval accuracy of DeepSeek-V4 series models to differ by up to 40.2 percentage points.

The team further discovered this issue is related to a long-context optimization technique adopted by DeepSeek-V4—

Chunked KV Cache Compression.

This technology was originally intended to make the model more memory-efficient and faster when processing long text.

But after compression, it ended up making the model perform inconsistently (doge).

With Two Extra Tokens, DeepSeek Suddenly Gets It Right

The ByteDance Seed researchers initially experimented with DeepSeek's own code.

The test subject was DeepSeek-V4-Flash-Base.

They took a snippet of FP8 quantization function from DeepSeek-V4's official inference code and asked the model to complete the last token.

The correct answer should be 8, as this code segment needed to complete a type conversion related to FP8.

But sometimes the model insisted it should be 32.

To figure out what was going on, the researchers added a purely decorative docstring in front of the code, containing some repeated equal signs.

Then, they started adjusting the number of equal signs.

The code itself wasn't changed, the position to be completed wasn't changed, the correct answer certainly wasn't changed... the only variation was the few meaningless tokens added in front.

The result? DeepSeek's answer started flipping back and forth.

When the padding length fell at certain positions, the model was more inclined to answer incorrectly with 32.

Move forward one or two tokens, it started leaning towards the correct answer, 8.

Continue moving, and the wrong answer came back—

The entire process repeated with a period of 4 tokens.

More specifically, among the 16 padding lengths tested in the paper, when the length modulo 4 had a remainder of 0 or 1, the model favored the wrong answer 32; with a remainder of 2 or 3, it favored the correct answer 8.

The researchers also tallied the probabilities the model assigned to the two candidate answers.

At one set of positions, the average probability for the wrong answer 32 reached 71.3%, while the correct answer 8 only got 26.4%.

Switch to another set of positions, and the situation directly reversed:

The average probability for the correct answer 8 rose to 91.5%, leaving the wrong answer 32 with only 7.2%.

In other words, a change in the length of a few irrelevant characters in front was enough to make the model produce completely different judgments for the same problem.

This is a bit hard to assess.

When debugging code, programmers usually check logic, variables, and dependencies first.

Now it seems they might also need to check if they typed two extra equal signs in front.

However, a single code completion case isn't enough to show how widespread the problem is.

So the Seed team expanded their testing.

They turned their attention to a classic task in large language model long-context capabilities: Needle-in-a-Haystack.

The researchers constructed a 128K-token-long context containing approximately 16,000 key-value pairs.

For example, K1 corresponds to V1, K2 to V2...

Then they asked the model to find the Value corresponding to a specified Key.

During testing, the key-value relationships remained unchanged, the questions were the same, and the total context length was consistent.

The researchers focused on adjusting the position of the target information relative to the compression window boundaries.

They found that the accuracy curve of the DeepSeek-V4 series actually showed very明显的明显的 (obvious) periodic起伏 (fluctuations).

Among them, DeepSeek-V4-Flash-Base showed a maximum accuracy gap of 40.2 percentage points between different positions.

DeepSeek-V4-Pro-Base also reached 34.8 percentage points.

After post-training, the situation improved somewhat.

The gap for DeepSeek-V4-Flash-0731 narrowed to 19.1 percentage points, and DeepSeek-V4-Pro-0813 narrowed to 14.8 percentage points.

The newer DeepSeek-V4.1-Flash-0910 further reduced the gap to 6.1 percentage points.

But periodic differences still exist.

How did this disparity arise?

Looking closer, DeepSeek-V4's fluctuation period is 4 tokens, while DeepSeek-V4.1's changed to 2 tokens.

The researchers found this corresponds exactly to the KV Cache compression stride used by each generation of the model.

Well then, even the fluctuation period of answer performance matches the underlying compression configuration.

The Problem Lies in KV Cache Compression?

Let's talk about KV Cache compression then.

When large models process long contexts, they need to store Key and Value information corresponding to a large number of historical tokens for subsequent attention calculations.

The longer the context, the more memory and computational overhead this cache occupies.

Especially for long tasks involving tens or hundreds of thousands of tokens, KV Cache can easily become a bottleneck for inference efficiency.

Therefore, DeepSeek-V4 adopted Chunked KV Cache Compression.

The idea is to divide consecutive tokens into windows and then compress the information within each window into fewer cache entries.

This way, the model doesn't need to preserve cache of the same scale for every historical token.

It saves memory and reduces the cost of long-context attention calculations.

But the Seed team discovered that the reason for DeepSeek's "Jekyll and Hyde" performance might be hidden in this chunking process.

Suppose every 4 tokens form one compression stride.

Then, if the same piece of information appears at the 1st, 2nd, 3rd, or 4th position within the window, the conditions during compression might differ.

The paper calls this position relative to the compression window boundary the Phase.

Researchers found systematic differences in the model's retrieval ability for information in different phases.

They named this phenomenon Phase Sensitivity.

To give an analogy, giving the same document to the model:

In the first layout, the key number happens to land in a position the model finds easy to retain;

In the second layout, just a few extra words are added in front, changing the key number's position relative to the compression window.

The document is still the same document, but whether the model can later find the number might show a significant difference.

Moreover, this problem can't be simply attributed to "the information happened to be split between two windows."

Researchers found that even when both the Key and Value fall within the same compression window, retrieval accuracy can still vary greatly depending on their specific positions.

This indicates the issue also involves how the model writes information into the compressed cache and how it reads from it later.

To confirm further, the Seed team simply trained a batch of models from scratch.

Using the Qwen3-0.6B architecture as a base, they constructed various KV Cache compression schemes and set up a full-attention model without chunked compression as a control.

The focus was only on changing the compression mechanism to see if periodic fluctuations would appear accordingly.

The result: All chunked compression models tested exhibited periodic changes corresponding to their compression stride.

Conversely, the full-attention baseline model did not show the same level of periodicity.

The researchers also separately adjusted window size and compression stride, finding that the period mainly changed with the compression stride.

Stride of 4, performance fluctuated with a period of about 4 tokens;

Stride of 6, the period also became about 6 tokens;

Stride of 8, same thing;

...

Even without using RoPE position encoding, or replacing learnable compression weights with simple averaging, this phenomenon persisted.

In other words, the problem isn't caused solely by a particular position encoding or a special module.

The design of chunked compression itself may introduce periodic retrieval weaknesses.

The Seed team further intervened on attention heads, observing how the model's retrieval ability across different phases changed after removing different components.

They found that different attention heads contributed differently to different phases.

Some heads were better at processing information at certain positions, while others played a larger role at other positions.

The researchers termed this Phase Specialization.

That is, the model's internals seem to have formed a division of labor: different attention components developed preferences for different positions within the compression window.

This division of labor can help the model perform information retrieval, but it may also make certain positions relatively weak links.

The paper also analyzed the training process using a simplified theoretical model, finding that gradient flow might drive compression modules to form stable positional preferences.

This also explains why this periodicity isn't simple random noise. It might be a natural outcome of the model learning how to compress information.

But such problems might not be directly visible from ordinary benchmarks because conventional evaluations usually aggregate a large number of test results into a single average score.

Suppose the model performs very well at some positions but significantly worse at others; the final calculated average score might still look good.

Therefore, the Seed team suggests that when evaluating models using chunked KV Cache compression, one shouldn't just look at overall retrieval accuracy but also test the same piece of information placed at different compression phases separately.

Even though this position-dependent performance bias exists, the memory and computational costs of long-context reasoning are still there, making compression valuable for practical reasons.

But after saving on cache, the model might no longer treat information from different positions equally.

Of course, judging from this experiment's results, post-training and architectural iterations can indeed significantly narrow this gap.

Paper address: https://arxiv.org/pdf/2609.36322

This article is from the WeChat public account "QbitAI", author: Wen Le

Trending Cryptos

Related Questions

QAccording to the ByteDance Seed team's research, what is the primary technical reason behind DeepSeek-V4's inconsistent performance?

AThe inconsistent performance, termed 'Phase Sensitivity', is primarily linked to DeepSeek-V4's implementation of 'Chunked KV Cache Compression', a technique for long-context processing. The model's ability to retrieve information varies based on the information's position relative to the compression window boundaries.

QWhat specific pattern did researchers observe in DeepSeek-V4's performance when irrelevant tokens were added before a task?

AResearchers observed a cyclical pattern where the model's performance, specifically its tendency to give a correct or incorrect answer, changed periodically. For DeepSeek-V4, this fluctuation repeated every 4 tokens, directly corresponding to its KV Cache compression stride length.

QIn the 'Needle in a Haystack' test, what was the maximum accuracy gap observed for DeepSeek-V4-Flash-Base when key information was placed in different positions?

AIn the 128K-token 'Needle in a Haystack' retrieval test, the maximum accuracy gap for DeepSeek-V4-Flash-Base reached 40.2 percentage points depending on the target information's position within the compression window.

QWhat internal mechanism did the researchers identify as 'Phase Specialization' within the model's architecture?

A'Phase Specialization' refers to the phenomenon where different attention heads within the model develop specific preferences or specializations for processing information from different positions (phases) within the compression chunk, leading to uneven retrieval capabilities across positions.

QHow did the performance of later DeepSeek model versions (V4.1) compare to V4 regarding this 'Phase Sensitivity' issue?

ALater versions showed significant improvement. For instance, DeepSeek-V4.1-Flash-0911 reduced the maximum accuracy gap to 6.1 percentage points, down from 40.2 in V4-Flash-Base. However, a cyclical pattern still existed, but with a period of 2 tokens, matching its updated compression stride.

Related Reads

MSCI Confirms Removal of Strategy and Metaplanet From Stock Indexes

MSCI has confirmed that Strategy (NASDAQ: MSTR) and Japan's Metaplanet will be removed from its equity indexes following its November 2026 review, effective at the market close on November 30. Strategy will leave the MSCI USA Index, while Metaplanet will be removed from the MSCI Japan Small Cap Index. The decision follows MSCI's adoption of new eligibility rules for companies classified as non-operating businesses, including firms whose assets are largely held as investments rather than used in traditional operations. Four other companies were also identified for deletion under the screening: UK-based Yellow Cake, Iceland's Skel Fjárfestingafélag, UK-listed Agronomics, and US-based Upexi. Another 19 companies were placed on a watchlist but not confirmed for removal. MSCI's methodology examines operating assets and five financial measures, including operating expenses, cash flow, changes in the value of non-operating assets, and dependence on external financing. Companies whose operating assets exceed 50% of total assets pass the initial test; those below face further screening, and existing index members can be removed if they fail the required tests across their two most recent annual reporting periods. The decision drew attention because Strategy and Metaplanet have built large Bitcoin reserves as core corporate strategies. Strategy previously challenged MSCI's proposed rules, arguing that treating digital asset treasury companies as non-operating businesses was discriminatory. The removals could affect demand for their shares, as index-tracking funds generally adjust holdings when companies leave benchmarks, potentially creating selling pressure around November 30. Strategy warned its exclusion could lead to roughly $2.8 billion in share sales by index-tracking funds, though this remains an estimate. JPMorgan estimated potential selling could reach $8.8 billion if other index providers follow MSCI's approach. The decision does not require either company to sell its Bitcoin holdings; it changes their index eligibility, not their ownership of digital assets.

TheNewsCrypto42m ago

MSCI Confirms Removal of Strategy and Metaplanet From Stock Indexes

TheNewsCrypto42m ago

The 10 People Who Decide Jensen Huang's Fate: A Power Map of Nvidia's Board

The article discusses the uncertainty surrounding Jensen Huang's future at Nvidia amid a scandal, and profiles the 10 board members who will decide his fate. The board includes: 1. Stephen C. Neal – Joined 2019; honorary chairman and senior counsel at Cooley LLP, Nvidia's longtime legal counsel; Harvard and Stanford Law graduate. 2. Tench Coxe – Billionaire VC and early Nvidia investor; board member since 1993; former managing director at Sutter Hill Ventures. 3. John O. Dabiri – Joined 2020; Caltech professor of aeronautics and mechanical engineering; National Medal of Science recipient. 4. Dawn Hudson – Board member since 2013; former NFL CMO and CEO of Pepsi-Cola North America. 5. Harvey C. Jones – Joined 1993; former Synopsys CEO; managing partner at Square Wave Ventures. 6. Melissa B. Lora – Joined 2023; former president of Taco Bell International; praised for financial expertise. 7. A. Brooke Seawell – Joined 1997; partner at New Enterprise Associates; former NetDynamics executive. 8. Aarti Shah – Joined 2020; former Eli Lilly SVP and chief information and digital officer; PhD in applied statistics. 9. Mark A. Stevens – Board member 1993–2006 and again since 2008; managing partner at S-Cubed Capital; former Sequoia Capital managing partner. 10. Suzanne Nora Johnson – Newest member, joined July; former Goldman Sachs vice chairman with 21 years at the firm. The board's decision will significantly impact Nvidia's future.

Odaily星球日报48m ago

The 10 People Who Decide Jensen Huang's Fate: A Power Map of Nvidia's Board

Odaily星球日报48m ago

Trading

Spot

Hot Articles

How to Buy T

Welcome to HTX.com! We've made purchasing Threshold Network Token (T) simple and convenient. Follow our step-by-step guide to embark on your crypto journey.Step 1: Create Your HTX AccountUse your email or phone number to sign up for a free account on HTX. Experience a hassle-free registration journey and unlock all features.Get My AccountStep 2: Go to Buy Crypto and Choose Your Payment MethodCredit/Debit Card: Use your Visa or Mastercard to buy Threshold Network Token (T) instantly.Balance: Use funds from your HTX account balance to trade seamlessly.Third Parties: We've added popular payment methods such as Google Pay and Apple Pay to enhance convenience.P2P: Trade directly with other users on HTX.Over-the-Counter (OTC): We offer tailor-made services and competitive exchange rates for traders.Step 3: Store Your Threshold Network Token (T)After purchasing your Threshold Network Token (T), store it in your HTX account. Alternatively, you can send it elsewhere via blockchain transfer or use it to trade other cryptocurrencies.Step 4: Trade Threshold Network Token (T)Easily trade Threshold Network Token (T) on HTX's spot market. Simply access your account, select your trading pair, execute your trades, and monitor in real-time. We offer a user-friendly experience for both beginners and seasoned traders.

15.0k Total ViewsPublished 2024.03.29Updated 2026.06.02

How to Buy T

Discussions

Welcome to the HTX Community. Here, you can stay informed about the latest platform developments and gain access to professional market insights. Users' opinions on the price of T (T) are presented below.

活动图片