ByteDance Discovers Why DeepSeek's Performance Fluctuates

marsbitPublished on 2026-10-09Last updated on 2026-10-09

Abstract

ByteDance's Seed team has discovered a significant cause of the seemingly erratic performance of DeepSeek-V4 models. They identified a phenomenon called "Phase Sensitivity," where the model's performance on a task varies cyclically based on the position of key information relative to a compression window boundary, even if the input context remains semantically identical. The core issue is linked to DeepSeek-V4's long-context optimization technique: Chunked KV Cache Compression. This method groups tokens into chunks (e.g., every 4 tokens for V4, 2 for V4.1) to compress Key-Value memory for efficiency. However, researchers found the model develops systematic biases—called Phase Specialization—in how it encodes and retrieves information from different positions (phases) within these chunks. For example, in a code completion task, padding the input with a few extra irrelevant tokens could cause the model's answer to flip between correct and incorrect every 4 tokens. In a 128K-token needle-in-a-haystack retrieval test, accuracy gaps between different phases reached up to 40.2 percentage points. Controlled experiments training models with various compression schemes confirmed this cyclical performance fluctuation is intrinsically tied to the chunk compression step size, not to factors like positional encoding. While post-training and newer model versions (e.g., DeepSeek-V4.1) have reduced the performance gap, the underlying cyclical bias persists. The findings suggest that evalua...

DeepSeek's "Jekyll and Hyde" duality has been caught by ByteDance's Seed team.

The same problem, with nothing changed but a few irrelevant characters added in front, and the model suddenly fails???

And it's not just an occasional hiccup.

Seed researchers found that DeepSeek-V4's performance actually changes periodically every 4 tokens depending on the input position of the information.

What does this mean? Whether the model remembers something actually depends on *where* that information is located in the input??

Add two tokens in front, the answer might change from wrong to right; add two more, it flips back again.

Great, now models need to worry about positioning too.

In 128K long-context retrieval tests, researchers found that simply moving the same piece of information to a different position could cause the retrieval accuracy of DeepSeek-V4 series models to differ by up to 40.2 percentage points.

The team further discovered this issue is related to a long-context optimization technique adopted by DeepSeek-V4—

Chunked KV Cache Compression.

This technology was originally intended to make the model more memory-efficient and faster when processing long text.

But after compression, it ended up making the model perform inconsistently (doge).

With Two Extra Tokens, DeepSeek Suddenly Gets It Right

The ByteDance Seed researchers initially experimented with DeepSeek's own code.

The test subject was DeepSeek-V4-Flash-Base.

They took a snippet of FP8 quantization function from DeepSeek-V4's official inference code and asked the model to complete the last token.

The correct answer should be 8, as this code segment needed to complete a type conversion related to FP8.

But sometimes the model insisted it should be 32.

To figure out what was going on, the researchers added a purely decorative docstring in front of the code, containing some repeated equal signs.

Then, they started adjusting the number of equal signs.

The code itself wasn't changed, the position to be completed wasn't changed, the correct answer certainly wasn't changed... the only variation was the few meaningless tokens added in front.

The result? DeepSeek's answer started flipping back and forth.

When the padding length fell at certain positions, the model was more inclined to answer incorrectly with 32.

Move forward one or two tokens, it started leaning towards the correct answer, 8.

Continue moving, and the wrong answer came back—

The entire process repeated with a period of 4 tokens.

More specifically, among the 16 padding lengths tested in the paper, when the length modulo 4 had a remainder of 0 or 1, the model favored the wrong answer 32; with a remainder of 2 or 3, it favored the correct answer 8.

The researchers also tallied the probabilities the model assigned to the two candidate answers.

At one set of positions, the average probability for the wrong answer 32 reached 71.3%, while the correct answer 8 only got 26.4%.

Switch to another set of positions, and the situation directly reversed:

The average probability for the correct answer 8 rose to 91.5%, leaving the wrong answer 32 with only 7.2%.

In other words, a change in the length of a few irrelevant characters in front was enough to make the model produce completely different judgments for the same problem.

This is a bit hard to assess.

When debugging code, programmers usually check logic, variables, and dependencies first.

Now it seems they might also need to check if they typed two extra equal signs in front.

However, a single code completion case isn't enough to show how widespread the problem is.

So the Seed team expanded their testing.

They turned their attention to a classic task in large language model long-context capabilities: Needle-in-a-Haystack.

The researchers constructed a 128K-token-long context containing approximately 16,000 key-value pairs.

For example, K1 corresponds to V1, K2 to V2...

Then they asked the model to find the Value corresponding to a specified Key.

During testing, the key-value relationships remained unchanged, the questions were the same, and the total context length was consistent.

The researchers focused on adjusting the position of the target information relative to the compression window boundaries.

They found that the accuracy curve of the DeepSeek-V4 series actually showed very明显的明显的 (obvious) periodic起伏 (fluctuations).

Among them, DeepSeek-V4-Flash-Base showed a maximum accuracy gap of 40.2 percentage points between different positions.

DeepSeek-V4-Pro-Base also reached 34.8 percentage points.

After post-training, the situation improved somewhat.

The gap for DeepSeek-V4-Flash-0731 narrowed to 19.1 percentage points, and DeepSeek-V4-Pro-0813 narrowed to 14.8 percentage points.

The newer DeepSeek-V4.1-Flash-0910 further reduced the gap to 6.1 percentage points.

But periodic differences still exist.

How did this disparity arise?

Looking closer, DeepSeek-V4's fluctuation period is 4 tokens, while DeepSeek-V4.1's changed to 2 tokens.

The researchers found this corresponds exactly to the KV Cache compression stride used by each generation of the model.

Well then, even the fluctuation period of answer performance matches the underlying compression configuration.

The Problem Lies in KV Cache Compression?

Let's talk about KV Cache compression then.

When large models process long contexts, they need to store Key and Value information corresponding to a large number of historical tokens for subsequent attention calculations.

The longer the context, the more memory and computational overhead this cache occupies.

Especially for long tasks involving tens or hundreds of thousands of tokens, KV Cache can easily become a bottleneck for inference efficiency.

Therefore, DeepSeek-V4 adopted Chunked KV Cache Compression.

The idea is to divide consecutive tokens into windows and then compress the information within each window into fewer cache entries.

This way, the model doesn't need to preserve cache of the same scale for every historical token.

It saves memory and reduces the cost of long-context attention calculations.

But the Seed team discovered that the reason for DeepSeek's "Jekyll and Hyde" performance might be hidden in this chunking process.

Suppose every 4 tokens form one compression stride.

Then, if the same piece of information appears at the 1st, 2nd, 3rd, or 4th position within the window, the conditions during compression might differ.

The paper calls this position relative to the compression window boundary the Phase.

Researchers found systematic differences in the model's retrieval ability for information in different phases.

They named this phenomenon Phase Sensitivity.

To give an analogy, giving the same document to the model:

In the first layout, the key number happens to land in a position the model finds easy to retain;

In the second layout, just a few extra words are added in front, changing the key number's position relative to the compression window.

The document is still the same document, but whether the model can later find the number might show a significant difference.

Moreover, this problem can't be simply attributed to "the information happened to be split between two windows."

Researchers found that even when both the Key and Value fall within the same compression window, retrieval accuracy can still vary greatly depending on their specific positions.

This indicates the issue also involves how the model writes information into the compressed cache and how it reads from it later.

To confirm further, the Seed team simply trained a batch of models from scratch.

Using the Qwen3-0.6B architecture as a base, they constructed various KV Cache compression schemes and set up a full-attention model without chunked compression as a control.

The focus was only on changing the compression mechanism to see if periodic fluctuations would appear accordingly.

The result: All chunked compression models tested exhibited periodic changes corresponding to their compression stride.

Conversely, the full-attention baseline model did not show the same level of periodicity.

The researchers also separately adjusted window size and compression stride, finding that the period mainly changed with the compression stride.

Stride of 4, performance fluctuated with a period of about 4 tokens;

Stride of 6, the period also became about 6 tokens;

Stride of 8, same thing;

...

Even without using RoPE position encoding, or replacing learnable compression weights with simple averaging, this phenomenon persisted.

In other words, the problem isn't caused solely by a particular position encoding or a special module.

The design of chunked compression itself may introduce periodic retrieval weaknesses.

The Seed team further intervened on attention heads, observing how the model's retrieval ability across different phases changed after removing different components.

They found that different attention heads contributed differently to different phases.

Some heads were better at processing information at certain positions, while others played a larger role at other positions.

The researchers termed this Phase Specialization.

That is, the model's internals seem to have formed a division of labor: different attention components developed preferences for different positions within the compression window.

This division of labor can help the model perform information retrieval, but it may also make certain positions relatively weak links.

The paper also analyzed the training process using a simplified theoretical model, finding that gradient flow might drive compression modules to form stable positional preferences.

This also explains why this periodicity isn't simple random noise. It might be a natural outcome of the model learning how to compress information.

But such problems might not be directly visible from ordinary benchmarks because conventional evaluations usually aggregate a large number of test results into a single average score.

Suppose the model performs very well at some positions but significantly worse at others; the final calculated average score might still look good.

Therefore, the Seed team suggests that when evaluating models using chunked KV Cache compression, one shouldn't just look at overall retrieval accuracy but also test the same piece of information placed at different compression phases separately.

Even though this position-dependent performance bias exists, the memory and computational costs of long-context reasoning are still there, making compression valuable for practical reasons.

But after saving on cache, the model might no longer treat information from different positions equally.

Of course, judging from this experiment's results, post-training and architectural iterations can indeed significantly narrow this gap.

Paper address: https://arxiv.org/pdf/2609.36322

This article is from the WeChat public account "QbitAI", author: Wen Le

Trending Cryptos

Related Questions

QAccording to the ByteDance Seed team's research, what is the primary technical reason behind DeepSeek-V4's inconsistent performance?

AThe inconsistent performance, termed 'Phase Sensitivity', is primarily linked to DeepSeek-V4's implementation of 'Chunked KV Cache Compression', a technique for long-context processing. The model's ability to retrieve information varies based on the information's position relative to the compression window boundaries.

QWhat specific pattern did researchers observe in DeepSeek-V4's performance when irrelevant tokens were added before a task?

AResearchers observed a cyclical pattern where the model's performance, specifically its tendency to give a correct or incorrect answer, changed periodically. For DeepSeek-V4, this fluctuation repeated every 4 tokens, directly corresponding to its KV Cache compression stride length.

QIn the 'Needle in a Haystack' test, what was the maximum accuracy gap observed for DeepSeek-V4-Flash-Base when key information was placed in different positions?

AIn the 128K-token 'Needle in a Haystack' retrieval test, the maximum accuracy gap for DeepSeek-V4-Flash-Base reached 40.2 percentage points depending on the target information's position within the compression window.

QWhat internal mechanism did the researchers identify as 'Phase Specialization' within the model's architecture?

A'Phase Specialization' refers to the phenomenon where different attention heads within the model develop specific preferences or specializations for processing information from different positions (phases) within the compression chunk, leading to uneven retrieval capabilities across positions.

QHow did the performance of later DeepSeek model versions (V4.1) compare to V4 regarding this 'Phase Sensitivity' issue?

ALater versions showed significant improvement. For instance, DeepSeek-V4.1-Flash-0911 reduced the maximum accuracy gap to 6.1 percentage points, down from 40.2 in V4-Flash-Base. However, a cyclical pattern still existed, but with a period of 2 tokens, matching its updated compression stride.

Related Reads

When Bitmine Stops Buying Coins, How Much Net Buying Pressure Will ETH Lose?

Bitmine (NYSE: BMNR) is nearing its long-stated goal of holding 5% of Ethereum's total supply. As of November 10, the company disclosed holdings of 6,070,092 ETH worth roughly $13.6 billion, or 4.97% of supply—99% of its "Alchemy of 5%" target set in June 2025. Chairman Tom Lee has confirmed 5% is a hard cap, meaning Bitmine's year-long accumulation campaign will soon pause. The core question is how much this matters for ETH. Bitmine is the world's second-largest crypto treasury after Strategy, with over 80% of its ETH staked. However, its aggressive buying phase has already passed. Nearly all of its heavy accumulation occurred in the second half of 2025, when it bought over 4.1 million ETH at rates of 100,000–200,000 per week. Since mid-2026, as holdings approached the 5% cap and BMNR's stock fell into discount territory, weekly net purchases slowed to just 8,000–15,000 ETH—roughly $20–30 million per week. Against a market with daily trading volumes in the tens of billions and ETF flows exceeding $100 million in a single day, this marginal buying has had limited price impact for months. Other buyers can't realistically replace Bitmine: secondary treasury companies like SharpLink (~870k ETH) and The Ether Machine (~496k ETH) face market-cap and mNAV constraints, while spot ETFs are pro-cyclical, chasing rallies and redeeming in declines. Neither can replicate Bitmine's counter-cyclical, price-insensitive accumulation. The conclusion: when the final ~35,000 ETH gap is filled, the market will lose only a weak marginal bid of about 10,000 ETH per week. The market has already adapted to Bitmine's absence as a major buyer. What remains is 6.07 million ETH locked in treasuries and staking nodes—a supply-lock experiment that has largely completed its historical task.

marsbit9m ago

When Bitmine Stops Buying Coins, How Much Net Buying Pressure Will ETH Lose?

marsbit9m ago

Just Now, Gemini 4 Carbon Is Fully Rolled Out Globally: 6.5 Million Lines of Mathematical Proofs in 4 Weeks Verifying "Signal Security"

Google DeepMind has announced the full public rollout of Gemini 4 Carbon, described as Google's most capable model to date, designed for complex, long-horizon tasks. Unlike models that rush to a quick answer, Carbon is built to invest more reasoning resources—using Extended Verification Loops and additional thinking tokens—to iteratively execute, check, and correct its work across software engineering, cybersecurity, knowledge work, and scientific research. CEO Demis Hassabis demonstrated its power by having Carbon reverse-engineer his original Theme Park game binary, add a DeepMind-themed ride, and port the entire game to the web in one pass. The model is available to consumers via Google AI Ultra through the Gemini app and AI Mode in Search, to developers and enterprises through Google AI Studio, Android Studio, Antigravity, and Gemini for Business, and to researchers and cyber defenders through special access programs. Carbon ranks first on independent workflow benchmarks including CursorBench 4.0, Perplexity WANDR, and Zapier AutomationBench, at less than half the per-task cost of the best external model. Google also previewed a /swarm feature in Antigravity that lets multiple agents collaborate on architecture, coordination, implementation, and review. To showcase scientific capability, DeepMind introduced AlphaProof Code, which used Carbon-driven agents to formally verify the Signal encryption protocol's underlying code, generating over 6.5 million lines of machine-checked Lean proofs in just four weeks—work that would traditionally take experts years. The verification extended to compiled AArch64 machine code, not just source, and the system also successfully produced counterexample proofs when two previously fixed vulnerabilities were deliberately reintroduced. DeepMind hopes to extend the approach to medical device firmware and power grid software, integrating it with its CodeMender vulnerability-fixing tool to create a fuller security loop from bug discovery to formal verification. The release signals a broader shift: AI is moving from answering questions and writing code toward proving, mathematically, that code is trustworthy.

marsbit33m ago

Just Now, Gemini 4 Carbon Is Fully Rolled Out Globally: 6.5 Million Lines of Mathematical Proofs in 4 Weeks Verifying "Signal Security"

marsbit33m ago

Global banks commit $270M to stablecoin venture targeting 2027 launch

A consortium of 21 financial institutions, including Citigroup, Goldman Sachs, Banco Santander, Deutsche Bank and BBVA, has committed about $270 million to Project Faro, a stablecoin initiative that aims to act as a stablecoin issuer and launch a US dollar stablecoin in the first half of 2027. The funding is intended to cover five years of operations, and the company plans to issue stablecoins pegged to the currencies of all G7 countries, with a euro-backed coin expected to follow the dollar launch. The group first announced the initiative in September, and the company is expected to be incorporated in the US. Similar initiatives are also underway, including Stripe-led Open USD and Qivalis, a European banking consortium. Several Faro members, such as BBVA, Rabobank and BNP Paribas, are also involved in competing projects, highlighting how traditional financial institutions are positioning themselves across multiple initiatives rather than committing exclusively to a single stablecoin project. Qivalis has made regulatory progress, receiving authorization from the Dutch central bank as an electronic money institution in October 2026. Despite growing institutional interest, Faro is entering a market dominated by crypto-native issuers: Tether's USDT and Circle's USDC account for approximately 85% of the roughly $290 billion stablecoin market as of Nov. 10. Distribution could offer the bank-backed venture a competitive advantage, since its participating institutions already have established banking customers, payment networks and relationships that could help distribute a new stablecoin. The challenge will be translating that existing financial infrastructure into a competitive onchain payment network.

cointelegraph1h ago

Global banks commit $270M to stablecoin venture targeting 2027 launch

cointelegraph1h ago

Strategy buys 1,036 Bitcoin as BTC holdings climb to 850,364

Strategy bought 1,036 Bitcoin for $81.9 million last week, bringing its holdings to 850,364 BTC. The company paid an average of $79,012 per BTC, according to its Monday filing with the US Securities and Exchange Commission. Strategy said it paid a total of $64.16 billion for its Bitcoin holdings, at an average price of $75,456 per coin. Its holdings now account for roughly 4.05% of Bitcoin's maximum 21 million supply. Since the start of October, Strategy has spent about $1.06 billion buying back its STRC preferred shares, nearly five times the $219.7 million it spent on Bitcoin over the same period. Strategy raised $283.8 million in net proceeds last week by selling about $2.05 million shares of its common stock (MSTR) through its at-the-market offering program. Strategy used $188.3 million of those proceeds to repurchase about 1.89 million STRC shares, following $208.2 million in buybacks the previous week. It spent $81.9 million on Bitcoin and allocated another $13.6 million to its USD Cash balance. Strategy's separate US dollar reserve fell to $4.77 billion from $4.79 billion after the company drew $21.1 million to pay dividends on its preferred stock. Its USD Cash balance rose about 10% to $146.2 million from $132.7 million. The latest Bitcoin purchase follows Strategy's acquisition of 811 BTC for $65.5 million the previous week. Strategy's common stock fell 3% to $125 in Monday's premarket trading, while its STRC preferred shares edged down 0.06% to $99.7, according to Yahoo Finance.

cointelegraph3h ago

Strategy buys 1,036 Bitcoin as BTC holdings climb to 850,364

cointelegraph3h ago

Trading

Spot

Hot Articles

How to Buy T

Welcome to HTX.com! We've made purchasing Threshold Network Token (T) simple and convenient. Follow our step-by-step guide to embark on your crypto journey.Step 1: Create Your HTX AccountUse your email or phone number to sign up for a free account on HTX. Experience a hassle-free registration journey and unlock all features.Get My AccountStep 2: Go to Buy Crypto and Choose Your Payment MethodCredit/Debit Card: Use your Visa or Mastercard to buy Threshold Network Token (T) instantly.Balance: Use funds from your HTX account balance to trade seamlessly.Third Parties: We've added popular payment methods such as Google Pay and Apple Pay to enhance convenience.P2P: Trade directly with other users on HTX.Over-the-Counter (OTC): We offer tailor-made services and competitive exchange rates for traders.Step 3: Store Your Threshold Network Token (T)After purchasing your Threshold Network Token (T), store it in your HTX account. Alternatively, you can send it elsewhere via blockchain transfer or use it to trade other cryptocurrencies.Step 4: Trade Threshold Network Token (T)Easily trade Threshold Network Token (T) on HTX's spot market. Simply access your account, select your trading pair, execute your trades, and monitor in real-time. We offer a user-friendly experience for both beginners and seasoned traders.

15.0k Total ViewsPublished 2024.03.29Updated 2026.06.02

How to Buy T

Discussions

Welcome to the HTX Community. Here, you can stay informed about the latest platform developments and gain access to professional market insights. Users' opinions on the price of T (T) are presented below.

活动图片