The name is right on the tip of your tongue.
You see a familiar face but can't for the life of you remember their name. Psychology has a name for this phenomenon: 'tip-of-the-tongue' (TOT).
Now, Google has found in a study that GPT-5 and Gemini 3 suffer from the same affliction.
These models have stored 95%–98% of tested facts in their parameters. Yet, when asked directly, they fail to answer 26%–34% of these facts.
Enabling 'thinking' (chain-of-thought) for a while longer, 11%–12% still remain unreachable.
When a large model can't answer, it's often not because the knowledge isn't in its 'brain'; it's learned it, but can't retrieve it.
This research is titled 'Empty Shelves or Lost Keys?' and was accepted at ICML 2026.
On August 12, Google Research summarized the findings in a blog post, also releasing a benchmark called WikiProfile and the complete dataset.
Shelves Are Not Empty, Keys Are Lost
Models can answer incorrectly due to two main flaws.
One is never having learned the information in the first place, which would require adding more data and parameters during pre-training. The other is having learned it but failing to retrieve it when phrased differently, which is a matter of post-training and inference.
Both problems have historically been lumped together as an accuracy issue.
Google's proposed 'Knowledge Profile' framework moves beyond asking 'did it answer this question correctly?' to judge 'what state is this fact in within the model's memory?' It categorizes facts into five states.
The 'Knowledge Profile' categorizes a fact into five states: encoding failure, recall failure, direct recall, recall via thinking, and correct answer via reasoning without encoding.
The same wrong answer, in different states, requires completely different fixes.
How to determine if a fact was encoded?
The method is to truncate the Wikipedia text right before the answer appears and have the model continue writing; or to ask a direct question based on the same context.
Both tasks are done without enabling thinking, to separate 'memorized' from 'inferred'.
Left side tests encoding: truncating Wikipedia for completion. Right side tests recall: rephrasing questions, changing positive/negative directions.
The benchmark, WikiProfile, consists of 2,150 facts, all sourced from English Wikipedia.
Each fact is paired with 10 questions: 2 to test encoding, 4 for free-form answering, and 4 multiple-choice. Totaling 21,500 questions.
Question generation was delegated to models: Gemini-2.5-Pro generated them, each was verified via Google Search. Facts with non-unique answers or ambiguous questions were discarded along with the entire fact. A final manual check removed less than 2%.
13 models were benchmarked, covering the Gemini 3, GPT-5, GPT-4.1, and Gemma 3 families. Each model was run with thinking on and off, each question sampled eight times, amassing about 4.5 million responses.
Benchmarking a state-of-the-art model completely costs about $500.
The result is the numbers from the beginning.
Among fact errors in state-of-the-art models, 'failure to retrieve' now outweighs 'never learned,' becoming the primary issue.
What Can't Be Retrieved Gets Stuck in Two Key Places
First is obscure knowledge.
The past mainstream explanation was insufficient model capacity, where obscure facts simply weren't learned.
According to this data, obscure knowledge was actually encoded. Gemini-3-Pro encoded 94.5% of obscure facts, compared to 99.5% for popular facts, a gap of only 5 percentage points.
But when answering freely, obscure fact recall dropped to 63.3%, while popular facts remained at 84.7%, a gap of 21 points.
Almost no difference during encoding, but a fourfold difference during recall.
Comparison between obscure facts (bottom 20%) and popular facts (top 20%). Encoding rate gap is small; direct recall rate gap widens significantly.
GPT-5 shows an even larger gap.
It encoded 90.7% of obscure facts and 98.4% of popular facts. But when answering, obscure facts fell to 52.9%, while popular facts were at 77.8%.
Nearly half of the obscure facts, though present in the model, remain inaccessible.
Little difference during storage, big difference during retrieval. The truth about long-tail problems is not that the model hasn't seen those obscure facts, but that those memories are harder to activate.
The second scenario is reverse questioning. Ask the AI, 'Who is Tom Cruise's mother?' and it answers quickly and accurately. Ask the reverse, 'Who is this woman's son?' and it stumbles.
A 2023 paper named this the 'reversal curse,' where the model learns 'A is B' but doesn't automatically generalize to 'B is A.' The explanation then was: the bidirectional association simply wasn't learned.
Google reformatted the same set of questions into multiple-choice, and the result immediately reversed.
GPT-5's recall rate for forward questions was 82.9%, dropping to 74% for reverse. But once switched to multiple-choice recognition, reverse questions weren't harder than forward; in fact, 9 out of 13 models performed better on reverse, the other 4 were equal.
It can handle multiple-choice, but fails when answering freely.
Same set of forward/reverse questions. In multiple-choice recognition, reverse isn't harder; in free-form generation, reverse is significantly harder.
This indicates the association is clearly present; what's stuck is how to extract it.
The authors explain that when a fact is encoded, the surrounding context, phrasing, and sequence also get embedded.
If the question phrasing deviates from the training pattern, the key no longer fits the lock.
Thinking's Other Role: Aiding Recall
Questions in WikiProfile are straightforward 'ask and answer' with no intermediate reasoning steps.
'Which club hosted Oasis's first performance?' This type of question is either known or not.
Logically, letting the model think longer shouldn't help much.
But thinking actually rescues them. It retrieves 40%–65% of facts that were encoded but initially unreachable.
The key clue lies in its preference: for facts never encoded, thinking only rescues 5%–15%.
Proportion of facts recovered after enabling thinking. For facts already encoded: 40%–65% recovered. For unencoded facts: only 5%–15%.
It's not reasoning with other knowledge; it's searching for the key in place.
The authors also ruled out a simpler explanation.
One might think thinking just lets the model generate more verbose answers, increasing the chance of guessing correctly once in eight tries. If that were true, answers should be more scattered.
The opposite happened: with thinking enabled, the number of correct answers per question across eight samples became more concentrated.
The biggest gains were precisely in the two most congested areas mentioned earlier.
For Gemini-3-Pro, the obscure/popular fact gap narrowed from 21.4 points to 12.5 points. For GPT-5, the forward/reverse gap narrowed from 9 points to 2 points. It exerts effort where it's most needed.
A companion paper from Google, 'Thinking to Recall,' offers a mechanistic explanation:
One is computational buffering. The output thinking tokens, even if semantically meaningless, buy the model extra implicit compute.
The other is fact priming. The model stating a batch of related facts first builds a semantic bridge to the correct answer.
In human cognition, this is called 'spreading activation.'
When you can't recall a name, you first recall their company, hairstyle, last meeting context—the name often pops up on its own.
The way to escape 'tip-of-the-tongue' is surprisingly the same for humans and models.
But this approach has a cost.
The same paper warns: if the intermediate facts in the chain of thought are fabricated, the hallucination probability of the final answer actually increases.
A bridge built wrong leads further astray.
Scaling Equals Stocking Shelves, Not Providing Keys
Using the open-source Gemma 3 family as a ruler.
As parameters increased from 1B to 27B, encoding failure rate dropped from 85% to 23%.
Scaling does stock the shelves; they become visibly fuller. But recall failure didn't drop proportionally. Its share of the remaining errors increased steadily, becoming the primary component of errors at 27B.
Looking at state-of-the-art models is more extreme: recall failure accounts for over 70% of all errors in GPT-5.2, and this proportion is higher in stronger models.
Distribution of the five states across 13 models. Encoding failure steadily shrinks with larger parameters, while recall failure remains almost unchanged.
Scaling improves how much is stored, not how much can be retrieved.
This isn't a death sentence for scaling, but it draws a boundary.
Since the proportion encoded is nearing the ceiling of 95%–98%, further adding parameters and data will yield diminishing returns in accuracy gains.
The next gains are more likely to come from methods in post-training and inference—how to better utilize already encoded knowledge.
Retrieval-Augmented Generation (RAG) can certainly fill gaps.
But the authors also caution: parametric knowledge still matters for fluency, response speed, and cross-context integration, and cannot be fully replaced by external retrieval.
It Must First Know It Can't Recall
So, when should the model think longer?
The paper frames this as a metacognition problem: the model's judgment of its own state. It must first realize 'I can't answer this directly' to know this question deserves extra thinking.
Over the past two years, the default answer for making models more reliable has been feeding more data and scaling parameters.
This research makes it clear: at the fact level, state-of-the-art models' shelves are largely stocked. The next challenge is whether they can retrieve the goods themselves.
The trickier next step: they must first know they can't recall.
References:
https://research.google/blog/empty-shelves-or-lost-keys-recall-is-the-bottleneck-for-parametric-factuality/
https://aihot.virxact.com/items/cmsqe6mat01bsroli3hw71nz7
This article is from the WeChat public account '新智元' (Xin Zhi Yuan), author: ASI启示录
