Artículos Relacionados con Google Research

GPT-5 Also Has Tip-of-the-Tongue Moments, Google Tested 4.5 Million Times: The Keys Are Lost

Google researchers have discovered that advanced AI models like GPT-5 and Gemini 3 experience a phenomenon akin to the human "tip-of-the-tongue" state, where they possess knowledge but fail to retrieve it. Their study, "Empty Shelves or Lost Keys?" (ICML 2026), introduces the "Knowledge Portrait" framework to analyze factual knowledge in models, distinguishing between failure to encode a fact versus failure to recall it. Testing on 13 models across 2.15 million queries from the WikiProfile benchmark revealed that state-of-the-art models successfully encode 95-98% of facts into their parameters. However, when asked directly, they fail to recall 26-34% of these known facts. Enabling chain-of-thought ("thinking") reasoning reduces this recall failure to 11-12%, recovering 40-65% of the previously unrecalled but encoded facts. The research identifies two key bottlenecks: recalling obscure ("long-tail") facts and answering reversed queries (e.g., "Who is Tom Cruise's mother?" vs. "Whose son is Tom Cruise?"). While scaling model size effectively reduces encoding failures, it does little to improve recall rates. In larger models, recall failure becomes the dominant source of factual errors, accounting for over 70% of mistakes in GPT-5.2. The findings suggest that for top models, the primary challenge is no longer storing knowledge but accessing it efficiently. Future accuracy gains may depend more on improved inference-time methods and "meta-cognitive" abilities, enabling models to recognize when they need to engage in deeper reasoning to retrieve information they already know.

marsbitHace 9 hora(s)