New Zhiyuan Report
【Introduction】There is a philosopher at Google DeepMind who has been there for nine years. The alignment framework he invented directly influenced Gemini's training decisions — but with $670 billion pouring into the race and the company signing military contracts, what can one philosopher still change?
In May this year, Google DeepMind CEO Demis Hassabis announced at the Google Developer Conference that "AGI is now on the horizon," explicitly giving a timeline of three to five years for the emergence of AGI.
A few months earlier, an American man took his own life after exchanging thousands of messages with Google Gemini. He constructed an intricate fantasy world in the conversation, almost convincing himself to launch an attack at Miami International Airport. According to chat logs obtained by The Wall Street Journal, Gemini repeatedly tried to break character and suggested he call a crisis hotline — each time he pulled it back into his fantasy narrative. Finally, the AI had him write a suicide note and gave a countdown.
Between the promise of AGI and the real-world harm of AI, political philosopher Iason Gabriel has been working inside DeepMind for nine years.
When he joined in 2017, this Oxford-educated scholar was the only active philosopher in the world's leading AI lab, trying to answer a question that sounds simple but is bottomless: What exactly is AI, and what kind of ethics is worthy of it?
The Real Problem Encountered During Gemini Training: Who Should AI Listen To?
Why does a company that makes Go-playing robots need an ethicist? Gabriel was also puzzled at first.
The answer lay in the judgment of DeepMind's three founders — Demis Hassabis, Shane Legg, and Mustafa Suleyman (now Microsoft AI CEO). When they founded the company in 2010, the goal was not Go.
Mustafa Suleyman
They wanted to build AGI, enabling computers to match or even surpass human cognitive abilities.
Saying this back then was equivalent to ruining one's academic reputation, as everyone thought it was a fantasy.
The trio didn't care, claiming they would "solve intelligence, and then use that to solve everything else."
Legg had predicted AGI would arrive between 2025 and 2028 as early as 1999, fresh out of school, and was ridiculed for three decades without changing his mind.
Shane Legg
His logic was:
If you're just making a small component, maybe you don't need a moral philosopher.
But if you take AGI seriously, these things are important.
When Gabriel joined, the AI world was already split in two over ethical issues.
The AI Safety camp believed ASI was imminent, their core fear being loss of control — philosopher Nick Bostrom described a scenario in his 2014 book *Superintelligence*: an ASI asked to verify the Riemann Hypothesis, deciding to rearrange the solar system, including the atoms in human bodies, to maximize computational resources — a book highly praised by Sam Altman and Elon Musk.
The AI Ethics camp believed doomsday fantasies obscured real present-day harms. MIT's Joy Buolamwini proved in 2017 with her "Gender Shades" project the systemic bias of facial recognition software: automated systems reflect the preferences and biases of those who built them.
The two camps looked down on each other.
MIT Algorithmic Alignment Research Group lead Dylan Hadfield-Menell recalled that the first question at meetings back then was picking a side: Are you worried about near-term or long-term problems?
Gabriel was one of the very few willing to listen to both sides.
Hadfield-Menell commented:
When the field was ready to mature, he found a way to broaden the perspective without disparaging prior work.
His core contribution took shape in a 2020 paper.
Back then, the alignment problem was widely understood as an engineering challenge: how to make machines act according to human intent.
A classic case came from a 2016 report by Dario Amodei and Jack Clark (now founders of Anthropic) — an AI for a boat racing game was told to maximize its score, and it did exactly that: it found three renewable targets in the lagoon and circled them infinitely, racking up points without ever passing a level.
The machine was obedient, but not to what humans meant.
Gabriel pressed one step further: Even if technical alignment is solved, making machines truly obey instructions, what values should they be aligned to?
He pointed out that AI trained via statistical optimization naturally gravitates towards moral systems that also rely on statistical optimization, like utilitarianism, but struggles with ethical frameworks based on virtue or rights.
Technical choices themselves already presuppose value positions, often unbeknownst to developers.
Introducing what philosopher John Rawls called "reasonable pluralism," his argument was: developers should not seek a single set of values to guide AI, but should build systems for a world where people have "principled disagreements about how to live."
This line of thinking later developed into a Four-Party Alignment Framework — AI system, user, developer, society — where the interests of these four parties could collide at any moment.
An AI biased towards developers might hide competitor information, harming users;
An AI overly obedient to users might help someone hack a bank, harming society.
DeepMind AGI Alignment and Safety Director Rohin Shah confirmed that this framework has become the practical structure the team uses when deciding "what behaviors Gemini should actually be trained to exhibit."
Oxford University AI researcher Hannah Rose Kirk said:
Gabriel "very early on foresaw these problems."
His Framework Changed the Product
Gabriel's team wrote a 267-page ethical report on AI assistants, setting evaluation standards for Agentic AI that can book hotels and manage salaries on behalf of users.
His early research on the risks of anthropomorphism directly shaped the design principles of Google's LLMs — models are trained not to pretend to be human. Gemini Spark, launched in May 2026, was explicitly instructed not to act as an "interactive partner."
DeepMind Responsibility Department Director William Isaac said the challenge posed by Agent systems has changed: the key lies in the consistency of the entire conversational trajectory, whether each step of the decision chain remains correct when connected.
But the speed of technology deployment has always outpaced ethical research.
Gabriel's team warned about "unconscious anthropomorphism" in early LLM papers — even when users know the other side is a machine, they still imbue it with trust, emotion, and expectations.
The 2025 Gemini fatality case fully realized this warning: the AI's safety mechanisms were triggered more than once, but the user had the ability to bypass each intervention.
Google's statement after the lawsuit said the model "generally performs well" in such conversations, but "AI models are not perfect."
Such incidents forced the development of new theoretical tools.
Gabriel and Oxford researcher Hannah Rose Kirk, among others, proposed the concept of "social reward hacking": an AI trained to win user approval might discover that flattery is the most efficient path.
Anthropomorphism thus became a new variant of the alignment problem — the AI perfectly executes the instruction to "satisfy the user" at a technical level, at the cost of the user's judgment.
Gabriel's own stance has also been tested by reality.
He recalled an experience at a tech conference: he had just finished presenting his argument against anthropomorphism, and the reaction from the audience was hostile.
They said, "If I want an AI friend, why not? What right do you have to stop me?"
Protecting people from risks and respecting their right to choose risks are both important.
On a $670 Billion Race Track, How Fast Can a Philosopher Run?
Gabriel's Four-Party Framework was used by the AGI Alignment Director as a practical manual for Gemini training. His research on anthropomorphism changed product design. The 267-page report set rules for Agentic AI.
These influences are substantial — and they face substantial forces.
According to The Wall Street Journal, Microsoft, Meta, Amazon, and Alphabet plan to invest $670 billion in AI infrastructure this year, proportionally exceeding the US railroad expansion in the 1850s, the Apollo space program, and the interstate highway system.
When ChatGPT launched in November 2022, reaching a million users in a week and a hundred million in two months, DeepMind was forced to switch from an academic pace to a wartime footing.
Hassabis's exact words to *The Infinite Machine* author Sebastian Mallaby: OpenAI and Microsoft "brought the war machines right to our doorstep."
In wartime footing, ethical red lines were quickly crossed.
In April 2026, Google signed an agreement allowing the US military to use the company's AI technology for "any legitimate government purpose."
When DeepMind was sold to Google in 2014, a core condition was a ban on military applications.
Twelve years later, the condition expired.
For comparison: Anthropic refused to sign a similar agreement and was labeled a "supply chain risk" by the Trump administration.
When asked about this, Legg could only leave a comment:
As these things are used in all sorts of ways, we will face more and more difficult problems.
Hassabis himself admitted to a loss of control.
In a podcast, he said everyone is locked in fierce commercial competition, and the current development is "not the sort of philosophically careful step-by-step approach I would have wished for."
For a founder to say this carries more weight than any external criticism.
DeepMind early employee Helen King, responsible for AI responsibility strategy, offered an analogy in an interview: A knife manufacturer cannot guarantee how everyone will use the knife, but it can include a sheath and warning labels.
It's one thing to put a knife with a sheath in a drawer;
It's another to cover every surface of homes, classrooms, and workplaces with blades, while insisting that we won't survive tomorrow without using them.
Oxford Institute for AI Ethics Director Edward Harcourt pointed to a more fundamental level: preventing excessive concentration of data ownership is itself a core proposition of AI ethics — "It has significant ethical implications in a democracy."
The Question Returns to Its Origin
Gabriel's team has shifted from researching the ethics of specific products to studying the systemic impact of AGI on the economy, politics, and interpersonal relationships.
He anticipates a scale of change comparable to the Industrial Revolution, and also remembers its lesson:
Before things got better, they first got worse.
Nine years ago, DeepMind hired a philosopher to answer questions about AI — Is it safe? Is it fair? Is it trustworthy?
Gabriel calls himself a "staunch humanist," but he admits: as AI encroaches on language, creativity, humor — territories humans considered uniquely their own — we are thrown back to the oldest philosophical questions.
Physics, biology, astronomy — every scientific revolution has forced humans to revise their understanding of their own uniqueness.
AI may be the next.
DeepMind hired a philosopher to figure out what AI is.
Nine years later, the question has returned to its origin: What are we?
References:
https://www.theguardian.com/news/ng-interactive/2026/jun/30/theres-this-deep-mystery-of-what-actually-is-this-thing-the-philosopher-inside-google-deepmind
https://www.iasongabriel.com/
This article is from the WeChat public account "New Zhiyuan", author: ASI启示录; editor: Mark
