H-JEPA: LeCun's World Model Startup Delivers, Code and Weights Fully Open-Sourced

marsbitPublished on 2026-10-10Last updated on 2026-10-10

Abstract

Yann LeCun's AI startup, AMI (Advanced Machine Intelligence), along with researchers from NYU, INRIA Paris, and Brown University, has introduced H-JEPA, a hierarchical world model for visual planning. The model stacks multiple Joint-Embedding Predictive Architectures (JEPAs) in layers, each predicting future states over different time horizons in its own representational space. For planning, a high-level layer first formulates a coarse plan, then passes predicted intermediate states as sub-goals to lower layers, which refine them step-by-step into executable actions. This hierarchical approach allows the model to manage both long-term goals and immediate actions. In the Visual AntMaze navigation task, a three-layer H-JEPA achieved a 73% success rate, significantly outperforming a single-layer model at 18%. It also demonstrated better performance with less computational cost in other simulated tasks. The work embodies LeCun's long-held view that intelligent systems require hierarchical planning across different levels of abstraction and timescales. To address the issue of representational collapse in world model training, H-JEPA employs the SIGReg method from prior work. The team has open-sourced the code and pre-trained model weights, enabling researchers to reproduce the planning experiments. This release follows AMI's commitment to open research, announced when the company launched with $1.03 billion in funding at a $3.5 billion pre-money valuation earlier this year. The ...

LeCun's world model startup, AMI (Advanced Machine Intelligence), has just quietly delivered its answer.

In the latest paper, H-JEPA, researchers from AMI, New York University, INRIA Paris, and Brown University introduced a hierarchical world model for visual planning.

H-JEPA stacks multiple JEPAs (Joint Embedding Predictive Architectures) layer by layer, allowing each layer to predict the state after different time spans within its own representation space.

During planning, the high-level layer first formulates a rough plan, then passes the predicted intermediate states as subgoals to the low-level layer. The low-level layer takes over and refines the plan step by step, ultimately generating executable actions.

Through this division of labor, the world model can both plan for long-term goals and handle immediate actions.

In the Visual AntMaze simulated maze task, the three-layer H-JEPA achieved a success rate of 73%, compared to 18% for the single-layer model.

Simultaneously, in several simulation tasks, the hierarchical model achieved better performance with less computational cost for planning.

Besides delivering tangible test results, H-JEPA also continues the open-source path that LeCun emphasized from the startup's inception.

The team has simultaneously open-sourced the code and pre-trained model weights, allowing researchers to directly load the model and replicate planning experiments.

Why Does a World Model Need Hierarchical Planning?

LeCun has actually been talking about this for a long time.

Back in 2022, while still serving as Chief AI Scientist at Meta, he proposed the concept of hierarchical JEPA:

Let the model predict at different levels of abstraction and time scales, and then implement complex goals as actions through hierarchical planning.

This might sound a bit abstract at first, but it's easy to understand using an example LeCun often cites.

Suppose you are sitting in your office at New York University, planning to go to Paris. When planning the entire trip, you focus on the destination, flights, and schedule.

After deciding on the itinerary, you start planning further: how to leave the office, go downstairs, and take a taxi to the airport.

When you actually walk out of the office, the information you focus on becomes the ground in front of you, the stairs, and where to place your next step, which foot to move first.

△

These planning levels cover different time spans and require different information. When planning the flight, the height of a specific step on a staircase is not important; when walking down the stairs, this detail directly determines how you should step.

This is not a characteristic unique to human planning. For robots, the high level needs to understand task objectives, environmental relationships, and future directions, while the low level needs to master specific actions and body control details.

For example, a quadruped robot may have already reached the target location, but if its leg posture differs from the target image, the model might still consider itself far from the goal due to these differences.

So, the question becomes:

How to let different levels learn information representations suitable for themselves, and then gradually implement high-level plans into low-level actions?

How Exactly Does H-JEPA Work?

Specifically, to solve the above problem, H-JEPA configures three components in each layer to handle state, action, and prediction respectively.

Among them, the state encoder is responsible for extracting the representation of the current environment. The lowest layer directly reads images, while higher layers further abstract based on lower-layer representations, retaining information more important at that time scale.

The action encoder is responsible for describing the changes brought about by actions. The low layer represents specific actions, while the high layer compresses a sequence of continuous actions into more coarse-grained changes.

The predictor combines the current state and action to predict future states. Different levels cover different time spans. In the paper's experiments, one step at a higher adjacent layer corresponds to two-step predictions at the lower layer.

Regarding model training, the layers above are learned jointly through end-to-end training, with different levels connected via subgoals.

The intermediate states predicted by the high level become subgoals for the low level. The low level transforms its predicted states into the high-level representation space, compares them with the subgoals specified by the high level, and then adjusts its actions accordingly.

However, when training the world model, there is a classic problem that needs to be addressed: representation collapse.

If the encoder compresses all inputs into the same vector, and the predictor only outputs this vector, it can achieve very low prediction error, but the model hasn't actually learned any useful information.

To address this, H-JEPA adopts the SIGReg proposed by LeCun's team in their previous LeJEPA work, constraining the representation distribution to prevent different states from being compressed into the same vector.

This year's LeWorldModel applied this method to world model training. H-JEPA builds upon it as the lowest layer, adding more JEPAs on top to form a structure trained jointly and planning from top to bottom.

In experiments, the researchers further observed information trade-offs at different levels. In the maze task, the high level gradually weakened the leg posture but still retained the robot's position information.

However, hierarchy also has its applicable conditions. The two-layer model performed well in the Push-T task, but adding a third and fourth layer led to decreased performance.

The paper suggests that shorter task trajectories might limit the training data available to higher layers.

LeCun, Still the Same LeCun

Overall, these designs of H-JEPA continue a fundamental proposition of JEPA: performing prediction in the representation space.

JEPA stands for "Joint Embedding Predictive Architecture." LLMs typically learn to generate content by predicting the next token; visual JEPA first encodes images or videos into internal representations, then predicts the representations of target parts, without needing to generate image pixels one by one.

In world models like H-JEPA, the model also incorporates actions to predict the representations of future states. The planner then compares the consequences of different actions to find a path towards the goal.

LeCun has always believed that understanding the world, predicting the consequences of actions, and planning are essential capabilities for achieving human-level intelligence. His assessment of LLMs has always revolved around this point.

In September this year, at the EC CV conference in Malmö, Sweden, his presentation slides once again featured that familiar piece of advice:

If you are interested in human-level AI, don't work on LLMs.

After this slide spread on social media, some doubted if it was fake. LeCun replied:

Guaranteed real.

He later added that LLM-based AI tools are very useful and everyone is using them. But he believes that LLMs alone cannot lead to human-level intelligence.

Now, this research philosophy has also become AMI's entrepreneurial direction.

In March this year, AMI announced the completion of a $1.03 billion funding round, with a pre-money valuation of $3.5 billion.

The company is chaired by LeCun, with Alexandre LeBrun as CEO, Saining Xie as Chief Scientist, Pascale Fung responsible for Research and Innovation, and Michael Rabbat responsible for World Models. Rabbat is also one of the authors of this H-JEPA paper.

However, it must be said that while research results are research results, another expectation for AMI from the outside world remains its products.

At least from the current official website, the company primarily showcases its technical direction, team, and recruitment information, with no public-facing product portal visible yet.

On the other hand, World Labs, which also bets on world models, has reached a new turning point.

On September 28th, AMD announced an all-stock acquisition agreement with World Labs, founded by Fei-Fei Li, worth approximately $8.2 billion.

It can be said that one side is moving towards integration with chip companies, while the other continues to advance along its own research path.

But with Fei-Fei Li's world model startup achieving a milestone, is Yann LeCun's major result also just a matter of time...?

References

[1]https://venturebeat.com/technology/h-jepa-teaches-world-models-to-plan-at-multiple-levels-of-abstraction

[2]https://arxiv.org/pdf/2610.06805

This article is from the WeChat public account "QbitAI", author: henry

Related Questions

QWhat is H-JEPA, and how does it differ from single-layer world models?

AH-JEPA is a Hierarchical Joint Embedding Predictive Architecture proposed by AMI (Advanced Machine Intelligence). It stacks multiple JEPA layers to enable planning at different levels of abstraction and temporal scales. In the Visual AntMaze task, its success rate was 73%, compared to 18% for a single-layer model, showcasing its superior ability to handle long-term goals and immediate actions through layered, hierarchical planning.

QWhy does LeCun advocate for a hierarchical approach in world models like H-JEPA?

ALeCun advocates for a hierarchical approach because he believes complex tasks, like a human planning a trip, require reasoning at different abstraction levels and time horizons. High levels handle overarching goals (e.g., booking a flight), while lower levels manage immediate actions (e.g., walking down stairs). This division allows models to both plan for the future and execute detailed, context-specific actions, which he considers essential for achieving human-level AI.

QHow does H-JEPA's training prevent the issue of representation collapse?

AH-JEPA prevents representation collapse by using a method called SIGReg (Sparse Information Gap Regularization), which was introduced in LeCun's prior work, LeJEPA. SIGReg constrains the distribution of representations, preventing the encoder from mapping different input states to the same vector. This ensures the model learns meaningful and distinct representations necessary for accurate prediction and planning.

QWhat significant business developments are mentioned regarding AMI and World Labs?

ATwo significant developments are mentioned: AMI, LeCun's startup, completed a $10.3 billion funding round with a pre-money valuation of $35 billion. Separately, AMD announced an all-stock acquisition agreement with World Labs, a company founded by Fei-Fei Li, valued at approximately $8.2 billion. This highlights the growing investment and strategic moves in the world model AI sector.

QWhat is LeCun's stance on large language models (LLMs) in relation to achieving human-level AI?

ALeCun believes LLMs alone are insufficient for achieving human-level AI. He has publicly stated, 'If you are interested in human-level AI, don't work on LLMs.' He acknowledges that LLM-based tools are useful, but argues that true intelligence requires capabilities like understanding the world, predicting action consequences, and hierarchical planning—areas where models like H-JEPA are focused.

Related Reads

Anthropic IPO: Same Offering Price, But Are Retail Investors Getting a More Expensive Share?

Anthropic, the maker of Claude, is preparing for its IPO at $70–78 per share, implying a base valuation of about $1 trillion (up to $1.4 trillion fully diluted), with pricing expected November 18 and a Nasdaq listing under ANT the next day. It would be the first frontier AI lab to go public, raising up to $90.4 billion across public offering and private placement. A key overlooked detail: seven strategic investors—AMD, Broadcom, Google, Micron, Microsoft, Nvidia, and TSMC—are buying $26 billion of shares via private placement at the same IPO price as retail investors, but with extra advantages. They pay no underwriting fees, hold zero-vote Class N shares, face resale restrictions, and are price-insensitive, having committed to buy at the final price regardless. That placement accounts for nearly 29% of the total raise. The core issue is "same price, different value." Google, for instance, also stands to receive roughly $250 billion in compute payments from Anthropic over seven years, so its dollars buy both stock and future revenue. Strategic buyers also reduce tradable supply, supporting a higher opening price, while only about 20% of shares are allocated to retail investors—those who miss out must buy in the secondary market after the first-day pop. The article also highlights AI's "circular financing" structures (Microsoft/Nvidia/Anthropic, Amazon/Anthropic, AMD/OpenAI, Nvidia/OpenAI), where investors are simultaneously shareholders and suppliers. Risks flow along this loop: ordinary shareholders sit behind all contractual obligations, so if Anthropic's compute commitments exceed its ability to pay, retail investors bear the tail risk. Retail investors pay the same price as Google but receive fewer rights, may pay more after listing, and stand last in line in the circular financing chain.

Odaily星球日报22m ago

Anthropic IPO: Same Offering Price, But Are Retail Investors Getting a More Expensive Share?

Odaily星球日报22m ago

Regarding the New CEO and the Transition Period, What Information Did Nvidia's Earnings Call Reveal?

Nvidia's board has explicitly stated it has no intention of reinstating Jensen Huang as CEO, despite Huang saying he would accept a return if the board deemed it in the best interest of shareholders, employees, the company, and the U.S. The statement came during the company's Q3 earnings call, attended by board chairman Harvey Jones and interim CEO/CFO Colette Kress. Huang resigned on November 12 over personal conduct violating company standards, and the board unanimously accepted his resignation, appointing Kress as interim CEO. The board has formally launched a search for a permanent CEO, hiring top executive search firm Heidrick & Struggles, and will consider both internal and external candidates. Jones stressed the board takes the process seriously and wants a leader capable of steering Nvidia through a market reshaped by advanced chip technology and national security concerns, but will not rush the appointment. Kress emphasized stability during the transition, saying the company's long-term goals and commitments remain unchanged. She acknowledged diverse internal voices, including employee petitions supporting Huang's return, but said staff remain dedicated to their "life's work" at Nvidia. She praised management's performance and said "business as usual" continues, with customers, suppliers, and partners offering 100% support; TSMC management visited Nvidia's offices this week, and memory and advanced packaging supply has been secured. Nvidia also announced a new National Security and Export Control Oversight Committee, comprising independent directors, external advisors, and legal experts, to strengthen board oversight of national security policy, export control compliance, and global trade risks. Overall, the transition plan is clear: Kress handles daily operations, the board searches for a permanent CEO via an external firm, and governance around national security and export controls is being reinforced.

Odaily星球日报23m ago

Regarding the New CEO and the Transition Period, What Information Did Nvidia's Earnings Call Reveal?

Odaily星球日报23m ago

MARA's Turnaround: Mortgaging 18,750 BTC to Buy a Power Plant, Selling 40,000 Coins in Nine Months

MARA Holdings has fundamentally shifted from hoarding Bitcoin to actively managing it as a financial asset. As of its Q3 report on November 9, MARA had sold about 40,581 BTC year-to-date, raising roughly $3 billion. After committing in 2024 to retain all mined Bitcoin, the company began allowing sales of newly mined coins in 2025 and expanded to selling legacy reserves in 2026. A key transaction involved the acquisition of Long Ridge Energy & Power. To fund the deal, MARA pledged 18,750 BTC to secure $750 million in loans from Coinbase Credit and Two Prime Lending—$600 million in new capital and $150 million for refinancing. After Q3, MARA sold 13,597 BTC at an average of about $81,039, raising roughly $1.1 billion, fully repaid both loans, and released 17,477 BTC from collateral. Management estimated the borrow-then-sell strategy yielded about $170 million in net gains. Long Ridge's core asset is an operating natural gas plant in Ohio with 485 MW of permitted capacity, expected to reach 505 MW by H1 2027, with about 70% of output covered by long-term contracts, plus over 1,600 acres for data center development. MARA also secured board approval for a stock repurchase program of up to $500 million and listed AI infrastructure investment, convertible note put obligations, and potential buybacks as possible uses for future Bitcoin liquidation. The mining industry has diverged strategically: American Bitcoin sells some output to cover costs while still accumulating; Riot Platforms sold 20,264 BTC to fund data center development; IREN's AI cloud revenue ($200 million) now exceeds mining revenue ($28 million). For MARA, future Bitcoin reserve levels will depend on when new projects begin generating cash and how much capital must be deployed beforehand. In Q3, MARA mined 2,247 BTC (up 5% YoY) but total revenue fell 35% to about $164 million, while net income was about $632 million, including $885 million in digital asset fair value gains.

marsbit52m ago

MARA's Turnaround: Mortgaging 18,750 BTC to Buy a Power Plant, Selling 40,000 Coins in Nine Months

marsbit52m ago

US Stock Market Trends (November 18): Nvidia Q3 Earnings Beat Expectations, Stock Rises Nearly 4% After Hours; Dow Plunges 848 Points

Nvidia reported Q3 revenue of $115.84 billion and guided Q4 to about $130 billion, both above expectations, sending shares up nearly 4% after hours. The report followed a fourth straight down day for US stocks. The Dow fell 848 points (-1.64%) to 50,759.42, the S&P 500 dropped 1.03% to 7,340.53 (its lowest close since late July), and the Nasdaq lost 0.74% to 24,328.38. All 11 S&P sectors declined, with financials and utilities among the weakest. A CNN report of a classified NSA network glitch, later described by the agency as a temporary performance dip during routine maintenance, weighed on sentiment alongside recent FAA, Microsoft Outlook and bank app outages. Software stocks were hit hardest: Salesforce fell 9.45%, Intuit 7.17% and ServiceNow 6.51%, with analysts citing fund repositioning ahead of Anthropic's IPO and AI-disruption worries. Nike, UnitedHealth and IBM also fell. Chip stocks were mixed, with Broadcom up 2.46%. Nvidia disclosed selling 25% of its Intel stake for nearly $5.5 billion. Nvidia's board said it unanimously accepted Jensen Huang's resignation and has no plans to reappoint him, hiring a search firm for a permanent CEO; interim CEO Colette Kress said it was "business as usual." October retail sales rose 0.2%, in line with expectations, while import prices gained 1.1%. St. Louis Fed's Musalem suggested multiple rate hikes may be needed, and markets priced about a 95% chance of a 25bp December hike. The 10-year Treasury yield fell 2bp to 5.102%. WTI rose 0.34% to $86.01 and Brent 0.76% to $93.14. Bitcoin traded near $64,900, down over 20% in a month.

marsbit1h ago

US Stock Market Trends (November 18): Nvidia Q3 Earnings Beat Expectations, Stock Rises Nearly 4% After Hours; Dow Plunges 848 Points

marsbit1h ago

Trading

Spot
活动图片