AI2 Releases Fully Open-Source Web Agent MolmoWeb: Controlling Web Pages Using Only "Vision"

marsbitPublicado em 2026-03-26Última atualização em 2026-03-26

Resumo

AI2 has released MolmoWeb, a groundbreaking, fully open-source web agent that operates solely by analyzing screenshots, marking a significant leap in vision-driven web navigation. Unlike traditional agents that rely on DOM, MolmoWeb captures and interprets visual data to make decisions—such as clicking, scrolling, or typing—making its process transparent and robust. Despite its compact size (4B and 8B parameters), MolmoWeb performs impressively: it scores 78.2% on the WebVoyager benchmark, nearing OpenAI’s proprietary o3 model (79.3%), and achieves up to 94.7% success with multiple attempts. It even surpasses Anthropic’s Claude3.7 in UI element localization. AI2 also released MolmoWebMix, a massive open dataset with 36K human-browsing tasks, over 2.2M screenshot-QA pairs, and GPT-4o-verified synthetic data. The model and data are fully available on Hugging Face and GitHub under Apache 2.0, promoting transparency and collaboration in AI development. Challenges remain in complex instructions, logins, and legal compliance.

The Allen Institute for Artificial Intelligence (AI2) recently released the groundbreaking fully open-source web agent MolmoWeb . Unlike traditional agents that rely on a webpage's underlying code (DOM), MolmoWeb makes decisions solely by reading screenshots, marking a significant leap forward in "vision-driven" web navigation technology.

Core Technology: "Seeing" Web Pages Like a Human

MolmoWeb's operating logic is very intuitive: it captures a screenshot of the current browser window, decides the next action (such as clicking, scrolling, or paging) through visual analysis, then executes it and repeats. This "what you see is what you get" model makes it more robust than traditional agents because the visual layout of a webpage is generally more stable than its underlying code, and its decision-making process is completely transparent and explainable to human users.

Performance Leap: Small Model Outperforms Giants

Despite having parameter sizes of only 4B and 8B, MolmoWeb demonstrates a "small but mighty" performance:

  • Topping the Charts: In the WebVoyager test, the 8B version scored an impressive 78.2%, not only ranking among the top open-source models but also approaching the performance of OpenAI's proprietary model o3 (79.3%).

  • Huge Potential: Research found that by running tasks multiple times and selecting the optimal result, its success rate could further jump to 94.7%.

  • Precise Localization: In UI element localization benchmark tests, it even surpassed Anthropic's Claude3.7.

Data Support: The Largest Open Dataset to Date

AI2 has not only open-sourced the model weights but also contributed a massive dataset named MolmoWebMix. This dataset contains:

  • 36,000 real browsing tasks completed by human volunteers.

  • Over 2.2 million screenshot-question-answer pairs.

  • Automated synthetic data verified by GPT-4o. Experiments show that synthetic data is even better than human trajectories at guiding the agent to find the "optimal path".

Open-Source Spirit and Future Challenges

Currently, MolmoWeb is fully available under the Apache 2.0 license on Hugging Face and GitHub. Although it still faces challenges in handling complex instructions, login authentication, and legal compliance (such as terms of service), AI2 firmly believes that only through complete transparency and community collaboration can we truly counter the data monopoly of large tech companies.

Perguntas relacionadas

QWhat is the name of the fully open-source web agent released by the Allen Institute for AI (AI2) that navigates using only screenshots?

AThe web agent is called MolmoWeb.

QHow does MolmoWeb's approach to web navigation differ from traditional web agents?

AUnlike traditional agents that rely on a webpage's underlying code (DOM), MolmoWeb makes decisions by reading and analyzing screenshots, making it a 'vision-driven' technology.

QWhat was the performance score of the 8B parameter version of MolmoWeb on the WebVoyager test, and how does it compare to OpenAI's model?

AThe 8B version scored 78.2% on the WebVoyager test, which is very close to the performance of OpenAI's proprietary model o3, which scored 79.3%.

QWhat is the name of the large, open dataset released alongside MolmoWeb, and what does it contain?

AThe dataset is called MolmoWebMix. It contains 36,000 real browsing tasks completed by human volunteers, over 2.2 million screenshot-QA pairs, and automated synthetic data verified by GPT-4o.

QOn which platforms has MolmoWeb been made available, and under what license?

AMolmoWeb has been fully released on Hugging Face and GitHub under the Apache 2.0 license.

Leituras Relacionadas

In Conversation with Ray Dalio: We Are Currently in an AI Bubble, with 1% of My Portfolio in Bitcoin

Ray Dalio, founder of Bridgewater Associates, warns in an interview that the current AI boom shows classic bubble characteristics, which could lead to significant economic downturns as seen in past cycles like 1929 or 2000. He explains that speculative enthusiasm, fueled by debt and overvaluation, often precedes a crash when rising rates or taxation force asset sales, causing widespread losses and recession. Dalio also outlines his "Big Cycle" theory, describing an approximate 80-year pattern where widening wealth gaps, massive government deficits, and shifting geopolitical power (like China's rise) create internal conflict and global instability. He emphasizes that we are in a late-cycle, transitional phase where traditional powers like the US and UK face decline. For personal wealth protection, Dalio advises diversification beyond cash into assets like stocks, bonds, real estate, and particularly gold, which he prefers over Bitcoin. While he holds about 1% of his portfolio in Bitcoin as a non-printable hard asset, he views gold as more secure from technological or governmental threats. Regarding AI's impact, Dalio believes it will disproportionately benefit capital owners, worsening inequality by replacing both physical and cognitive labor. He suggests that human intuition and emotional intelligence, combined with AI, will be key for future workers. On taxation, Dalio argues that wealth taxes are impractical and risk triggering asset sell-offs, reducing productive investment. He points to the UK as a cautionary example of debt, low productivity, and political strife. Geopolitically, Dalio foresees a more regionalized world, with the US showing weakness in prolonged conflicts like with Iran, akin to past imperial declines. The ideal outcome, he suggests, is coexisting powerful blocs (e.g., Americas, China-Asia Pacific) without major war.

marsbitHá 3h

In Conversation with Ray Dalio: We Are Currently in an AI Bubble, with 1% of My Portfolio in Bitcoin

marsbitHá 3h

Daily 7.2 Trillion KRW: Foreign Capital's Record Net Buying on Friday! Wall Street Says Headwinds for Korean Stock Fund Flows Have Subsided

South Korean stock market sees a dramatic shift in fund flows. On July 31, foreign investors made a record net purchase of approximately KRW 7.2 trillion in KOSPI stocks, marking a fundamental reversal from the persistent large-scale net outflows seen in previous months. This contributed to a significant narrowing of foreign net selling in July to KRW 9.8 trillion, down sharply from KRW 48.4 trillion in June and KRW 44.5 trillion in May. Simultaneously, domestic institutional pressure eased. South Korean pension funds and asset managers turned to a net buying position in July, purchasing KRW 1.0 trillion worth of KOSPI shares, contrasting with net sales in May and June. Market volatility is expected to be dampened by new financial regulations. Effective July 31, the Financial Services Commission tightened access for retail investors to single-stock leveraged ETFs by raising the minimum cash deposit requirement. Trading volumes for these products subsequently dropped to about 50% of their monthly average. Citigroup Research maintains its year-end KOSPI target of 10,000 points. The firm cites several supportive factors: the substantial easing of headwinds from capital outflows, a robust fundamental outlook for the semiconductor sector, historically low market valuations, strong economic fundamentals, and the potential for policy support from financial authorities if needed.

marsbitHá 3h

Daily 7.2 Trillion KRW: Foreign Capital's Record Net Buying on Friday! Wall Street Says Headwinds for Korean Stock Fund Flows Have Subsided

marsbitHá 3h

Breaking! OpenAI's Next-Gen AI Solves 10 Fields Medal-Level Problems

OpenAI's next-generation AI model Astra achieves breakthroughs in 10 long-standing mathematical conjectures. The results, including constructing the first known infinite, finitely presented non-sofic group—resolving a major question since 1999—and advancing the high-dimensional sphere packing problem beyond a 46-year-old barrier, are detailed in a 249-page paper. Key proofs have been formally verified using Lean 4. The AI also refuted a rigidity conjecture by Fields Medalist Alain Connes. According to OpenAI, generating these proofs cost under $2,000. Experts describe the findings as potentially Fields Medal-worthy and a landmark moment for both mathematics and AI, showcasing the model's ability to produce profound, human-like reasoning across diverse fields like group theory, geometry, and operator algebras.

marsbitHá 5h

Breaking! OpenAI's Next-Gen AI Solves 10 Fields Medal-Level Problems

marsbitHá 5h

How to Make Yourself Irreplaceable by AI Forever

This article argues that the primary threat from AI is not job replacement, but remaining trapped in "wage slavery"—financial dependence on employers. The path to becoming irreplaceable is not resisting AI, but becoming an "unemployable" individual who builds their own meaningful enterprise. The author identifies five key elements for this: Agency (acting without permission), Taste (judging what's worthwhile), Persuasion, Persistence, and Iteration. The solution is to stop being a "pawn" in someone else's game. To start, you must fundamentally change your identity and environment, then engage in rapid, real-world trial and error. While both coding and creating media (content) are powerful, content is more crucial. AI can generate assets, but true value lies in subjective, human-driven content that builds trust and narrative. The actionable advice is to carve out 15 minutes to answer foundational questions: 1) Uncover your "raw material"—what you know deeply or solve effortlessly. 2) Define your contrarian perspective—what common beliefs you think are wrong. The intersection of these answers is your direction. Finally, you must launch by publishing your first core idea immediately, using the feedback to iterate and develop the skills needed for a self-directed life and career.

marsbitHá 5h

How to Make Yourself Irreplaceable by AI Forever

marsbitHá 5h

Thanks to Dice Rolls, Bitcoin Keys Are Stored Offline, But Not Everyone Will Do It

The article discusses using dice rolls to generate secure Bitcoin wallet seeds, providing entropy independent of potentially flawed hardware random number generators. It explains that each fair dice roll offers about 2.585 bits of entropy, with around 50 rolls needed for a standard 12-word seed phrase and 99+ recommended for higher security. This method gained attention after a vulnerability was revealed in some Coldcard hardware wallets, where a faulty firmware RNG (dating back to 2021) compromised generated keys. The analysis notes that while a dice-generated main seed was safe from this specific flaw, other Coldcard functions (like creating paper wallets, backup keys, or passwords) could still be vulnerable if they used the defective RNG. The piece argues that while dice-based entropy is technically robust, the manual process is error-prone, tedious, and unrealistic for most new users, who might make mistakes in recording or inputting rolls. It concludes that while manual entropy generation should remain an option for advanced users, the long-term goal is to develop reliable, user-friendly hardware and software that securely generates randomness without requiring specialized knowledge. Coldcard users are advised to check their firmware version and replace any secondary secrets (like paper wallet keys) created with vulnerable devices, while also considering multi-signature setups with devices from different manufacturers for added security.

cryptonews.ruHá 9h

Thanks to Dice Rolls, Bitcoin Keys Are Stored Offline, But Not Everyone Will Do It

cryptonews.ruHá 9h

Trading

Spot
活动图片

Categorias popularesright-arrow

Etiquetas Popularesright-arrow