# Transformer İlgili Makaleler

HTX Haber Merkezi, kripto endüstrisindeki piyasa trendleri, proje güncellemeleri, teknoloji gelişmeleri ve düzenleyici politikaları kapsayan "Transformer" hakkında en son makaleleri ve derinlemesine analizleri sunmaktadır.

10000 Hours of Human Data, Trains the World's First Whole-Body Mobile Manipulation Implicit World-Action Model

"Being-M0.7" is the world's first latent world-action model for whole-body mobile manipulation in humanoid robots, developed by Zhi Zai Wu Jie (Beyond Being). Trained on over 10,000 hours of human-centric multimodal data, it aims to overcome key industry challenges: the high cost and scarcity of real robot demonstration data, the computational inefficiency of pixel-level video prediction models, and the lack of full-body coordination in existing approaches. The model is based on a Vision-Motion Mixture-of-Transformers (MoT) architecture, which allows training on a mixture of paired video-motion data, pure video data, and pure motion sequences. A key design is a unified motion representation that bridges human and robot morphology, enabling knowledge transfer from vast human behavioral data to specific robot control. After pre-training on human data, the model is adapted to a real robot (Unitree G1) using a small amount of teleoperated demonstration data via a lightweight "Action Expert" module. This process decouples low-frequency world planning from high-frequency motion control. The model was tested in four challenging real-world demos: fishing a toy fish from water (liquid interaction), retrieving an object using a mirror (visual reasoning), a multi-step pick-and-place task, and obstacle avoidance while carrying a box. In comparative tests against other models, Being-M0.7 showed stronger performance in tasks requiring indirect reasoning and full-body coordination. This work represents a shift in humanoid robotics competition from hardware spectacle to model capabilities rooted in scalable data and training paradigms, using human experience as a foundation for physical world understanding and action.

marsbit07/15 01:48

10000 Hours of Human Data, Trains the World's First Whole-Body Mobile Manipulation Implicit World-Action Model

marsbit07/15 01:48

Deforming the Transformer, LLMs Become Smarter

A new research paper proposes "Tapered Language Models (TLMs)," a method that improves large language model performance without adding any parameters. It challenges the standard Transformer design where each layer has the same number of parameters ("feed-forward network" width). Building on evidence that layers are not equally important—earlier layers handle foundational information like grammar, while later layers often reinforce existing judgments—the researchers suggest reallocating model capacity from later to earlier layers. The core idea is to make the layer width taper off monotonically from start to end, keeping total parameters and compute constant. Experiments compared linear, cosine, and sigmoid tapering curves on a 440M parameter model. The cosine curve (e.g., starting width 1.5x baseline, ending 0.5x) achieved the best result, reducing perplexity by 1.84 points compared to the uniform baseline—a significant gain at zero cost. This finding proved robust across four different model architectures (including gated attention and memory-augmented models) and at larger scales (760M and 1.3B parameters), consistently improving performance on commonsense reasoning and language modeling tasks without harming long-context retrieval ability. The work highlights a long-overlooked design dimension: optimal parameter allocation across depth. It offers a "free lever" for efficiency, potentially applicable beyond language models to vision Transformers and diffusion models. The study was conducted by researchers from Mila, Cornell University, and the University of Montreal.

marsbit06/29 12:53

Deforming the Transformer, LLMs Become Smarter

marsbit06/29 12:53

The Eight Fathers of Transformer: Where Are They Now?

The founding authors of the seminal 2017 paper "Attention Is All You Need," which introduced the Transformer architecture, have all departed from Google and are now pioneering diverse ventures across the AI landscape. Their current paths highlight the field's evolution and fragmentation. Ashish Vaswani co-founded Essential AI after a stint at Adept, but recent reports suggest he and his team are being recruited by Nvidia. Noam Shazeer left Google twice, first to found Character.AI—later acquired by Google—before recently joining OpenAI. Niki Parmar, after co-founding both Adept and Essential AI with Vaswani, is now at Anthropic. Jakob Uszkoreit is the CEO of Inceptive, applying AI to RNA and drug design. Llion Jones co-founded Sakana AI in Tokyo, exploring collective intelligence from smaller models. Aidan N. Gomez is the CEO of Cohere, focusing on enterprise AI solutions. Łukasz Kaiser remains in research at OpenAI, contributing to models like GPT-4 and o1. Illia Polosukhin co-founded the NEAR Protocol blockchain, positioning it for an AI-agent economy. Despite their divergent paths—spanning biotech, enterprise software, blockchain, and foundational research—a shared sentiment persists: the Transformer is not the final answer. Each founder continues to seek the next significant architectural breakthrough, driven by the same innovative spirit that created the modern AI era.

marsbit06/28 05:30

The Eight Fathers of Transformer: Where Are They Now?

marsbit06/28 05:30

He Kaiming's Team's New Work: After Deleting VAE and Private Data, Text-to-Image Generation Becomes Even Stronger

KaiMing He's team introduces **MiniT2I**, a minimalist text-to-image (T2I) model that challenges the complexity of mainstream approaches. It eliminates components commonly considered essential: the VAE encoder-decoder, AdaLN conditioning mechanisms, auxiliary losses, private training data, and post-training alignment stages like RL/DPO. Instead, it uses a pure flow-matching objective trained directly on RGB pixels. The model employs a simplified **MM-JiT** Transformer architecture. It removes AdaLN blocks for conditioning and instead prepends two lightweight text adapter blocks to a standard pre-norm Transformer, allowing frozen T5 text features to adapt to the denoiser. Training follows a two-stage, LLM-like paradigm using only public datasets: pre-training on LLaVA-recaptioned CC12M for coverage, followed by fine-tuning on ~120k high-quality image-text pairs. With just 258M parameters (B/16), MiniT2I achieves competitive scores (0.87 on GenEval, 84.2 on DPG-Bench), outperforming larger pixel-space models. Scaling to 912M parameters (L/16) yields results comparable to SD3-Medium (~2B parameters) in style, composition, and imagination, though it lags in text rendering and named entities due to public data limitations. Key advantages include lower computational cost (~570 GFLOPs vs. ~1379 for latent models) and architectural simplicity. Acknowledged limitations include patch boundary artifacts in pixel space, side effects of high CFG scales, resolution ceilings for sequences longer than 1024 tokens, and the aforementioned data bottlenecks. The work demonstrates that high-performance T2I generation is possible with a radically simplified, publicly reproducible baseline.

marsbit06/22 10:17

He Kaiming's Team's New Work: After Deleting VAE and Private Data, Text-to-Image Generation Becomes Even Stronger

marsbit06/22 10:17

Google's AI Talent Loss: A Stress Test or a Prelude to an 'Obituary'?

The departure of high-profile AI talent from Google, including transformer pioneer Noam Shazeer and AlphaFold's John Jumper, has sparked debate about the company's competitive position. However, this is less a sign of decline than a pressure test typical of the pre-IPO talent wars among AI giants like OpenAI and Anthropic. Google's true strength lies not just in models but in its unparalleled full-stack position: extensive AI research, proprietary infrastructure (TPU, Google Cloud), a massive user base across products (Search, YouTube, Android), and established enterprise relationships. It is simultaneously a competitor and a key infrastructure provider for rivals. While facing the innovator's dilemma inherent to protecting its core search business, Google is aggressively integrating AI across its ecosystem. The narrative shouldn't focus on individual departures but on Google's systemic advantages—its deep talent pool, integrated product strategy, and capacity to compete across every layer of the AI stack for the long term.

marsbit06/21 08:55

Google's AI Talent Loss: A Stress Test or a Prelude to an 'Obituary'?

marsbit06/21 08:55

Two Legends Lost in Three Days: Is Google's AI Talent Dam Cracking?

In three days, Google lost two AI legends. On June 18, Noam Shazeer, co-author of the seminal "Attention is All You Need" paper and Gemini co-lead, left for OpenAI. Just 48 hours later, John Jumper, 2024 Nobel laureate and AlphaFold lead, departed DeepMind for Anthropic. This follows Andrej Karpathy joining Anthropic in May. These moves highlight a structural trend: top AI talent is concentrating at mission-driven, pre-IPO firms like OpenAI and Anthropic, while Google becomes a primary source. The exodus stems from a core mission mismatch. Google's ad-centric model often subordinates AI research to product and revenue goals, creating friction for pioneers like Shazeer, who returned in 2024 only to leave again. In contrast, OpenAI and Anthropic offer singular focus on pushing AI boundaries, whether towards AGI or safety-aligned models, which deeply appeals to top researchers like Jumper. Financial incentives amplify the pull. With both OpenAI and Anthropic nearing IPO, employees stand to gain immensely from equity, an upside Google's mature stock cannot match. Furthermore, the 2023 merger of Google Brain and DeepMind, intended to consolidate strength, has instead created cultural tension and slowed the path from research to product, as evidenced by Gemini's pace. This talent redistribution is reshaping the AI landscape. While Google retains vast data and compute resources, its true crisis is the quiet, continuous loss of the people who define the field's future. The real moat in AI is not infrastructure, but the concentration of brilliant minds—a battle Google is currently losing.

marsbit06/20 04:02

Two Legends Lost in Three Days: Is Google's AI Talent Dam Cracking?

marsbit06/20 04:02

The Entire Internet Hails Noam's Joining, But OpenAI's Loss Bill Just Got Thicker

While the AI community celebrates Noam Shazeer, co-author of the "Attention Is All You Need" paper, joining OpenAI as Head of Architectural Research, the company's audited financials reveal a starkly different reality. In 2025, OpenAI reported $13.07 billion in revenue but a massive $20.92 billion operating loss. Even excluding a one-time accounting charge, the cash burn is severe, with $3.7 billion consumed in Q1 2026 alone. This high-profile hiring occurs against a backdrop of significant internal research talent drain, with key founders and researchers departing as the company's focus shifts from exploratory research to product iteration. Meanwhile, OpenAI's fundamental business model faces a deep crisis. It paid Microsoft $10.59 billion for compute in 2025, while its vast user base of 9 billion weekly actives includes only 50 million paying customers, making growth a direct driver of escalating costs. The article argues Shazeer's recruitment is less about technical necessity and more about crafting a compelling narrative for OpenAI's upcoming IPO, aiming to justify a rumored $1 trillion valuation to future public market investors. It contrasts OpenAI's strategy with Anthropic's reported path to profitability, which relies on a strong enterprise customer base and cost control, rather than star-powered narratives. Ultimately, the piece concludes that while Shazeer's architectural work may take 1-2 years to materialize, OpenAI's financial clock is ticking much faster, with its massive losses undercutting the celebratory headlines.

marsbit06/19 02:27

The Entire Internet Hails Noam's Joining, But OpenAI's Loss Bill Just Got Thicker

marsbit06/19 02:27

Breaking: Google Gemini Co-Head Poached by OpenAI

In a significant industry move, Noam Shazeer, a former DeepMind researcher and co-lead of Google's Gemini project, has officially joined OpenAI as the Lead for Architecture Research. Shazeer is a legendary figure in AI, best known as a core author of the seminal 2017 paper "Attention Is All You Need," which introduced the Transformer architecture foundational to modern models like GPT and Gemini. Shazeer announced his move on social media, expressing excitement to join OpenAI's team while also thanking his former colleagues at Google. His career spans over 18 years at Google, where he contributed to key technologies including the Mixture of Experts (MoE) architecture, Mesh TensorFlow, and the LaMDA model. He left Google in 2021 to co-found the conversational AI company Character.AI, which achieved a multi-billion dollar valuation. In 2024, he returned to Google's DeepMind via a major technology licensing deal to help lead the Gemini project. His departure is viewed as a substantial loss for Google and a major talent acquisition for OpenAI, which is intensifying its competition with rivals like Anthropic. At OpenAI, Shazeer will focus on exploring next-generation AI model architectures and the evolution beyond the current Transformer paradigm. The move underscores the fierce competition for top AI research talent as companies race to develop advanced AI systems.

marsbit06/18 04:19

Breaking: Google Gemini Co-Head Poached by OpenAI

marsbit06/18 04:19

After 10 Years, Altman Finally Has the Person He Wanted

After a decade of waiting, OpenAI CEO Sam Altman has finally secured his desired collaborator: Noam Shazeer, a legendary AI researcher and co-author of the seminal "Attention Is All You Need" paper that introduced the Transformer architecture. Shazeer has announced his departure from Google to join OpenAI as Head of Architectural Research. Shazeer, a crucial early Google employee who returned to Google DeepMind in a high-profile $2.7 billion deal two years ago, confirmed his move on social media platform X. Altman expressed his long-standing desire to work with Shazeer, stating the 10-year wait would be worth it. OpenAI's research lead, Mark Chen, welcomed Shazeer, highlighting his foundational work on Transformer, Mixture-of-Experts (MoE) models, and efficient decoding, which have profoundly shaped modern AI. His departure is seen as a significant blow to Google's Gemini project, where he served as a technical co-lead. Industry observers note this move represents a major win for OpenAI in the ongoing AI talent war, with some quipping that OpenAI acquired his expertise "for free" after Google's massive investment.

marsbit06/18 04:15

After 10 Years, Altman Finally Has the Person He Wanted

marsbit06/18 04:15

Tsinghua '00s Alumnus Wang Guan's New Work: Disrupting Transformer Pretraining Models with 1/900 Tokens, 1/432 Compute Power

Tsinghua alumnus Wang Guan's team proposes HRM-Text, a novel pre-training paradigm using a Hierarchical Recurrent Model to replace standard Transformers. With just 1B parameters and 40B unique tokens trained at a cost of ~$1500, HRM-Text achieves performance comparable to 2B-7B open-source models, using up to 900x fewer tokens and 432x less estimated compute. Key innovations include a dual-timescale recurrent architecture for greater effective depth, a task-completion objective training only on answer tokens with PrefixLM masking, and techniques like MagicNorm and Warmup Deep Credit Assignment for stability. Evaluations show strong results on benchmarks like MMLU (60.7%) and GSM8K (84.5%). The work highlights how architectural priors and targeted objectives can lower pre-training barriers, though limitations include knowledge-reasoning coupling, fixed compute per token, and scalability beyond 3B parameters.

marsbit05/26 03:16

Tsinghua '00s Alumnus Wang Guan's New Work: Disrupting Transformer Pretraining Models with 1/900 Tokens, 1/432 Compute Power

marsbit05/26 03:16

活动图片

Bitcoinright-arrow

Trading Strategiesright-arrow