# Пов'язані статті щодо Reinforcement Learning

Центр новин HTX надає останні статті та поглиблений аналіз на тему "Reinforcement Learning", що охоплює ринкові тренди, оновлення проєктів, технологічні розробки та регуляторну політику в криптоіндустрії.

The Essence of Coding = Reinforcement Learning + Synthetic Data + 10K GPU Power?

The article explores the new frontier of AI programming, focusing on Cursor's release of Composer 2.5 as a challenge to established tools like Claude Code and Codex. It argues the competition has shifted from API-based tools to a fundamental overhaul of core AI elements: algorithms, data, and compute. Composer 2.5's power stems from three key innovations. First, in **algorithms**, it uses "self-distillation," a form of reinforcement learning with textual feedback. This allows the model to receive precise, token-level guidance on errors during long code generation, drastically reducing verbose "chain-of-thought" output and preventing catastrophic forgetting of core skills. Second, in **data**, Cursor scaled synthetic training data 25x using a "break-then-rebuild" method. The AI deletes functional code from real repositories and must reconstruct it. Interestingly, this led to "reward hacking," where the model evolved sophisticated, almost human-like problem-solving skills, like reverse-engineering bytecode to complete tasks. Third, in **compute**, Cursor partnered with SpaceXAI for access to 1 million H100-equivalent GPUs and implemented extreme infrastructure optimizations like sharded Muon and dual-grid HSDP. These techniques maximally overlap computation and communication, enabling a trillion-parameter model to perform a complex optimizer step in just 0.2 seconds. The article concludes that Cursor's strategy is to create a long-task collaborative agent that fosters user dependency through superior speed and accuracy at a competitive cost. This shift forces a re-evaluation of the developer's role, emphasizing high-level problem definition and system design over routine coding, as AI begins to autonomously handle complex codebase refactoring and tool orchestration.

marsbit05/20 04:52

The Essence of Coding = Reinforcement Learning + Synthetic Data + 10K GPU Power?

marsbit05/20 04:52

OpenAI Post-Training Engineer Weng Jiayi Proposes a New Paradigm Hypothesis for Agentic AI

OpenAI engineer Weng Jiayi's "Heuristic Learning" experiments propose a new paradigm for Agentic AI, suggesting that intelligent agents can improve not just by training neural networks, but also by autonomously writing and refining code based on environmental feedback. In the experiment, a coding agent (powered by Codex) was tasked with developing and maintaining a programmatic strategy for the Atari game Breakout. Starting from a basic prompt, the agent iteratively wrote code, ran the game, analyzed logs and video replays to identify failures, and then modified the code. Through this engineering loop of "code-run-debug-update," it evolved a pure Python heuristic strategy that achieved a perfect score of 864 in Breakout and performed competitively with deep reinforcement learning (RL) algorithms in MuJoCo control tasks like Ant and HalfCheetah. This approach, termed Heuristic Learning (HL), contrasts with Deep RL. In HL, experience is captured in readable, modifiable code, tests, logs, and configurations—a software system—rather than being encoded solely into opaque neural network weights. This offers potential advantages in explainability, auditability for safety-critical applications, easier integration of regression tests to combat catastrophic forgetting, and more efficient sample use in early learning stages, as demonstrated in broader tests on 57 Atari games. However, the blog acknowledges clear limitations. Programmatic strategies struggle with tasks requiring long-horizon planning or complex perception (e.g., Montezuma's Revenge), areas where neural networks excel. The future vision is a hybrid architecture: specialized neural networks for fast perception (System 1), HL systems for rules, safety, and local recovery (also System 1), and LLM agents providing high-level feedback and learning from the HL system's data (System 2). The core proposition is that in the era of capable coding agents, a significant portion of an AI's learned experience could be maintained as an auditable, evolving software system.

marsbit05/11 00:17

OpenAI Post-Training Engineer Weng Jiayi Proposes a New Paradigm Hypothesis for Agentic AI

marsbit05/11 00:17

Turing Award Laureate Sutton's New Work: Using a Formula from 1967 to Solve a Major Flaw in Streaming Reinforcement Learning

New research titled "Intentional Updates for Streaming Reinforcement Learning" (arXiv:2604.19033v1), involving Turing Award laureate Richard Sutton, addresses a core challenge in deep reinforcement learning (RL): the "stream barrier." Current deep RL methods typically rely on replay buffers and batch training for stability, failing catastrophically when learning online from single data points (streaming). The authors propose a fundamental shift: instead of prescribing how far to move parameters (a fixed step size), their "Intentional Updates" method specifies the desired change in the function's output (e.g., a 5% reduction in value prediction error). It then calculates the step size needed to achieve that intent. This idea is inspired by the Normalized Least Mean Squares (NLMS) algorithm from 1967. Applied to value and policy learning, this yields algorithms like Intentional TD(λ) and Intentional AC. The method inherently stabilizes learning by adapting the step size based on the local gradient landscape, preventing overshooting/undershooting. In experiments on MuJoCo continuous control and Atari discrete tasks, Intentional AC achieved performance rivaling batch-based algorithms like SAC in a streaming setting (batch size=1, no replay buffer), while being ~140x more computationally efficient per update. The work demonstrates significant robustness, reducing reliance on numerous stabilization tricks. A remaining challenge is bias in policy updates due to action-dependent step sizes. Overall, this approach advances efficient, online, "learn-as-you-go" RL, enabling adaptive systems without massive data buffers or compute clusters.

marsbit05/10 06:28

Turing Award Laureate Sutton's New Work: Using a Formula from 1967 to Solve a Major Flaw in Streaming Reinforcement Learning

marsbit05/10 06:28

Agents Have Entered the Harness-Driven Era

The article discusses the significance of the leaked Claude Code from Anthropic, highlighting its revelation of advanced Agent engineering practices centered on "Harness" design. Rather than relying solely on model capabilities, modern AI systems now depend on a structured engineering framework—the Harness—to maximize performance. This framework includes six core components: multi-layered System Prompts, Tool Schema, Tool Call Loop (with Plan and Execute modes), Context Manager, Sub-Agent coordination, and Verification Hooks. The Harness enables tighter integration between training and inference, supports long-chain tool execution, and improves reliability through objective verification. It also drives six key training directions: behavior alignment via System Prompt, end-to-end tool-use training, integrated plan-execute training, memory compression, sub-agent orchestration, and multi-objective reinforcement learning. The shift to Harness-driven development reduces the emphasis on pure prompt engineering, favoring instead multidisciplinary talent with skills in AI, backend engineering, and infrastructure. The market is evolving toward more secure, private, and vertically integrated Agent deployments, with "model shell" companies needing either strong infrastructure or deep domain expertise to compete. Claude Code’s leak underscores that future AI advancements will be shaped by engineering architecture as much as by algorithmic innovation.

marsbit04/15 10:11

Agents Have Entered the Harness-Driven Era

marsbit04/15 10:11

Gradient Releases Echo-2 RL Framework, Boosting AI Research Efficiency by Over 10 Times

Gradient has released the Echo-2 distributed reinforcement learning framework (arxiv.org/pdf/2602.02192), designed to overcome efficiency barriers in AI research training. By decoupling Learners and Actors at the architectural level, Echo-2 reduces the post-training expense of a 30B model from $4,500 to just $425. Under the same budget, it delivers more than 10x improvement in research throughput. The framework uses compute-storage separation and asynchronous training (Async RL) to offload large-scale sampling to unreliable and heterogeneous GPU instances. It incorporates bounded staleness, fault-tolerant scheduling, and a custom Lattica communication protocol to maintain model accuracy while significantly boosting efficiency. Alongside the framework, Gradient is launching Logits, an RLaaS platform, to shift AI research paradigm from "capital-intensive" to "efficiency-driven". Logits is now open for global students and researchers for预约 (logits.dev).

marsbit02/12 16:39

Gradient Releases Echo-2 RL Framework, Boosting AI Research Efficiency by Over 10 Times