In the last two days, a post by X user Max For AI@MaxForAI has sparked heated discussion among netizens: "This is insane — now you can even livestream and watch how a 535B large model is being trained."
It turns out that Stanford professor and Simile AI founder Percy Liang@percyliang announced the launch of training for the Marin 535B-A23B model by Marin Open Lab, and the entire training process will be conducted publicly.
Marin has a total of 535 billion parameters and 23 billion active parameters. For this, Percy Liang and the team prepared a total of 18.75 trillion tokens of training data and deployed 11 sets of GB200 NVL72 systems, equivalent to approximately 792 GB200 GPUs.
The model is expected to train continuously for about 3 months, with a total training compute of approximately 2.7e24 FLOPs. After training is complete, the team will continue with the post-training phase.
In other words: For the next few months, everyone can watch a cutting-edge large model grow from scratch.
Also worth noting is that Percy Liang stated that before officially starting this training run, the team first trained a 4-stage Scaling Ladder, ranging from 1.6B-A61M (48B tokens) all the way up to 27.7B-A1.2B (926B tokens).
"Doing this serves two purposes: first, to use these small-scale experiments to discover and debug potential issues in advance; second, to predict our 'hero run' (main model training) based on the training performance of models at different scales."
Of course, Percy Liang also said, "This is by far the largest-scale training we have ever undertaken, so we are indeed expecting some unexpected situations to arise during the process."
Currently, the main model has officially begun running, and anyone can directly view the real-time training curves. Subsequently, training data, experiment logs, and engineering issues will also be continuously made public.
For example, on the data front, this includes the training data mixture ratios, data processing methods, and how data from different domains enters the model; on the engineering front, including training configurations, code, and experiment design; and the training process, including real-time loss changes, model states, and prediction results at different stages.
Simultaneously, Percy Liang also made public specific viewing channels.
View data composition: https://storage.googleapis.com/marin-public/held/harrier-k40-cluster-overview/2026.08.18/index.html?revision=uniform-sampling
Watch the training process live on Weights & Biases (wandb): https://wandb.ai/marin-community/marin_moe/reports/535B-A23B-18T-Token-Hero-Run-Scaling-Ladder--VmlldzoxNzc2MDM5Ng
View all details on GitHub: https://github.com/marin-community/marin/issues/8435
After understanding the details, it's not hard to see why Percy Liang's move has attracted such widespread attention. Because in the past, for models like GPT-4, Claude, and Gemini, no one knew exactly how they were trained behind the scenes; it was like a "black box."
People could only see the final model capabilities, benchmark scores, and limited descriptions in papers. The data mixture ratios, what failures occurred during training, which hyperparameters were effective, how the loss changed, and whether the Scaling law predictions were accurate—all these were unknown.
Marin is trying to change this. Perhaps model training can also be like papers and open-source software: observable, discussable, and collaborative.
Since this news was announced, the enthusiasm of netizens continues to grow.
Some netizens believe this embodies the true spirit of open source: "Even if problems arise or things deviate from expectations during training, nothing is hidden."
Other netizens remarked that even though training the model itself is massive, what's truly astonishing is that people can now, via the WandB platform, watch in real-time how a large model training effort worth tens of billions of dollars grows step by step?
"It's as if the tightly shut 'dark room' inside top AI labs has suddenly been opened, and everyone can see what's happening inside."
Over the next three months, perhaps the most anticipated thing is not the final model's capabilities, but observing how many times this model will experience "explosions," crashes, and unexpected situations during training...
Furthermore, some netizens believe this move not only provides a perspective to watch the entire model training process but even offers a platform for learning and discussion.
Netizen James Thewlis@jdthewlis has been following the training process in real-time. During this, he didn't understand "why there is a spike in norms (parameter norms) around approximately step 500?"
After figuring it out himself, he then answered his own question in the comments: "This spike is entirely caused by the router_bias. It is not a parameter obtained through gradient training but is part of the token balancing heuristic in MoE (Mixture of Experts) models. Therefore, it has a different dynamic pattern from ordinary model parameters."
There are many situations like this.
Additionally, it's worth mentioning that Percy Liang, as a Stanford professor, besides this hands-on practice, also teaches a theoretical course: "CS336: Language Models From Scratch," with the goal of enabling students to fully understand how a large language model is built.
It seems Percy Liang really wants to teach everyone how to "build" large models. Interested netizens can learn more about it.
Course link: https://www.youtube.com/playlist?list=PLoROMvodv4rMqXOcazWaTUHhq-yembLCV
So, what do you think? How do you view this livestream that truly brings large model training to the forefront? Feel free to leave your thoughts in the comments!
References:
https://x.com/MaxForAI/status/2091270116067750089
https://x.com/percyliang/status/2090918065634684997
https://x.com/CopyRebeldia/status/2091231950774112412?s=20
https://www.youtube.com/playlist?list=PLoROMvodv4rMqXOcazWaTUHhq-yembLCV
This article is from the WeChat public account "Almost Human" (ID: almosthuman2014), author: Following AI
