Everyone says Claude Code wastes tokens, but how much exactly? Finally, someone has put numbers to it.
There was a recent interesting comparative experiment from the Composio team. They used the same model, Kimi K3, and ran it in three different agent frameworks (harnesses)—Claude Code, Hermes, and Kimi Code—testing 28 identical tasks.
The results showed similar task success rates for the three harnesses: Kimi Code succeeded in 22 out of 28 tasks, Hermes in 21, and Claude Code in 20. The gap isn't large.
The real difference was in token consumption. For the same task, token usage across different harnesses varied by up to 30 times!
Looking at the median, Kimi Code used about 61,000 tokens, Hermes around 67,000, while Claude Code soared to 340,000—roughly 6 times that of Kimi Code.
Based on the price of $3 per million input tokens for Kimi K3 (input tokens typically account for about 95% in agent workflows), the average cost per task is approximately: Kimi Code $0.22, Hermes $0.28, and Claude Code $2.00. The difference is evident.
Speed also differed. The median time: Hermes was fastest at 179 seconds; Kimi Code 297 seconds; Claude Code 348 seconds.
So, Hermes was the fastest, while Kimi Code was the most token-efficient—they didn't overlap.
From this, the Composio team drew a straightforward conclusion: If you want to reduce agent costs, first look at which harness you're using, rather than rushing to change models. In their data, the harness itself could create a 9x cost difference, while model performance was quite similar.
Sebastian Raschka saw the results and posted about them, saying it aligns with his previous observations using Qwen3.6: Claude Code often uses 2 to 3 times more tokens than many other harnesses, with similar success rates.
He suggested several possible reasons: Is it not optimized? Is there a bug? Or was it intentionally designed this way (perhaps aiding with harder tasks)? He said more time is needed to investigate carefully.
He then added observations from his article last month on writing a local coding agent. At that time, he analyzed why Claude Code uses more tokens and found the difference mainly stems from input tokens, not output tokens. In other words, Claude isn't writing twice as much content. Logs show that Claude's harness repeatedly feeds more context back into the model across multi-turn interactions, including previous messages, tool calls, command outputs, and file contents. For example, in one run, Claude used about 578,000 input tokens but only about 4,500 output tokens, spanning 25 turns. Thus, a more likely explanation is that Claude's harness accumulates or accounts for a larger prompt-side history during multi-step agent runs.
These test results seem to reveal an undeniable trend: The importance of the harness is now on par with the model itself.
A recent paper (from Writer, a company building an enterprise AI Agent platform) systematically highlights this point: Using controlled experiments, it demonstrates that switching the harness layer is more effective at cutting costs than switching models, benefiting all models.
Specifically, they conducted a rigorous "controlled variable" experiment: With 22 enterprise tasks and 6 base models (Claude Sonnet 4.6, Gemini 3.1, Gemini Flash 3.5, Qwen 3.6, GLM 5.1, Palmyra X6) held constant, they only replaced the orchestration layer—swapping the traditional production-grade agent loop for Writer's own Harness.
The results showed: Average task cost reduced by 41% ($0.21 → $0.12), median latency shortened by 44% (48s → 27s), token consumption decreased by 38% (14.2k → 8.8k), while task completion quality remained largely unchanged (0.78 → 0.81, considered insignificant due to small sample size). In terms of cost-effectiveness, the quality gained per dollar increased by 82%, and the number of tasks completed per million tokens jumped from 54.9 to 92.0.
So, as models become like "utilities," is the harness the air conditioner that determines your electricity bill? In other words: People used to say "the model is the product," but now it's "the harness is the product"?
Since the harness is so important, perhaps future accounting needs to be more granular.
Some point out that we need to add a "harness tax" item to existing benchmarks. Especially considering that once tool calls and retries enter a loop, this tax grows non-linearly.
In other words, in the future agent competition, the first half is about "whether it can be done," and the second half will be about "who does the same thing more efficiently"—and the secret to saving money lies not in the model, but in the harness.
Have you had similar experiences while running agents? Feel free to discuss in the comments.
This article is from the WeChat official account "Almost Human" (ID: almosthuman2014), by Almost Human.
