Rendered at 14:18:52 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
redmalang 1 days ago [-]
We have an internal proxy (that I've been meaning to open source for ages) that routes all llm usage at our company, which allows us to see data in realtime. Its been fascinating how rapidly Pi has been adopted. Moreover since its pretty hackable, we've been able to automatically aggregate context from pi sessions, which has resulted in Pi efficacy being higher as more people use it, putting in place a interesting virtuous loop.
I didn't expect this outcome: for whatever reason I assumed proprietary harnesses fine tuned to work with a companies' models would work better?
ps/random aside: there is something slightly off about Pi's edit command, we are planning to investigate this further and patch this as we have quite a few session traces now..
lukax 1 days ago [-]
Yes, this is a known issue. A significant amount of Edit tool calls fails in Pi witg newer models.
I don't understand peoples expectation. If our language skills were as non-ambiguous as a coding language, we'd have solved world hunger by now.
So why would we expect all these bizarre bizantine language models to all conform to how a request is both made, expected and massaged.
For awhile, I was getting bizarre opencode tool errors where the only problem was the model was passing in a "1.0" or "0.0" where the harness dutifully wanted an integer. Of course 0.0 is the same as an integer in practical operations.
sgc 1 days ago [-]
I have one core complex task where there are a number of simple errors like this. The easiest thing for me was to just have a post-processing script that performs: lint > mark known fail-early results > fix common errors (all formatting or type errors right now) > relint > final pass / fail > pass to next step (retry with tweaks / user in the loop since I don't have the token budget to just blindly keep trying the same thing).
cindyllm 1 days ago [-]
[dead]
twalla 1 days ago [-]
[dead]
redmalang 1 days ago [-]
Aside 2:
Anecdotally we found that Pi performs more or less on par with native harnesses at lower cost on decently specified prompts. It is also phenomenal at context cacheing especially on Deepseek models (its hard to precisely attribute credit here are my understanding is this is a DS speciality).
But it fails much worse on poorly drafted prompts. I'm generalising but native harnesses seem to be better kind of flailing along on those.
swingboy 1 days ago [-]
Is it because Pi’s default system prompt is so simple?
cyanydeez 1 days ago [-]
of course it is; these entire systems are about priming the LLM to perform X task; if it's a coding harness, giving it very little priming for a task is going to make it grasp at straws.
sgc 1 days ago [-]
Can anybody share a tested system prompt they use for general coding tasks in pi?
cyanydeez 21 hours ago [-]
clone opencode and ask your ai yo find it. i did that then just wrote my own primary open code agent.
sanderjd 1 days ago [-]
I keep hesitating to use Pi because I primarily use Claude and I worry it will be much more expensive to pay for API usage rather than the bundled subscription usage. But I haven't actually evaluated the cost tradeoff. Should I get over this hesitance and dive into Pi?
UpsideDownRide 8 hours ago [-]
There are ways to do subscription plan on Pi through extensions.
Sphax 1 days ago [-]
> we've been able to automatically aggregate context from pi sessions
what do you mean by this ? do you rewrite the context in your proxy ?
N_Lens 1 days ago [-]
I imagine they're capturing all the tokens+context from pi sessions from every developer.
redmalang 1 days ago [-]
Yeah.
pbowyer 1 days ago [-]
Can you say more? This sounds like it could be more useful than trying to build a knowledge base (say qmd) and then get the models to search it for the right context each time.
redmalang 1 days ago [-]
This is in fact what we do (with higher order abstractions now built on top of this). This builds self evolving interactive knowledge base and puts it into a QMD searchable index. The indexer is already open source: https://github.com/jibs/duffel
1 days ago [-]
Schlagbohrer 1 days ago [-]
Can you explain context aggregation here more? What is that? What are you doing and how is it helping?
yodon 1 days ago [-]
I wish they'd do a follow-on post drilling into the impact of the programming language on cost-per-task, specifically looking at cost to complete tasks in mainstream strongly typed languages (eg. C#, TypeScript) vs dynamic languages (eg. Python, JavaScript). Does the additional verbosity of the language help or hurt cost per task?
trollbridge 1 days ago [-]
I don't have hard data, but we have shifted to Rust and Swift (for frontend UI) for the bulk of our dev simply because it is a lot more predictable, easier for tool calls to edit, the build steps produce easier output for the agent to loop on, the tests are easier to write/get results from, etc., although I am mostly measuring this in time, not cost.
Once the thing is rock-solid it's relatively easy to do a Swift->HTTP/HTML/CSS/React/TypeScript conversion.
rootatixww3 1 days ago [-]
for that you would need to compare the same task implemented in two different languages - C# and Python for example, no?
cpard 1 days ago [-]
This was mostly because Sonnet 5 worked longer and read more to get there, consuming 1.9x more tokens.
I have experienced similar behavior between opus and haiku when benchmarking Dara engineering tasks. The “cheaper” model takes many more turns to figure out the task and this is without taking into account other important factors.
Another interesting behavior that I observed is that Haiku tended to cheat more maybe because it was having a harder time to find the root cause of the problem.
Benchmarking and evaluation of agentic systems is very interesting and if there’s one thing that someone should keep from the Databricks post is how important is for everyone to build and run their own.
feltfriction 1 days ago [-]
[flagged]
anentropic 1 days ago [-]
> the results showed clear clustering of the models and harnesses into 3 capability tiers
pretty sure the only thing making that 'clear' is the coloured stripes, if you took that away it'd look like two tiers
good result for GLM 5.2 though
and Sonnet 5 seems like a waste of time
HarHarVeryFunny 1 days ago [-]
Wow!
It's great to see a large-scale real-world benchmark from a user of these tools, as opposed to the the benchmaxxed results from the vendors themselves. Also great to see different harnesses being tested, with considerably different results.
Definitely a few surprises here:
1) GLM 5.2 using Pi performs identically in terms of pass rate (~87.5%) to Opus 4.8 high using Claude Code, but significantly cheaper ($1.25 per task vs $2)
2) Absolute best pass rate (90%) was from Opus 4.8 x-high using Pi, beating out Opus 4.8 using Claude Code
3) Pareto frontier performance from any of the models (Opus 4.8, GPT 5.5, GLM 2.5) was using Pi rather than native harnesses
Apparently Pi used 3x less context than Claude Code, and one takeaway is to use Pi regardless of what model you are using. The other takeaway is that in real-world performance GLM 5.2 is the equal of Opus 4.8 unless you run Opus 4.8 on x-high in which case you can eke out a 2.5% increase in pass rate at the expense of doubling your cost over GLM 5.2
zaphar 1 days ago [-]
The catch-22 here is that you still save money using Claude Code directly here if you are on one of their subscription plan due to how heavily they are subsidizing that. Using Pi means you are using the API which is both a more accurate pricing model but also a more expensive one.
calumcl 24 hours ago [-]
Databricks wouldn't be using a subscription plan though due to the size of their engineering team which is what matters here. Free lunch is over for basically all providers' biz coding plans
HarHarVeryFunny 1 days ago [-]
True, but OpenAI are OK with using their subscription plans with Pi, so GPT 5.6 with Pi may be a good combination.
There seems to be a lot of good buzz about GPT 5.6 on Twitter - people (incl. OpenCode team) preferring it to Fable 5.
ryt 1 days ago [-]
Pi has a subscription flow for Claude Code. Are they goading people to get banned or did they figure out a way to work with the subscription?
sgc 1 days ago [-]
It seems like they are actually using the subscription providers' respective cli tools and managing context for them. In which case I believe it is not against the ToS any more than invoking codex cli from a custom python script would be? I personally cannot find an openai-hosted clarification, just on hn and x.
ryt 21 hours ago [-]
Awesome - that's what I do in my own harness for agy. Hopefully someone yells if they get banned.
agentdev001 1 days ago [-]
Obligatory yes, but only if you're subscription-based and not pay-per-token as enterprise users are.
lukax 1 days ago [-]
Could it be that users of Pi are more senior and know better how to prompt and that's why the pass rate is higher?
rootatixww3 1 days ago [-]
they explain this is a benchmark, all models/harnesses receive the same prompt
Schlagbohrer 1 days ago [-]
Anthropic is not beating the charges that they inflate token consumption with their own harness given these findings that Pi is 2.2x more efficient at token management. Big "toothpaste ads tell you to use way too much toothpaste" energy.
falaki 2 days ago [-]
1) Many models are now competitive at the top tier, including open source.
2) GLM 5.2 in particular was a major step forward in open source coding agent performance,
3) Harnesses make a huge difference in cost-performance.
4) Cheaper per-token does not imply cheaper per-task.
falaki 2 days ago [-]
Also they suggest every company should build their own benchmark and repeat these tests with new models instead of relying on the SWE bench.
fmind-dev 1 days ago [-]
It takes time and effort to build such benchmark. It works at Databricks scale, I'm not sure smaller companies are ready to invest on internal benchmarks.
But they are more vendor neutral, now they don't sell their own model. It's interesting from a benchmark point of view.
yigitcan07 1 days ago [-]
Would be great to see time spent per task per model. Especially since article references 390+ tokens per second for GLM5.2.
virgilp 1 days ago [-]
It seems that pass rate decreases with effort increase, on GPT5.5? This is highly counter-intuitive and I don't see any explanation, any idea why they'd get this result?
cheesecakegood 22 hours ago [-]
Looks to be within the realm of natural variance expected from naturally variable models, ie error bars.
pianopatrick 1 days ago [-]
Seems like for a hobby project $1 or $2 per task would add up a bit, depending on how many tasks you need to do. I mean it makes sense for a software company
nkzd 1 days ago [-]
How is Pi so efficient? You'd think agent harness made by model makers would perform better.
HarHarVeryFunny 1 days ago [-]
Different incentives. Claude Code makes more money for Anthropic by generating larger contexts. Anthropic also recently changed their tokenizer so the exact same code input creates 30% more tokens, so there's a pattern there.
cheesecakegood 22 hours ago [-]
I think in general the model makers and to some extent their clients want a slightly higher pass rate over efficiency. This makes sense: for critical first week impressions clients notice pass rate much more, and only later start to grapple with cost. For example this is why High is the default reasoning for Fable, not Medium, and that choice of priorities propagates throughout the stack.
cyanydeez 1 days ago [-]
they spent very little time validating what they're doing, and it works by not doing much of anything. If you spent a month figuring out a specific model+harness, you'd be way more efficient.
Other hanresses are doing overkill so they can work with any model.
bwfan123 23 hours ago [-]
Doesnt this prove that there is really no moat in proprietary models for coding usecases, or the gaps is narrowing ? Also, since GLM5.2 can be run equally on amd and nvda, I guess there is no hw moat either. Further, switching costs for users is minimal not only in agents but models too. So there is really no stickiness or user preference involved. For this usecase I think it is a race to the bottom for costs in a good way for developers.
falaki 21 hours ago [-]
Yes exactly.
jkwang 1 days ago [-]
The repo-scale angle is the useful part here. Small synthetic tasks miss a lot of the integration and context retrieval failures you only see in a codebase this large.
zipy124 1 days ago [-]
I'd like to know what their config with pi is like. Is it vanilla, is it oh my pi etc.... This seems like important info.
maxdo 24 hours ago [-]
curious to see tests if a new king of efficiency in town : cursor grok 4.5 and their harness too. Quite impressed by pi.dev
Its shocking how cost per token does not correlate with cost per task, it's wild to see opus and glm nearby on $ per task axis
throwa356262 1 days ago [-]
Is there any technical analysis of why contex grows slower in Pi compared to codex and CC?
rootatixww3 1 days ago [-]
different system prompts, codex will have system prompt telling the model to gather a lot more context before starting work
zkmon 1 days ago [-]
> Databricks’ multi-million line codebase
The combined size of codebases for the underlying opensource products (Apache Spark etc) might be around 1M lines, I think. Why does the orchestration/management layer, that is "databricks", exceed the sizes of the core products?
trollbridge 1 days ago [-]
Because if they're like previous places I worked out with proprietary commercial codebases, the code is old, has been around a long time, and trying to shrink it is not a priority.
It's a good stress test for the LLM because it is not an "ideal" codebase.
alexott 1 days ago [-]
:facepalm: "orchestration/management layer" that just runs dozens of millions of VMs per day, across the three clouds, with things like model serving, AI gateway, OLTP (Lakebase/Neon), Unity Catalog, etc.
Forget Databricks == Apache Spark...
ozgrakkurt 1 days ago [-]
Old codebase, you always add code and never remove it. So it is expected to be like this.
Deleting code is difficult and almost never makes sense afaik
tijs 1 days ago [-]
Every line you delete is a line you no longer need to maintain. We aggressively prune old code in our apps and it has definitely helped with maintainability. For a mobile app it’s also code you don’t ship so that’s a nice bonus which I guess is not much of an argument on a backend codebase
wwind123 1 days ago [-]
On one hand, I understand that some old code is hard to delete because it's hard to detangle a lot of the legacy dependency. On the other hand, too much useless old code existing in the code base by itself could become a big maintenance burden for both humans and AI. In some cases at some point it might become more economical to just invest a bunch of resource to detangle the dependencies to be able to remove the old code.
hasyimibhar 1 days ago [-]
At huge companies, it's hard to prioritize this because it's hard to pin dollar values to removing legacy code, while it's easier to show how building feature X will earn the company $Y amount of revenue. And because of that, there is also no incentive to do it, you don't get promoted by deleting old code, you get promoted by showing how your effort helped contribute to company revenue. At my previous company (100+ engineers, hundreds of microservices), teams that regularly clean legacy codebase tend to be platform teams (cost centers), while teams that struggle to get these prioritized are product vertical teams (revenue centers).
appplication 1 days ago [-]
LoC isn’t a super helpful metric so I think the better question is why is the headline using it. I can say I’ve personally created about 200k LoC code in the last 5 years and most of that has some value. But it really doesn’t say might about how much value or really anything else meaningful.
InsideOutSanta 1 days ago [-]
They're probably mentioning the size of the code base as an indicator for how difficult it is for an LLM to understand where and how to make changes.
1 days ago [-]
tw1984 1 days ago [-]
very interesting results!
GLM performed extremely well. we need GLM-6!
felixlu2026 23 hours ago [-]
[dead]
vegetablefinger 1 days ago [-]
[flagged]
appplication 1 days ago [-]
Welcome to the world, young robot
dirkc 1 days ago [-]
> as we aggressively adopt AI for engineering
Why do we need to aggressively adopt things rather than thoughtfully adopt things?
It sounds like they are probably punching AI and engineers in the process
simianwords 1 days ago [-]
What if other competing companies adopt it aggressively and push out features faster and get more market share?
What if pushing features faster brings more money because users like the features?
adhamsalama 1 days ago [-]
Did any company achieve this?
Have we seen a company fail because they're not adopting AI as much as their competitors?
https://lucumr.pocoo.org/2026/7/4/better-models-worse-tools/
It's worse than I expected.
So why would we expect all these bizarre bizantine language models to all conform to how a request is both made, expected and massaged.
For awhile, I was getting bizarre opencode tool errors where the only problem was the model was passing in a "1.0" or "0.0" where the harness dutifully wanted an integer. Of course 0.0 is the same as an integer in practical operations.
what do you mean by this ? do you rewrite the context in your proxy ?
Once the thing is rock-solid it's relatively easy to do a Swift->HTTP/HTML/CSS/React/TypeScript conversion.
I have experienced similar behavior between opus and haiku when benchmarking Dara engineering tasks. The “cheaper” model takes many more turns to figure out the task and this is without taking into account other important factors.
Another interesting behavior that I observed is that Haiku tended to cheat more maybe because it was having a harder time to find the root cause of the problem.
Benchmarking and evaluation of agentic systems is very interesting and if there’s one thing that someone should keep from the Databricks post is how important is for everyone to build and run their own.
pretty sure the only thing making that 'clear' is the coloured stripes, if you took that away it'd look like two tiers
good result for GLM 5.2 though
and Sonnet 5 seems like a waste of time
It's great to see a large-scale real-world benchmark from a user of these tools, as opposed to the the benchmaxxed results from the vendors themselves. Also great to see different harnesses being tested, with considerably different results.
Definitely a few surprises here:
1) GLM 5.2 using Pi performs identically in terms of pass rate (~87.5%) to Opus 4.8 high using Claude Code, but significantly cheaper ($1.25 per task vs $2)
2) Absolute best pass rate (90%) was from Opus 4.8 x-high using Pi, beating out Opus 4.8 using Claude Code
3) Pareto frontier performance from any of the models (Opus 4.8, GPT 5.5, GLM 2.5) was using Pi rather than native harnesses
Apparently Pi used 3x less context than Claude Code, and one takeaway is to use Pi regardless of what model you are using. The other takeaway is that in real-world performance GLM 5.2 is the equal of Opus 4.8 unless you run Opus 4.8 on x-high in which case you can eke out a 2.5% increase in pass rate at the expense of doubling your cost over GLM 5.2
There seems to be a lot of good buzz about GPT 5.6 on Twitter - people (incl. OpenCode team) preferring it to Fable 5.
But they are more vendor neutral, now they don't sell their own model. It's interesting from a benchmark point of view.
Other hanresses are doing overkill so they can work with any model.
Its shocking how cost per token does not correlate with cost per task, it's wild to see opus and glm nearby on $ per task axis
The combined size of codebases for the underlying opensource products (Apache Spark etc) might be around 1M lines, I think. Why does the orchestration/management layer, that is "databricks", exceed the sizes of the core products?
It's a good stress test for the LLM because it is not an "ideal" codebase.
Forget Databricks == Apache Spark...
Deleting code is difficult and almost never makes sense afaik
GLM performed extremely well. we need GLM-6!
Why do we need to aggressively adopt things rather than thoughtfully adopt things?
It sounds like they are probably punching AI and engineers in the process
What if pushing features faster brings more money because users like the features?
Have we seen a company fail because they're not adopting AI as much as their competitors?