Rendered at 15:58:40 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
throwaway219450 20 hours ago [-]
Astra writes well (which is why I often like to work with it over Claude), but it tends to produce code golf that's borderline unreadable, like completely omitting line breaks. I thought I had something funky with my config, but it seems to be a known issue.
EDIT: You can search for "tenobrus astra" on X, but link removed.
The generated code certainly does what's asked of it, but it's as sloppy as anything else I've seen. I don't think this benchmark can control for that. Maybe if it had a metric for minimum diff to satisfy the new requirement with the existing constraints on dead code etc?
I use the frontier models only in plan mode; coding is held out to cheaper models.
I run Paseo as an orchestrator with agent profiles. "Distinguished Member of Technical Staff" -> Fable, plan mode. "End-User Customer Advocate" -> Astra, plan mode. A few others. Basically, these agents (with profile prompts) can only be "consulted" for input on design elements; it's OK if they produce a bit of verbosity here. This has worked fantastically for me, really awesome to see a self-constructed team own the work end-to-end.
A "Director of Engineering" (6.1 Sol or Opus 5.5) also in plan mode organizes the team for the task and then owns coordination and instruction. You can think of this in essence like a "model router".
Coding is handled by Haiku 5.5/GPT 6 Luna/DeepSeek v4.1 Flash. One "Principal Engineer" Sonnet 5.5 or 6.1 Sol for some tasks.
grim_io 18 hours ago [-]
You are deep in it.
You may be larping for no additional benefit vs just using Opus 5.5 and just asking it to implement a feature.
ac29 1 hours ago [-]
Yeah at the point that Opus has read enough context to hand off a detailed plan, implementing the plan may not be that many more tokens.
Also having spent the weekend with the supposedly great Deepseek Flash 4.1 1017, I wonder if the people singing its praises have used recent frontier models. DS Flash feels like maybe Sonnet 4.5-4.6. Absolutely falls apart at things Opus/Sol currently excel at with low reasoning, and even has basic problems with failing tool calls, which hasnt been an issue with Anthropic/OpenAI in forever.
Could be a configuration issue with the provider I tried but honestly, I didnt get much better out of GLM5.3-flash. GLM5.3 and Kimi3 are much better but since most inference providers cant take a loss they are more expensive than just using Opus/Sol on subs, which are almost certainly currently a loss leader for their respective API offeringss.
14u2c 17 hours ago [-]
You are correct, at least based on my experimentation. This subagent workflow really only delivers meaningful results when working across model families (ex: Opus asking Astra for a review) or when working across conceptual hardware boundaries (cloud offloading to local inference, etc). The planner / overseer model ends up reading all the output of the coder model anyway, at least if you want quality output, so there's not much benefit otherwise.
CharlieDigital 16 hours ago [-]
> The planner / overseer model ends up reading all the output of the coder model anyway
I mean, even that's fine because output tokens cost ~5x input tokens. So that's still a win if the agent that does the heavy code reading and writing produces output that may be read again by the lead (not my observation of how agents in Paseo interact and of course, correctable to degrees with shared system prompt).
Here's a snippet of a sample prompt downwards from the lead for a piece of code I'm working on:
From agent: 3ff73f80-9713-458c-8271-d275f03609c6
<role>
You are the Principal Engineer on <snipped ... /> (.NET 11 / Vue MCP server; read CLAUDE.md in your working directory first). ...
</role>
<lead_role>
I am the Director of Engineering. The System Architect and the DMTS reviewed this approach ...
</lead_role>
<sniped ... />
<comms>
Run all gating commands in the FOREGROUND. End your turn ONLY with the final report: pass/fail per success criterion (a)-(i) with evidence, the branch tip SHA and `git log --oneline origin/main..HEAD`, lines added/deleted, the exact 0factor image digest and version you pinned, every deviation from this brief and why, risks you saw,...
</comms>
The agent is given explicit "comms" instructions upwards and I find that both agents in this exchange stick to the contract.
14u2c 16 hours ago [-]
> even that's fine because output tokens cost ~5x input tokens.
That's assuming you come in under a certain threshold of revisions, and excludes the tokens generated for all the exchanges. The later can be be significant, especially when there are significant issues to correct.
CharlieDigital 15 hours ago [-]
All I can say is give it a try. We can debate hypotheticals all day, but it costs nothing to try and see.
The Paseo harness/orchestrator is doing something right. I switched from TUI after trying it as a candidate for a non-TUI user.
It's not the same behavior as Claude starting a Claude sub-agent.
dhorthy 12 hours ago [-]
or someone could clone the SCB repo and make a fork that proves it one way or another
bob1029 9 hours ago [-]
> The planner / overseer model ends up reading all the output of the coder model anyway, at least if you want quality output, so there's not much benefit otherwise.
Welcome to why the RLM paper does not explore recursive depths beyond 1. The game of context telephone as you ascend the call stack quickly falls apart.
CharlieDigital 18 hours ago [-]
Nah; I can use my token budget across both providers this way, use tokens more sparingly with smaller models for heavy text in/out (why pay Opus prices to babysit a PR? run tests?), and you'd be surprised when the agents talk to each other and "role play" how much this improves things like the UX as well as technical design which are more subjective.
Distinguished Member of Technical Staff, for example, is asked to "expand the solution space" which I find has a strong effect on approaches considered (it doesn't know what was already considered on the main thread; only the domain and problem space).
This also helps with context growth since the sub-agents do the heavy lifting (e.g. search, read docs) and just provide the key bits back to the lead.
Like you, I was a single-model, single-TUI user up until three weeks ago when I picked up Paseo and it is the only tool that I've found that has significantly positively improved my throughput without sacrificing quality of product.
Is it "larping"? I ran an eval today on a contract ingest and extract agent I'm working on. Changing the <role> fragment of the prompt lifted the results by +7 points by simply changing the text to the effect of "You are a senior partner at a prestigious law firm that focuses on reading licensing contracts..." and some bits about the content the agent is reviewing + objective. No one would believe it without an eval.
Claude docs explicitly say to LARP[0]:
> Give Claude a role
>
> Setting a role in the system prompt focuses Claude's behavior and tone for your use case. Even a single sentence makes a difference
My conclusion: the agents like to LARP so I'm happy to LARP with them.
Why stop at "You are a senior partner at a prestigious law firm that focuses on reading licensing contracts..." why not "You are super programmer/developer that has been programming with Ada using punchcard, having Linus on quick dial, you can smell BS code from miles away and know exactly which question to ask".
CharlieDigital 5 hours ago [-]
Eval it and see.
You might be surprised.
(But also, this is more or less what I do with Paseo profiles)
elcritch 15 hours ago [-]
I hate to be pedantic, but LARP stands for live action role playing.
Only mentioning it because LLMs are already taking over software development and everything online. Let's keep them out of our meet space peculiarities (1) while we can. ;)
> I hate to be pedantic, but LARP stands for live action role playing.
> Only mentioning it because LLMs are already taking over software development and everything online.
In our glorious future, you'll have your agent LARP for you with your friends' agents, so you all can spend more time doing more fulfilling tasks like watching ads on Instagram.
disgruntledphd2 8 hours ago [-]
Yeah, they're clearly table top roleplayers.
The real question is are they white wolf or DnD?
watercolorblind 6 hours ago [-]
I'd say shadowrun.
what 17 hours ago [-]
He has to be deep in it because it’s the product he sells.
I have other products and I work for private equity (no affiliation with Paseo at all).
You've entirely missed the lede.
criley2 15 hours ago [-]
No "may be" about it. There is no benefit to having bad models write your code, and there's no benefit to "plan" mode. Plan mode is an outdated 2025 concept that should be removed from standard harnesses.
There are 2 scenarios for dumb builder agents:
1. Your "planner" actually implements the project almost completely, finds all the gotchas, solves all of the emergent problems, and fully plans the solution.
2. Your "planner" takes a half-assed first attempt, solving none of the major problems that will arise during implementation
In scenario 1 -- just pay cache output on a few more tokens to finish the project. No need to spend twenty minutes pumping a few million tokens through deepseek regardless of how cheap it is to do so.
In scenario 2 -- good luck getting quality output
I do think this "software team" LARPing is an engineering version of AI psychosis. You can get all of the little parts humming along, sending messages, and looking really cool. You can even convince yourself that all of these busy little bees actually make better product.
But a frontier model working alone can produce better output in a tiny fraction of the time. Just build the thing.
vintermann 6 hours ago [-]
An obvious benefit to plan mode is that your model tells you what it's planning to do before it does it.
criley2 5 hours ago [-]
An obvious downside to plan mode is that without access to the edit tool, the model can't confirm its hallucinations.
Removing edit tool from your model is like lobotomizing your engineer.
Here's the secret: Just building the thing is the modern "plan mode". You don't have to keep what it made. But a completed, verified, and tested output is always more valuable to my planning process than a hallucinated sloppy guess.
sausagefeet 1 hours ago [-]
My experience has been that building the thing is not modern "plan mode".
I do not use an explicit "plan mode" that tries to limit what the LLM can do. But I do tell it to explore and research and produce an artifact of what needs to happen to finish a large ticket and then use that as input to the actual implementation. I do have it even write small experiments to verify some-parts of the total design to connect it to reality.
The reason is that there can often be lots of decisions that need a large amount of context about the code base in order to reasonably determine what the right design and implementation is and this doesn't happen (in my experience) with the little Interview dialogue that pops up when it isn't sure. Or, even worse, Opus will just make decisions for you that are incoherent with the goals of the codebase.
So, I think, "plan mode" as an explicit mode is dead, but the act of planning is still a good thing to do.
FeepingCreature 3 hours ago [-]
In my experience Opus 5.5 is good but Opus very big picture plan mode to Qwen 3.8 code is so massively massively cheaper (I'm on API) that there's no competition. This is as of a week ago.
CharlieDigital 15 hours ago [-]
Try it. 3 weeks ago I was in the same boat. Single model, TUI. I only tried Paseo primarily to test multi-host orchestration (after giving up on Herdr) and not even multi-agent, multi-plat behavior.
Output quality, especially subjective bits, has improved relative to my attention needed. Autonomous decision making is much, much better for me adding one profile that specifically is instructed to focus on UI/UX, one for end user proxy, and one for MTS.
It costs you nothing to try except maybe 15 minutes of setup. Give it a solid, skeptical week. It's good even without the profiles, but way better with.
In your comparison, did you previously use a review with a different model family?
Because that step is imo crucial, and it's unclear to me whether additional benefits of the roles are worth it over the basic review.
I can see how roles that ask the "planner" to explore other approaches automatically could be useful.
However, I don't have any interest in using small or dumb models for implementation when the subscription allowance permits using Opus 5.5 everywhere.
CharlieDigital 5 hours ago [-]
Yes, have done cross plat plan review. But why it works so well with Paseo is that it's automatic and multi-turn (agents kind of "discuss" (I say "kind of" because agents don't really discuss like you or I would with short, multi-turn speak-listen-pivot-repeat).
Work tokens are severely limited to the extent that if I used Sonnet 5.5 for implementation, I'd be out 2h into a session (Team 5x). +$400/mo. at API rates on OpenAI. So Haiku + Luna it is. There is no other way this is even usable for agentic dev at work. Works fine for people who multi-turn with a single agent.
Personal accounts I run the $17 Anthropic plan + $100 Codex Pro + DeepSeek. It's insane how much this stretches with this setup.
Maybe the crux here is how hands on you are with your sessions. As I've moved more hands off, I'm relying on other agents to take my place by embedding my heuristics into their config.
user43928 5 hours ago [-]
It's automatic and multi-turn by default.
I've had an instruction to review the code with "claude -p" in place around the time GPT-5.5 released. I think it has served me well.
If I recall correctly it was before subagents were a thing. Even back then it was already automatic and would on its own request follow-up reviews or arbitrate a plan file if asked.
qvq 14 hours ago [-]
I tried it and figured out the benefit was getting the agent to look at solving the problem from different perspectives. Now I just include those perspectives in the prompt to a single agent.
Example: If I feel like the agent is going to just bolt on a feature without refactoring the core parts, I tell it to expand functionality of the core parts as needed.
5 hours ago [-]
rimliu 8 hours ago [-]
It is watching Linux boot sequence all over again.
8 hours ago [-]
eptcyka 20 hours ago [-]
Don’t apologize for x, just don’t post x links.
Aurornis 16 hours ago [-]
X links are fine. People who don't want to visit the site can choose not to visit it.
This current situation where people get shamed for sharing the information until they edit it out and leave obscure "search for___ on X" is infinitely worse.
encypruon 19 hours ago [-]
I think it's the best possible approach. It lets you refer to content that's unfortunately stuck on the site, while still signaling clearly that it's something to avoid if possible and to apologize for if not. Omitting the link just removes information without influencing consensus on whether to use X.
edit: I wish it would let me view the darn screenshot tho :c
uoaei 19 hours ago [-]
There are plenty of alternatives to link to X content without hitting X's servers.
flaburgan 18 hours ago [-]
Do you mind listing some? I only know xxcancel.com
Arubis 14 hours ago [-]
Twiiit.com/(your link) rewrites to a pool of nitter servers.
19 hours ago [-]
throwaway219450 18 hours ago [-]
I removed it, but it's frustrating when the options are someone's (reasonable) post on X, or slop-spam aggregators. I don't really want to drive visits to either, but at least the X poster can get the views. I don't really like xcancel either as it still encourages indirect engagement with X.
user43928 7 hours ago [-]
IMO just post the link, we are all professionals here.
Those who want the third-party aggregators can set up their browser to automatically use them. Others who are signed in on X can directly open.
pkulak 13 hours ago [-]
6.1 Sol does this too. Like, line break after a function, but absolutely nowhere else. This solved it for me:
- Use paragraph-oriented formatting: give distinct logical steps room to breathe. Keep closely related statements together, but separate changes of purpose and a block's final result with blank lines. Prefer readability over vertical compactness.
ModernMech 13 hours ago [-]
I've found passing its output through a formatter is essential, also turning dead code warnings into errors.
jeffy29 11 hours ago [-]
I fear OpenAI models are collapsing on themselves (idk if it's because of loop transformers or too much RL etc), which I didn't feel with 5.6 Sol at all. Need a 6.1 Sol nor Astra really feel like full blown legitimate models. More like autistic genius individuals who are partially brain dead.
Which can sometimes work incredibly well when working in conjunction with another model. It's unbelievable how incredible gpt-6.1-sol even on medium reasoning is at reviewing Opus 5.5's work it will find incredibly weird, but legitimate, edge cases, yet if you actually use it as the main model, it just misses forest for trees so much. They are more like an organ which can do an incredible job at a specialized task that the brain cannot handle efficiently but the second you try to do brain's work with time, it all comes crashing down.
I hope they are able to recover with future models. If you could could somehow combine how Opus approaches a task with the autistic brilliance of Astra/6.1, you would have an incredible model far exceeding today's models.
ttul 19 hours ago [-]
I've noticed the line breaks issue. It's easily fixed with a formatter like prettier and, frankly, that kind of quality-of-life thing is best handled deterministically anyway.
LukaD 18 hours ago [-]
I haven't used prettier much but this is what the prettier docs say about empty lines.
> It turns out that empty lines are very hard to automatically generate. The approach that Prettier takes is to preserve empty lines the way they were in the original source code. There are two additional rules:
>
> - Prettier collapses multiple blank lines into a single blank line.
> - Empty lines at the start and end of blocks (and whole files) are removed. (Files always end with a single newline, though.)
The last time I used astra it wrote a lot of rust code where it would insert almost no empty lines at all. Not before functions, structs, impl blocks, etc., and not inside functions to group statements. Similar to prettier, rustfmt doesn't really insert empty lines.
Edit: hmm maybe I misunderstood your and the parent's comment. Are you talking about missing regular newlines after statements, i.e. multiple statements on one line?
throwaway219450 16 hours ago [-]
I was talking about a function with 100 lines and no line breaks. Very difficult to read.
I use black/ruff on a lot of projects and it will only touch standard-y things like docstring spacing, or spacing between methods and classes, import sorting etc. It doesn’t touch intentional/aesthetic user spacing. With agents it is helpful to make them run ruff and mypy to keep them honest, but as long as the code adheres to the spec, it doesn’t get changed. Astra also creates things like “report” functions with 10 arguments, so it’s useful to turn on complexity checkers as well.
I don’t see how Prettier is any better here unless you really want everything to be deterministic; how could it possibly know (always) when to insert a line? Aside from some more obvious cases like adding a gap before a loop declaration. Sometimes I group lines because they’re part of a mathematical formula or other operation and it makes sense to me, but Prettier doesn’t know that.
ch_sm 6 hours ago [-]
> a function with 100 lines
there‘s your problem right there
rsfern 18 hours ago [-]
Doesn’t this mess up the context though? I use ruff and my agents get confused if I edit and run ruff out of band. How do people deal with that? I’m always having to remind agents to re-read files because they’ve changed
ttul 15 hours ago [-]
I find it is okay if I tell the agent that running the formatting step is required before every commit.
loveparade 14 hours ago [-]
Astra is some of the worst code I've seen in terms of slop, so the benchmark is kind of surprising to me. Astra gets things right, but I need Claude clean up its slop all the time. Many times I've cut down Astra changes from 1000+ lines to ~100 lines.
cultofmetatron 9 hours ago [-]
astra is great for exploration and planning. once I have a plan, I usually delegate to deepseek flash. its quick and doesn't stray
bartnp 6 hours ago [-]
Funny, I've been having the opposite nitpicky issue with astra and sol, where
Beware NSFW: redirects to private instances that redirect to porn for monetization
Edit: disable JS to prevent click-jacking
nozzlegear 16 hours ago [-]
Interesting, I've never seen it do that.
bocklund 2 hours ago [-]
For Opus, I don't really buy that xhigh is worse that medium. When scores are ~15%, it's unclear how much is just in the noise. The only things in favor of medium are the score and price: strict goes from (xhigh -> medium) 25 -> 28 and core goes 141 -> 145 out of 196. xhigh is better on strict, opus medium has _by far_ the most leftover code, callables, and not poor complexity of those callables.
If cost and score are the only things that matter, maybe medium is slightly better for a software factory. But I'd be curious if there were tasks in this benchmark that had twice as many checkpoints if medium's excess code would make it compare worse with xhigh.
taspeotis 17 hours ago [-]
It's Opus specific, but personally I'd never bother with Opus above High, CursorBench shows a clear inflection: https://cursor.com/cursorbench
CodesInChaos 7 hours ago [-]
Matches my subjective experience where Opus Extra likes starting sub-agents that burn tokens for little benefit. While High just does the work itself.
WASDx 21 hours ago [-]
Great benchmark!
I believe effort can mean both "solve harder problems" and "produce more stuff". I mostly don't want more stuff than I asked for, so a capable model at lower effort can indeed be the wiser choice. Faster response too.
osytro 7 hours ago [-]
[flagged]
dhorthy 16 hours ago [-]
yep agree
krzyk 3 hours ago [-]
Why they didn't test on high? (and also other levels for that matter, but high looks like the most obvious one because xhigh is to costly and doesn't give much benefit)
hedgehog 21 hours ago [-]
Is there a reasonable way to run a representative subset myself?
Edit:
> For folks who want to try to repro these findings: I used the pinned v1.0 catalog, the stock just-solve prompt, seed 42, the any-case runner pass policy, and Docker with Python 3.12 and uv.
Yeah and I started with subsets although I did not find them representative, eg I picked 6 challenges for a subset and fable scored 33%, while on the full challenge it landed closer to 14-15% range
The part II post linked in my top comment has a list of the challenges I used for that.
To run them, just point your model at the paper, it can find the repo, etc and with a key or sub, get it running
hedgehog 13 hours ago [-]
Thanks, that's helpful. Do you still have the full data of pass/fail for each test and model? It would be interesting retrospectively to see if there is such a subset.
I built some tools for personal research projects that could potentially help success rates on long dev tasks, empirically it seems to but I've never actually done a quantitative evaluation. SlopCodeBench looks promising for all the reasons you point out in your post.
softwaredoug 18 hours ago [-]
IMO slop usually comes from overly defensive programming from LLMs. They’re loath to remove code and simplify, because unless we humans tell them some code / use case is unimportant, they have no reason to not care about it.
So agents can make these overly complex solutions trying to make bulletproof implementations and prevent regressions.
It requires a human to look at code, ask about complexity, and guide simplification by subtracting requirements implicit in the code.
skrtskrt 18 hours ago [-]
They also seem to really bias towards tacking on new code to "work around" what's there, rather than breaking down what's there into smaller more composable bits that can serve the pre-existing use case and the new requirement elegantly.
Klathmon 18 hours ago [-]
LLMs write. It sounds glib but it's what they do, and so every problem they encounter they will try to write their way out of it without lots of hand holding.
melvinroest 9 hours ago [-]
Yea I say something similar. LLMs have a tendency to add, because there are many more tokens than a stop token. They can remove, they can edit, but their bias is to add. My suspicion is that training them when to stop when it is best to stop is harder than training them to make a plausible sounding story.
dhorthy 16 hours ago [-]
they yearn to create content, generate more tokens, etc etc - agree
ares623 18 hours ago [-]
It's like we took the worst behavior of our colleagues, and went and said "yes, let's automate _that_ part" and patted ourselves on the back.
kansface 17 hours ago [-]
It’s more likely emergent behavior.
softwaredoug 18 hours ago [-]
Yes exactly. And I’m not sure we want agents with that level of autonomy to decide what functionality is important and what to leave in place.
yxhuvud 16 hours ago [-]
No, but it would sometimes be nice if they sometimes raised questions about what is important and what isn't
1 hours ago [-]
furyofantares 21 hours ago [-]
For most things I find medium to be the highest quality output. However if I want it to do spatial reasoning or if I want it be funny / write something (not code) of almost-passable quality, then xhigh is where it's at.
drivebyhooting 20 hours ago [-]
First of all, what happened to Max and Ultra?
Secondly, I’m mostly using Astra Ultra or Sol 6.1 when tight on quota. I pay $200/month and rarely if ever run out. Am I doing something wrong? Or perhaps I’m not that productive as you guys?
steinvakt2 20 hours ago [-]
Same here. I’m feeling productive enough. If I was more “productive”, I wouldn’t be able to keep up with what I was producing. My attention and cognitive broadband is the bottleneck. I find it hard to believe that people are actually productive using 4 active chat windows simultaneously for 8+ hours a day.
trueno 9 hours ago [-]
> If I was more “productive”, I wouldn’t be able to keep up with what I was producing. My attention and cognitive broadband is the bottleneck.
this is so damn important. none of this shit matters at all if your headspace is destroyed.
I have a coworker who is not yielding to these feelings and i swear to god he just isn't the same person anymore and i legitimately worry about the guy. He's one of those that saw AI as an opportunity to do the "massive, grand no-finish-line" type projects: super DUPER knowledge bases/wikis or for his personal projects some super grand encyclopedia of his family history and profiles on each person and it's just like exhausting to hear about, it's even more exhausting to see how exhausted he looks all the time. The only thing that gives him dopamine is stuff he's done with AI, and I have really tried a few times now to say hey man this stuff is.. addicting. And with all things addicting, moderation is key. He's seriously anxious all the time now.
I just took a 1.5 month break from work, and in that time I haven't touched Claude. It's been great I can't lie. I'm so much happier and ironically busier, away from my computer. Feels like I got a nice dose of life and I'm reminded what's important and that really isn't omega-maxing my productivity.
This tech can help ya do more with less effort, sure. But everyone's kneejerk reaction is to fill it with doing bigger, grander, and .. more. Seems a lot of people have lost sight of how much should one human brain feasibly manage, even if it's simplified through prompting there is assuredly a mental toll with the amount of things and the size of things people are now taking on.
kooi 12 hours ago [-]
Yep, the context switching let's me do 2 or 3 active agents, but I have to slow down if real issues come up.
Maybe the agent maxing dudes are doing their 12+ agents on a monster vibe project and all they say is "fix it", "make it work" all day
killingtime74 5 hours ago [-]
Ultra is not a thinking level I understand. Max is the highest. Ultra gets it to spawn more subagents.
4b11b4 18 hours ago [-]
Same I just leave 6.1 on Max. Don't touch that dial
slopinthebag 17 hours ago [-]
max and ultra are so slow and tend to overengineer massively imo
agentdev001 15 hours ago [-]
Max and ultra are slow and overengineer, based on evidence from evaluations.
giancarlostoro 19 hours ago [-]
GPT 5.5 on Medium is all I use lately, I don't understand why anyone would pay drastically more for slightly more, when Opus has been good enough for a year now to do most things.
AnotherGoodName 18 hours ago [-]
You unknowingly substituted the name of another model when you’re otherwise talking about the competitor fwiw. Some if those in the reply chain didn’t even seem to notice.
That’s a little telling of the state of ai atm though isn’t it? It’s all pretty similar. The latest Opus/GPT/Gemini/Deepseek/Kimi on medium does indeed do the job
spijdar 17 hours ago [-]
It would be a little strange but not that strange for someone to have an idiosyncratic preference for GPT-5.5. It's still available as a Codex model (I think? It's selectable in Pi, not Codex itself), and it might perform differently enough from later GPT models that someone prefers it.
To your point, though, I don't know if I've noticed that big of an impact going from 5.5 to newer GPT models. If I had to switch back, I'm not sure how much I'd miss the newer ones. People talk about e.g. Opus 4.6 similarly, I've noticed.
The bigger change in OpenAI-land IMO is the models got cheaper, generally.
giancarlostoro 16 hours ago [-]
You're right, I meant to write: Opus for all of the above. I do think a lot of it is roughly similar, for most use cases, anything after Opus 4.5+ is "good enough" basically.
spijdar 19 hours ago [-]
Why not GPT-{5.6, 6, 6.1} Sol? They're all cheaper than GPT 5.5, and as "smart", if not "smarter", at least to me.
Though, it has been a while since I've used 5.5.
ttul 19 hours ago [-]
Surely they mean Opus-5.5...
giancarlostoro 16 hours ago [-]
Yeah, whoops.
oh_no 11 hours ago [-]
sol 6.1 is wildly cheaper, one reason to use something else
singingtoday 19 hours ago [-]
I have no data to support this, but when working on SaaS systems I find anthropic models more effective.
But for webdev, deep research, code review, and a few other tasks I find the current 6.1 sol better.
But I use both. And I use codex more.
theodorewiles 19 hours ago [-]
I have a simplify skill built from a rich hickey talk that basically solves the erosion and verbosity metrics SCB flagged (i.e. erosion and verbosity are way better and improve over time). However, after running the SCB benchmark I cannot find any evidence that it actually improves test pass rate trajectories. Food for thought on maintainability metrics.
sheepscreek 14 hours ago [-]
Suggestion to include GPT 6.1 Sol (Low) since that is configured as the default. I’d be interested to see how well it fares against the others.
mordae 19 hours ago [-]
This is interesting. Is nobody training LLMs to repeat resolving issues by patches from real projects where refactoring is required, judging the refactor similarity and not just tests passing?
jwpapi 17 hours ago [-]
Very interesting. If I’m honest with myself it could be very much a reality that I would be better off in only using for medium effor size tasked instead of trying to "offload" the necessary architecture thinking to "high" or even more.
They are good for inspiration though, but maybe I rather let me build a demo and then tell a medium model what I want from it.
xlayn 17 hours ago [-]
what are the odds of as purely side effect of benchmaxing that the models get actually good because the benchmarkers the nittier they get the better the model has to become?
corrolary: just as cars evolved for worst because they were measured for "race numbers", what are the odds that the models just become "that uber nerd that degrades anyone who doesn't follow all his uber-rules for uber-code" and in the process start creating code that cannot be read or maintained?
Wouldn't it be better to enforce pass via the harness and then measure cost/quality since that's more what people are likely to do?
meffmadd 19 hours ago [-]
Are the raw outputs published somewhere?
I am currently investigating SCB results and a larger, more representative dataset to work on would be really helpful!
18 hours ago [-]
snvzz 7 hours ago [-]
Medium vs xhigh is interesting, but I can't help but feel annoyed about the highlighting of medium vs xhigh alongside the omission of high and max.
jdthedisciple 19 hours ago [-]
Finally a serious benchmark!
visiondude 14 hours ago [-]
I can’t find this useful given they are trying to quantify wether the code ‘feels’ sloppy with some fuzzy math that can easily be manipulated. at least it’s an attempt, some feel it’s a good attempt, i feel it’s rather poor and misses the point. the best code is the one that's most easily understood and equally performant - some of the math here may go against that (given you can reduce
concentrations with helper functions and the like)
juleiie 19 hours ago [-]
Opus did better on medium? Can anyone confirm this somehow? What’s the sweetspot in your opinion?
I find QFN on xhigh usually overruns the context window for no good result. If it can do the task, it'll do it quicker on medium. If it can't do the task, xhigh won't change that.
rihegher 5 hours ago [-]
it looks like the post has been removed here a blog post with the results deepu.tech/qwen3.8-flash-next-vs-claude-opus-showdown/
I wish a company named human layer won't write their reports with so much Claudisms.
21 hours ago [-]
0c3ca83 21 hours ago [-]
[flagged]
Onavo 20 hours ago [-]
I think people downvoting you missed the sarcasm.
perching_aix 20 hours ago [-]
On the contrary.
thatslast 14 hours ago [-]
wondering how gemini 4 performs
0xbadcafebee 14 hours ago [-]
> The full benchmark is hard in a way that matters to me: the agents get the main behavior right much more often than they keep all tested behavior working as requirements change
So, it's like... software development? 90% of bugs come from unintended consequences of changes after the fact. There's a couple decades of design lessons people had to learn to keep this number low. The agents don't follow those lessons.
Good programmers rely on structure and reinforcement to prevent themselves from introducing bugs. Today that structure is a combination of software frameworks, SDLC gates, and system design patterns. For example, most any Python programmer can reliably churn out a Django app as a Microservice and be fairly sure it won't be that hard to make, or result in tons of unexpected bugs. The reason isn't that it's some perfect software design. It's that prescribed methods in themselves are bumper lanes, keeping you on track. A few of you may remember what programming was like before popular open source frameworks, or maybe you've worked on proprietary big-iron mega-projects. Left entirely to one's own devices, programmers can make a mess of things. Give them a formal structure and suddenly there's less to go wrong.
I wouldn't sit around waiting for models to become uber-programmers. Instead I'd treat 'em like humans, and enforce the structures we know have better results. You can try to impose good design as skills (https://github.com/ciembor/agent-rules-books) but the model will still do things its own way. That's why coding harnesses need to intervene and explicitly enforce the structure/framework/bumper lanes. Basically we need 1,000 different Pi extensions that will intervene as the agent codes in order to enforce the correct design. Otherwise we will forever be chasing some "perfect coding rlhf", and still the model will blunder into corners.
slopinthebag 17 hours ago [-]
not suprising. i think we also tend to do better at things when we only expend medium effort.
cute_boi 20 hours ago [-]
The irony is the article is very sloppy. Can we at least write it and review it properly..
19 hours ago [-]
jampekka 20 hours ago [-]
What's wrong with it? Reads fine to me.
a3w 19 hours ago [-]
There are 36 tests. You can pass 0 or 36 or any number inbetween. Lets give five significant digits of how accurate solving was.
That is not how data physics work. It should not be how data science works.
kirBurkhanov 6 hours ago [-]
Yeah, and the noise is also per model, run the same model five or ten times and put the confidence around it. Then try different models and different workflows. Thats actually how I do it when I build workflows and automations: test, test, and test, collect the statistic on where and what works better, then combine.
antipaul 20 hours ago [-]
This is true. I was unfamiliar with this concept (I use codex/claude daily), and am trying to get to the "bottom line".
Oh, it's there at bottom under "my takeaway". But I still didn't get it initially.
https://lucumr.pocoo.org/2026/9/7/astra-why/
EDIT: You can search for "tenobrus astra" on X, but link removed.
The generated code certainly does what's asked of it, but it's as sloppy as anything else I've seen. I don't think this benchmark can control for that. Maybe if it had a metric for minimum diff to satisfy the new requirement with the existing constraints on dead code etc?
I run Paseo as an orchestrator with agent profiles. "Distinguished Member of Technical Staff" -> Fable, plan mode. "End-User Customer Advocate" -> Astra, plan mode. A few others. Basically, these agents (with profile prompts) can only be "consulted" for input on design elements; it's OK if they produce a bit of verbosity here. This has worked fantastically for me, really awesome to see a self-constructed team own the work end-to-end.
A "Director of Engineering" (6.1 Sol or Opus 5.5) also in plan mode organizes the team for the task and then owns coordination and instruction. You can think of this in essence like a "model router".
Coding is handled by Haiku 5.5/GPT 6 Luna/DeepSeek v4.1 Flash. One "Principal Engineer" Sonnet 5.5 or 6.1 Sol for some tasks.
You may be larping for no additional benefit vs just using Opus 5.5 and just asking it to implement a feature.
Also having spent the weekend with the supposedly great Deepseek Flash 4.1 1017, I wonder if the people singing its praises have used recent frontier models. DS Flash feels like maybe Sonnet 4.5-4.6. Absolutely falls apart at things Opus/Sol currently excel at with low reasoning, and even has basic problems with failing tool calls, which hasnt been an issue with Anthropic/OpenAI in forever.
Could be a configuration issue with the provider I tried but honestly, I didnt get much better out of GLM5.3-flash. GLM5.3 and Kimi3 are much better but since most inference providers cant take a loss they are more expensive than just using Opus/Sol on subs, which are almost certainly currently a loss leader for their respective API offeringss.
Here's a snippet of a sample prompt downwards from the lead for a piece of code I'm working on:
The agent is given explicit "comms" instructions upwards and I find that both agents in this exchange stick to the contract.That's assuming you come in under a certain threshold of revisions, and excludes the tokens generated for all the exchanges. The later can be be significant, especially when there are significant issues to correct.
The Paseo harness/orchestrator is doing something right. I switched from TUI after trying it as a candidate for a non-TUI user.
It's not the same behavior as Claude starting a Claude sub-agent.
Welcome to why the RLM paper does not explore recursive depths beyond 1. The game of context telephone as you ascend the call stack quickly falls apart.
Distinguished Member of Technical Staff, for example, is asked to "expand the solution space" which I find has a strong effect on approaches considered (it doesn't know what was already considered on the main thread; only the domain and problem space).
This also helps with context growth since the sub-agents do the heavy lifting (e.g. search, read docs) and just provide the key bits back to the lead.
Like you, I was a single-model, single-TUI user up until three weeks ago when I picked up Paseo and it is the only tool that I've found that has significantly positively improved my throughput without sacrificing quality of product.
Is it "larping"? I ran an eval today on a contract ingest and extract agent I'm working on. Changing the <role> fragment of the prompt lifted the results by +7 points by simply changing the text to the effect of "You are a senior partner at a prestigious law firm that focuses on reading licensing contracts..." and some bits about the content the agent is reviewing + objective. No one would believe it without an eval.
Claude docs explicitly say to LARP[0]:
My conclusion: the agents like to LARP so I'm happy to LARP with them.[0] https://platform.claude.com/docs/en/build-with-claude/prompt...
You might be surprised.
(But also, this is more or less what I do with Paseo profiles)
Only mentioning it because LLMs are already taking over software development and everything online. Let's keep them out of our meet space peculiarities (1) while we can. ;)
1: https://epicarmoury.com/blogs/news/what-is-larp
> Only mentioning it because LLMs are already taking over software development and everything online.
In our glorious future, you'll have your agent LARP for you with your friends' agents, so you all can spend more time doing more fulfilling tasks like watching ads on Instagram.
The real question is are they white wolf or DnD?
I have other products and I work for private equity (no affiliation with Paseo at all).
You've entirely missed the lede.
There are 2 scenarios for dumb builder agents:
1. Your "planner" actually implements the project almost completely, finds all the gotchas, solves all of the emergent problems, and fully plans the solution.
2. Your "planner" takes a half-assed first attempt, solving none of the major problems that will arise during implementation
In scenario 1 -- just pay cache output on a few more tokens to finish the project. No need to spend twenty minutes pumping a few million tokens through deepseek regardless of how cheap it is to do so.
In scenario 2 -- good luck getting quality output
I do think this "software team" LARPing is an engineering version of AI psychosis. You can get all of the little parts humming along, sending messages, and looking really cool. You can even convince yourself that all of these busy little bees actually make better product.
But a frontier model working alone can produce better output in a tiny fraction of the time. Just build the thing.
Removing edit tool from your model is like lobotomizing your engineer.
Here's the secret: Just building the thing is the modern "plan mode". You don't have to keep what it made. But a completed, verified, and tested output is always more valuable to my planning process than a hallucinated sloppy guess.
I do not use an explicit "plan mode" that tries to limit what the LLM can do. But I do tell it to explore and research and produce an artifact of what needs to happen to finish a large ticket and then use that as input to the actual implementation. I do have it even write small experiments to verify some-parts of the total design to connect it to reality.
The reason is that there can often be lots of decisions that need a large amount of context about the code base in order to reasonably determine what the right design and implementation is and this doesn't happen (in my experience) with the little Interview dialogue that pops up when it isn't sure. Or, even worse, Opus will just make decisions for you that are incoherent with the goals of the codebase.
So, I think, "plan mode" as an explicit mode is dead, but the act of planning is still a good thing to do.
Output quality, especially subjective bits, has improved relative to my attention needed. Autonomous decision making is much, much better for me adding one profile that specifically is instructed to focus on UI/UX, one for end user proxy, and one for MTS.
It costs you nothing to try except maybe 15 minutes of setup. Give it a solid, skeptical week. It's good even without the profiles, but way better with.
Try my profiles as a starting point: https://gist.github.com/CharlieDigital/1c496a2799862795f0c31...
Because that step is imo crucial, and it's unclear to me whether additional benefits of the roles are worth it over the basic review.
I can see how roles that ask the "planner" to explore other approaches automatically could be useful.
However, I don't have any interest in using small or dumb models for implementation when the subscription allowance permits using Opus 5.5 everywhere.
Work tokens are severely limited to the extent that if I used Sonnet 5.5 for implementation, I'd be out 2h into a session (Team 5x). +$400/mo. at API rates on OpenAI. So Haiku + Luna it is. There is no other way this is even usable for agentic dev at work. Works fine for people who multi-turn with a single agent.
Personal accounts I run the $17 Anthropic plan + $100 Codex Pro + DeepSeek. It's insane how much this stretches with this setup.
Maybe the crux here is how hands on you are with your sessions. As I've moved more hands off, I'm relying on other agents to take my place by embedding my heuristics into their config.
I've had an instruction to review the code with "claude -p" in place around the time GPT-5.5 released. I think it has served me well.
If I recall correctly it was before subagents were a thing. Even back then it was already automatic and would on its own request follow-up reviews or arbitrate a plan file if asked.
This current situation where people get shamed for sharing the information until they edit it out and leave obscure "search for___ on X" is infinitely worse.
edit: I wish it would let me view the darn screenshot tho :c
Those who want the third-party aggregators can set up their browser to automatically use them. Others who are signed in on X can directly open.
- Use paragraph-oriented formatting: give distinct logical steps room to breathe. Keep closely related statements together, but separate changes of purpose and a block's final result with blank lines. Prefer readability over vertical compactness.
Which can sometimes work incredibly well when working in conjunction with another model. It's unbelievable how incredible gpt-6.1-sol even on medium reasoning is at reviewing Opus 5.5's work it will find incredibly weird, but legitimate, edge cases, yet if you actually use it as the main model, it just misses forest for trees so much. They are more like an organ which can do an incredible job at a specialized task that the brain cannot handle efficiently but the second you try to do brain's work with time, it all comes crashing down.
I hope they are able to recover with future models. If you could could somehow combine how Opus approaches a task with the autistic brilliance of Astra/6.1, you would have an incredible model far exceeding today's models.
> It turns out that empty lines are very hard to automatically generate. The approach that Prettier takes is to preserve empty lines the way they were in the original source code. There are two additional rules: > > - Prettier collapses multiple blank lines into a single blank line. > - Empty lines at the start and end of blocks (and whole files) are removed. (Files always end with a single newline, though.)
The last time I used astra it wrote a lot of rust code where it would insert almost no empty lines at all. Not before functions, structs, impl blocks, etc., and not inside functions to group statements. Similar to prettier, rustfmt doesn't really insert empty lines.
Edit: hmm maybe I misunderstood your and the parent's comment. Are you talking about missing regular newlines after statements, i.e. multiple statements on one line?
I use black/ruff on a lot of projects and it will only touch standard-y things like docstring spacing, or spacing between methods and classes, import sorting etc. It doesn’t touch intentional/aesthetic user spacing. With agents it is helpful to make them run ruff and mypy to keep them honest, but as long as the code adheres to the spec, it doesn’t get changed. Astra also creates things like “report” functions with 10 arguments, so it’s useful to turn on complexity checkers as well.
I don’t see how Prettier is any better here unless you really want everything to be deterministic; how could it possibly know (always) when to insert a line? Aside from some more obvious cases like adding a gap before a loop declaration. Sometimes I group lines because they’re part of a mathematical formula or other operation and it makes sense to me, but Prettier doesn’t know that.
there‘s your problem right there
they
constantly put
linebreaks
on
trivial things
in c#
like
string foo(
string bar){
throw new Exception(
"TODO");
}
https://xxcancel.com/tenobrus/status/2096636223414808719
Edit: disable JS to prevent click-jacking
If cost and score are the only things that matter, maybe medium is slightly better for a software factory. But I'd be curious if there were tasks in this benchmark that had twice as many checkpoints if medium's excess code would make it compare worse with xhigh.
I believe effort can mean both "solve harder problems" and "produce more stuff". I mostly don't want more stuff than I asked for, so a capable model at lower effort can indeed be the wiser choice. Faster response too.
Edit:
> For folks who want to try to repro these findings: I used the pinned v1.0 catalog, the stock just-solve prompt, seed 42, the any-case runner pass policy, and Docker with Python 3.12 and uv.
And https://github.com/gabeorlanski/scb-problems/
The part II post linked in my top comment has a list of the challenges I used for that.
To run them, just point your model at the paper, it can find the repo, etc and with a key or sub, get it running
I built some tools for personal research projects that could potentially help success rates on long dev tasks, empirically it seems to but I've never actually done a quantitative evaluation. SlopCodeBench looks promising for all the reasons you point out in your post.
So agents can make these overly complex solutions trying to make bulletproof implementations and prevent regressions.
It requires a human to look at code, ask about complexity, and guide simplification by subtracting requirements implicit in the code.
Secondly, I’m mostly using Astra Ultra or Sol 6.1 when tight on quota. I pay $200/month and rarely if ever run out. Am I doing something wrong? Or perhaps I’m not that productive as you guys?
this is so damn important. none of this shit matters at all if your headspace is destroyed.
I have a coworker who is not yielding to these feelings and i swear to god he just isn't the same person anymore and i legitimately worry about the guy. He's one of those that saw AI as an opportunity to do the "massive, grand no-finish-line" type projects: super DUPER knowledge bases/wikis or for his personal projects some super grand encyclopedia of his family history and profiles on each person and it's just like exhausting to hear about, it's even more exhausting to see how exhausted he looks all the time. The only thing that gives him dopamine is stuff he's done with AI, and I have really tried a few times now to say hey man this stuff is.. addicting. And with all things addicting, moderation is key. He's seriously anxious all the time now.
I just took a 1.5 month break from work, and in that time I haven't touched Claude. It's been great I can't lie. I'm so much happier and ironically busier, away from my computer. Feels like I got a nice dose of life and I'm reminded what's important and that really isn't omega-maxing my productivity.
This tech can help ya do more with less effort, sure. But everyone's kneejerk reaction is to fill it with doing bigger, grander, and .. more. Seems a lot of people have lost sight of how much should one human brain feasibly manage, even if it's simplified through prompting there is assuredly a mental toll with the amount of things and the size of things people are now taking on.
Maybe the agent maxing dudes are doing their 12+ agents on a monster vibe project and all they say is "fix it", "make it work" all day
That’s a little telling of the state of ai atm though isn’t it? It’s all pretty similar. The latest Opus/GPT/Gemini/Deepseek/Kimi on medium does indeed do the job
To your point, though, I don't know if I've noticed that big of an impact going from 5.5 to newer GPT models. If I had to switch back, I'm not sure how much I'd miss the newer ones. People talk about e.g. Opus 4.6 similarly, I've noticed.
The bigger change in OpenAI-land IMO is the models got cheaper, generally.
Though, it has been a while since I've used 5.5.
But for webdev, deep research, code review, and a few other tasks I find the current 6.1 sol better.
But I use both. And I use codex more.
They are good for inspiration though, but maybe I rather let me build a demo and then tell a medium model what I want from it.
corrolary: just as cars evolved for worst because they were measured for "race numbers", what are the odds that the models just become "that uber nerd that degrades anyone who doesn't follow all his uber-rules for uber-code" and in the process start creating code that cannot be read or maintained?
Benchmarking Opus 5 on SlopCodeBench - https://news.ycombinator.com/item?id=49076391 - July 2026 (119 comments)
I am currently investigating SCB results and a larger, more representative dataset to work on would be really helpful!
Recent evaluation result.
I find QFN on xhigh usually overruns the context window for no good result. If it can do the task, it'll do it quicker on medium. If it can't do the task, xhigh won't change that.
So, it's like... software development? 90% of bugs come from unintended consequences of changes after the fact. There's a couple decades of design lessons people had to learn to keep this number low. The agents don't follow those lessons.
Good programmers rely on structure and reinforcement to prevent themselves from introducing bugs. Today that structure is a combination of software frameworks, SDLC gates, and system design patterns. For example, most any Python programmer can reliably churn out a Django app as a Microservice and be fairly sure it won't be that hard to make, or result in tons of unexpected bugs. The reason isn't that it's some perfect software design. It's that prescribed methods in themselves are bumper lanes, keeping you on track. A few of you may remember what programming was like before popular open source frameworks, or maybe you've worked on proprietary big-iron mega-projects. Left entirely to one's own devices, programmers can make a mess of things. Give them a formal structure and suddenly there's less to go wrong.
I wouldn't sit around waiting for models to become uber-programmers. Instead I'd treat 'em like humans, and enforce the structures we know have better results. You can try to impose good design as skills (https://github.com/ciembor/agent-rules-books) but the model will still do things its own way. That's why coding harnesses need to intervene and explicitly enforce the structure/framework/bumper lanes. Basically we need 1,000 different Pi extensions that will intervene as the agent codes in order to enforce the correct design. Otherwise we will forever be chasing some "perfect coding rlhf", and still the model will blunder into corners.
That is not how data physics work. It should not be how data science works.
Oh, it's there at bottom under "my takeaway". But I still didn't get it initially.
Slop Code Bench? Need a Slop Writing Bench too