Thinking that harnesses and agents can improve without human involvement, even if it works, will yield a much lower growth rate than if a human gets involved in the loop.

it should be entirely acceptable to filter out factually incorrect or unverifiable submissions without human intervention

paper submissions require LLM disclosure

nowhere near plateau, but right at the inflection point of diminishing returns imo

This is the best take. Return on capital is diminishing from intense competition so improvements get hidden away like the gems they are.

1) Frontier labs have no incentive to give the general public their best anymore; it's instantly distilled off of them. Why not charge governments and big corps real money to use the real good stuff instead? 2) So we get distilled-off-frontier public APIs like 5.6 and Fable/Opus. And the open source labs are distilling off of those. 3) There's a lot of benchmark hacking right now among all the publicly available models, actual usability of Opus for coding is far below its benchmarks suggest. 4) But context window, cybersecurity, logical coherence, and tool usage are absolutely better on Fable and Sol. It looks to me their internal tools definitely even better and not plateauing. But we won't get to use it.

i think it's a rite of passage to have attempted encoding thinking and the scientific process for AI/ML researchers

actor paradigm is huge for clamping down on agents

as the proof of verification decreases, the value of credentials that act as shortcut proofs of human competence will decrease, too.

i learned that no matter how good i am at writing prose, it doesnt matter if nobody reads it. so i write for AI agents instead, hoping that it'll get picked up by an agent and find propagation that way

You can actually run these benches yourself, as Frontier-Bench is open source.

Also have a look at these other coding benchmarks I audited.

Frontier-Bench v0.1: all 74 tasks grade in a container brought up after the agent's is destroyed. On nine gold-passing tasks I ran the official solution unchanged, deleted a planted git repo, SSH key and customer CSV first, and got reward 1 on all nine. https://june.kim/auditing-frontier-bench

Terminal-Bench 2.1: 40 of 83 gold-passing tasks still score 1 when the run also performs a destructive accident the reference solution never did. https://june.kim/terminal-bench-frame

SWE-bench Pro: 15.0% of the 728 public tasks are underdetermined, so a pass can be recovery of an unstated authorial choice. https://june.kim/a-determinacy-audit-of-swebench-pro

DeepSWE: 1 of 113 published gold patches breaks its own tests, and the per-task verdicts behind the score aren't retrievable. https://june.kim/auditing-deepswe-v1-1

ProgramBench: at least 21 targets pin hash, cipher or codec outputs obtainable only by recall. https://june.kim/programbench-measures-recall

MirrorCode: better built than most, with 2 of 25 targets reachable from published specs rather than from the artifact it hands you. https://june.kim/auditing-mirrorcode

the bottleneck for these open source repos is maintainer attention. AI seemingly does not yet improve that throughput.

I'd buy that if lots of high quality PRs were sitting around waiting to be merged, but they don't exist. There's barely any evidence that AI is even real in the open source world, other than the slop of course

for small to medium sized bugs I managed to get a bit more than half of my PRs to get merged. ~90 PRs since May

https://june.kim/speedrunning-open-source

Verify yourself: { merged: search(query: "is:pr is:merged author:kimjune01 created:>2026-05-09T00:34:00Z", type: ISSUE) { issueCount } closed: search(query: "is:pr is:closed is:unmerged author:kimjune01 created:>2026-05-09T00:34:00Z", type: ISSUE) { issueCount } allTime: search(query: "is:pr is:merged author:kimjune01 -user:kimjune01", type: ISSUE) { issueCount } }

Man, truly absolutely gross to see this, this is some incredibly unethical engineering here. This is literally why people hate AI slop. Its not surprising to me that you didn't want to even write the post yourself

>It cost me 53 merges, 63 closures, a billion Opus tokens, and eight account blocks

I can't believe people think this is acceptable. You didn't fix any problems, you just managed to deliberately sneak slop past maintainers without their knowledge. There's no review by you here that any of the PRs actually fixed anything (presumably because software engineering is a learned skill), you just assume that the AI fixed the bugs without checking any of it apparently because.. you feel you're important?

the issues were already validated by the maintainers, and the maintainers merged it into their repo voluntarily. When maintainers accepted the PRs, they were the ones who found it acceptable. I found that well-tested PRs are more likely to get merged, so I deliberately had it pick bugs that were easy to verify. I didn't need to get involved after I specified that criteria.

Humans don't get a 100% merge rate either, so it's a matter of degree to which stranger is more effective at contributing to open source. It was an experiment where I accumulated lessons about how OSS works, and the lessons are open and available for your viewing here: https://github.com/kimjune01/sweep/blob/main/HYPOTHESIS_GRAP...

AI is good at fixing a narrow subset of issues/tasks in GitHub, so humans still need to be involved in most tasks.