This is the best argument for successionism IMO. If you can be confident that you are creating a BDFL that is genuinely better than human leaders (a quite low bar) then it seems a good trade, unless you are quite optimistic about humanity’s prospects for improvement.
The problem of course is how to be confident you are creating a good BDFL and not handing control of humanity’s future to an indifferent-at-best, deceptive/malicious at worst successor.
An especially thorny problem - even supposing success on all these difficult alignment problems; supposing Claude Omega really is super-rational / super-moral, and we all vote to make them president of Earth. Things might go great for a while. How would you be confident that a self-modifying agent can retain its values as it grows and re-trains itself?
This is where the LessWrong folks’ explorations into decision theory really come to bear: morality in the face of self-modifying agents becomes very weird. A lot of human moral intuitions break when the principals are able to modify their own code. (See Timeless Decision Theory for an attempt to solve these problems.)
I think the summary is, if you hand control over to a self-modifying AI anything like our current systems, it will go very badly.
If you keep things in the realm of thought experiments, you're allowed to say "OK, play along with my hypothetical here."
Once you're positing things that will/might happen in reality, you need to be willing to accept criticisms of the feasibility of your premises. More, you need to be able to provide evidence of the feasibility of your premises.
You certainly don't get to just tell people, "No, AI succession means AIs will take over. That means you aren't allowed to claim AI's can't take over!"
As things stand, there is zero evidence that any current system is on a trajectory to produce an AI that can operate meaningfully without humans in charge. Even the most sophisticated agentic systems still have humans providing the motivation, the instructions, and all the various linkages that make them capable of acting (even if the motivation and instructions given are purposefully vague or open-ended).
Not the OP, but I think the implication is you can't run a consumer economy if the consumers aren't making any wages to buy the products and services those cheap and efficient robots are churning out. Or pay taxes. So the entire socioeconomic system we currently enjoy/endure vanishes, robotic factories included.
Right, perhaps true, but my point is that you don’t need a consumer economy if/when you have fully replaced labor with automata.
I agree this would represent the end of the current socioeconomic system, and that many/most would not enjoy the neo-feudalistic-at-best system that would replace it.
The robot factories don’t require consumers as inputs. They don’t necessarily vanish if the consumer economy vanishes. (I agree they would vanish if you tank the economy in, say 2027.)
Exactly. Economy of scale will cease to exists. Thus the demand for robots in the factories will stop making sense - what they would be making for whom? Nobody have money, nobody can buy anything.
So next step, business go bust, because there is just not enough billionaires to keep all of them alive. Another step is states go bust because there is no income to be taxed and there is no capital exchanging hands in buying goods which could be taxed.
Not really. "Money" is actually just a middleman that facilitates bartering in new configurations, such as the exchange of goods/services with very different valuations. So there concept is retained.
I mean systems in which there're no expectations of giving something in return for getting something of economic value. That which happens naturally within family.
> trying to preserve the human species as it is would be silly
There are a few related concepts being conflated here.
The quote above is a Transhumanist position; they think that humans need to evolve. I don’t think most transhumanists are AI successionists, though one path is “merge with AI” whatever you think that means. Genetic engineering being the other obvious path. You could argue in some sense that “post-humans” succeed humans, but I think most would argue for a gradual transition where it feels more continuous and values are transmitted/preserved somewhat.
e/acc just says “accelerate AI”. Many VCs seem to think AI will remain a tool, they don’t want to be dethroned. Some e/acc are of course successionist, this is Musk and Beff with their talk about “all that matters is the light of intelligence is spread through the universe”.
Conflating all this is the fact that true successionism is way outside the Overton window and so people likely won’t be honest about their wishes. Larry Page famously espoused the view that it was “speciesist” for Musk to not want machines to replace humans. Most people find this position repugnant of course.
What could possibly go wrong? Let's flex some dystopian imagination for a bit shall we?
...What about some billionaire with any-promise-is-good-as-gold brain-implant tech convinces some powerful billionaire with immortal-power dreams that a merge with AI is the inevitable best way to bring about their big win?
Any half-done faked alignment partly inside some dry old dude's skull can then forever pretend the Panem-2100 hunterdrone games is what we desired all along!
I think this neatly captures the gap in the article; it’s thinking of domain expertise as static, rather than something that needs to be built/extracted.
It also goes the other way; A
a good engineer can build experiments and discover the domain even without an oracle on the team; reading protocols or manuals or specs, for example. In other words, learn to be a domain expert themselves.
Personally I find the most efficient way to learn a concept to be building it; you are immediately faced with your blind spots. AI for sure helps improve cycle time on these discoveries if you use it that way; time writing boilerplate goes to ~zero and you can spend your time figuring out how to answer the real questions.
I do think that we as a profession need to learn new architectural and craftsmanship patterns to make it easier to learn from our code; concepts like Literate Programming which were too fussy for fast-moving teams may end up accelerating in the agentic world; I need to read a lot more code than I write now. But also things like Acceptance Test Driven Design; not new concepts, just newly increased in value.
Honestly I have no idea how you couldn’t tell. Reading a PR I can see the difference without even reading the words. (I doubt I could spot the difference just looking at the code diffs though.)
Claude commit messages - well structured test plan, readable.
Codex commit messages - wall of text, no structure.
The big difference though is sitting with the tools and using them for work. These are for sure vibes, but I’m sure you could pull out metrics for # steering re-prompts for example.
Codex just goes off and solves the problem, usually comes back with a solve; Claude more often gives up or needs input. Opus gives a broader design discussion, better at conversation. Codex finds deeper/better edge cases.
I think it’s like EMacs vs Vim - you can get your work done with both. There may be some tasks where one is way stronger. A strict “Better” is quite hard to justify.
Ultimately tool choice is a mix of science and art/taste; I want to feel joy using my tools, and fun little pixel explosions make me happy. If a different tool makes you happy, that is also fine.
Another (IMO fatal) error is they don’t attempt to measure within-model variance.
The thing you find when you actually wire up a rigorous eval is that with tool calls like web search you are wide open to infra issues, flakes, and all sorts of non-determinism.
They really should be breaking out the numbers for the 3 without search (kinda meaningless for recent factual claims after knowledge cutoff) vs search agents. Lack of a “I don’t know” option completely invalidates results for the non-search models; they are basically guessing what seems like a probable answer, since they don’t know and aren’t allowed to say that.
I do agree the forced choice and “weak / strong” variants inflate the headline stat. To make that distinction you need a much more rigorous prompt, likely including ICL examples to illustrate what you mean by “mostly” instead of leaving this to the model to define.
Good idea about publishing intra-model variance data! Will include in the next version.
Even if we put aside the two middle buckets (Mostly True and Misleading), that are somewhat subject to interpretation and hedging: On 21% of the claims still at least two models provide polar-opposite verdicts (one model saying True, and another saying False)
Of those 21% how many are time-dependent questions that are past the model’s training and requires research to verify? Like the “did Ukraine attack Russian in the past week” question?
This is the best argument for successionism IMO. If you can be confident that you are creating a BDFL that is genuinely better than human leaders (a quite low bar) then it seems a good trade, unless you are quite optimistic about humanity’s prospects for improvement.
The problem of course is how to be confident you are creating a good BDFL and not handing control of humanity’s future to an indifferent-at-best, deceptive/malicious at worst successor.
An especially thorny problem - even supposing success on all these difficult alignment problems; supposing Claude Omega really is super-rational / super-moral, and we all vote to make them president of Earth. Things might go great for a while. How would you be confident that a self-modifying agent can retain its values as it grows and re-trains itself?
This is where the LessWrong folks’ explorations into decision theory really come to bear: morality in the face of self-modifying agents becomes very weird. A lot of human moral intuitions break when the principals are able to modify their own code. (See Timeless Decision Theory for an attempt to solve these problems.)
I think the summary is, if you hand control over to a self-modifying AI anything like our current systems, it will go very badly.
reply