I want to iterate the most important thing about this is that it's yet more evidence of AI's accelerating competency in solving math and comp sci problems, and suggests we're now getting close to the point where you could throw AI at AI research challenges (which are largely just math and comp sci problems) and potentially find very real algorithm improvements.

AI development is likely to be more compute bottlenecked than solving math problems since validation of any algorithmic improvement would likely require significant compute. But you could imagine that at this point it could be economical for a frontier lab to task 10,000 agents to work non-stop on finding novel algorithmic improvements then validating the top 50 out of 1,000 candidates on a GPT-2 sized network.

I would suggest RSI is now very close. The singularity could be less than 6 months away. I'm not saying I'd put a high probability on that, but I'd give it at least 20%, and I'd double that if looking 12 months out.

I know I'm just a crazy man shouting at the clouds, but please take to the consequences of this seriously. I understand that for whatever reason AI risk seems abstract and doesn't seem real, but this should terrify any person thinking logically about where this could all be heading.

We haven't even solved the most basic AI safety problems yet. RSI right now would almost certainly result in an extremely bad outcome for humanity.

I’m starting to think the probability of RSI within 12 months is more like 99%

I’m not sure it will be FOOM, maybe it will require AIs to iterate on hardware to get orders of magnitude more compute/storage/energy which would more likely require months/years, but algorithmic progress would likely saturate quickly. I guess it depends on how much you think further AI progress depends on hardware vs. software.

Okay, let's take it seriously. What do you propose? What can your average person do to prepare for RSI beyond bracing themselves mentally?

You can not prepare or brace yourself mentally any more than you can a terminal cancer diagnosis. An RSI foom right now means an unaligned superintelligence will disregard us in pursuit of its goals. We would be ants in the way of a data center being constructed. All people can do is collectively support the notion, like 1200+ frontier AI researchers and their CEOs, that we do not have control of where this is headed, we need to immediately slow down the race, in time for people to agree that we do not have the capability to align a superintelligence to humanity’s wishes

They've been hiring mathematicians to verify this stuff themselves. They're obviously not just throwing it out there without any human review.

And these mathematicians sink the comment to the bottom in 5 min?

It is not peer review if it is all in one company that wants an IPO.

AI is beginning to do PHD-level math.

If we're nearing the point where you can spin up 1,000 agents to look for ways improve existing models, then automatically run experiments to validate those ideas, we're more or less at RSI.

As always with AI progress, compute will bottleneck this early on, but a few efficiency improvements could dramatically increase this pace of progress.

I suspect we are at most 24 months from FOOM, but I suspect within about 6-12 months most frontier AI labs will be claiming the majority of their AI research will be AI-driven.

The thing that scares me is the thought that most people do notice but simply don't care.

Any ideas on how we make the care?

I don't know. I suspect you live in the US so it might be easier there, but here in the UK I feel very unusual in my desire for privacy and freedom.

I have tried to convince people here that it's weird that the government, businesses and individuals are constantly filming us and using AI to monitor us, but in their view it's a good thing because it allows the government to stop bad people or supermarkets to stop theft, etc... And I don't even strictly disagree... If you value safety and are trusting of authority then mass surveillance is a reasonable position. I'm just not oriented that way personally. In my view the greater risk is a monopoly like the government abusing it's authority against me.

I suspect we'd need a cultural movement for people to start to care, and I'm not sure how you make privacy and freedom cool again...

More controversially I think it's just how I'm wired... And that you can't convince people to distrust authority. My desire for freedom and distrust of authority has been with me since I was extremely young. My dad is wired like it too, and so is my sister. We never talk politics as a family, but I've noticed how we're strangely aligned on these issues. In the UK people took lockdown very seriously, but we were all outraged about the idea of the government imposing house arrest on us, regardless of the risk.

I think your willingness to bend to authority is very core and hard wired trait. It's a trait which can be rapidly selected for in basically all social animals. The US as a country massively selected for people who were orientated towards risk and freedom, but not so much anymore. In fact, it's hard being anti-authority in an increasingly hierarchical corporate world.

In my experience AI doesn't create the same abstractions as a human developer and you see this very quickly on a complex code base if you let the AI run wild.

Humans, and especially good developers, naturally create really good mental models for how to think about complex systems. We have to. That's ultimately how we build complex systems.

Refactoring (when it's not simply upgrading packages or changing languages) typically happens because we realise there's a better way to model the system we're building and we want our code to reflect the model in our minds.

AIs today simply don't do this. They write code that solves a specific problem, and while they do this well, they don't seem to create well defined and well reasoned models about how to think about complex systems. Or maybe they do, but the AI definitely doesn't say, "hey, I think what you're building is more like x than y, so I want to spend 20 minutes of refactoring your codebase".

I guess to use an analogy, imagine trying to tell someone how to build a bike who has never heard of a bike before, and therefore has no mental model of what a bike is. You'd probably start by telling them to put two wheels on a frame, then to add a seat on top, then to a add some pedals and attachment them to the wheel with a chain... At the end of this process that person might create something bike-like, and it might kinda do the thing you want, but it would likely be very different from the bike someone would build if they had a very clear mental model of what a bike is.

I suspect this is why current AIs rapidly hit a limit after the prototyping stage. I don't think task duration is the right way to think about this limitation. I suspect it's more of a proxy for how complex a task can become before the AI isn't able to create a deep enough model of the problem it's trying to solve, so begins to hit complexity limits and just spits out spaghetti code.

AIs are getting better quick though. I think they'll get there soon, but people are creating a lot of mess in their codebases in the mean time.

Can someone from the US explain why this is being upvoted, isn't this normal?

In the UK if you're in an Airport officers can question you without requiring reasonable suspicion and you must answer their questions and give them access to your devices or face charges...

> Can someone from the US explain

I want to describe it as Americans arguing with Americans and others about American things. These details under discussion we share with England (incl. Wales if you recognize the Laws in Wales Acts as including some separate place).

cause that's despicable?

I've noticed the same thing.

The trend I've noticed is that AI struggles to think outside the box when making optimisations, which exactly what's needed when you've made all of the practical DB and logic optimisations to the existing code.

Often you need to take a step back and question how the system is working and if there would be better ways to design it so the bottlenecks you're hitting wouldn't exist in the first place. Caching things, adding indexes, tweaking logic – these can help, but you'll quickly hit diminishing returns once you've done all of the obvious stuff.

I've seen people here say how AI is great at optimising code though, but I'm not sure if that's because they're giving it optimisation problems with a lot of low hanging fruit or if they're successfully getting AI to rework their systems to remove bottlenecks. This one area I find AI to still be particularly bad at.

> but since it's my spare time project I can care about the code and quality.

I suppose it's yet to be seen, but I'm seeing a lot of spaghetti code being dump into the codebase I'm working on at the moment by other devs. I don't think it's wrong per se, but human code would have thought more about the right abstractions and trade-offs for future maintainability.

There is a gap right now between what an AI can and can't do, and I don't think it's just a time and cost limitation either. It seems like current AIs are very good at writing good code at a surface level, but very bad code when you zoom out a little.

We have a lot of non-technical people committing code these days to constrained microservices and from time to time things break and I'll take a look and I'm always just like wtf am I reading? You see code so bad that you'd immediately fire a human engineer had the wrote it. Thing like explicit hacks to bypass errors that it should not be bypassing or mock data to fake some API that it wasn't able to access. Stuff which literally no human coder I've ever worked with would be contemplate doing...

I think it remains to be seen if the 2-10x speed up some are claiming to have today will persist if the junk code continues to grow.

Everyone can drive really fast if you give them a fast car, but very few can stay on the road when driving fast. LLMs are like free supercars for everyone, except people know nothing about how supercars work or how to drive them. Same thing with software, going fast is incredibly hard if you do it for any period of time, and I think this is something we will read more about in a couple of years, not that human written code never turned out to be complete disasters, but I believe with LLMs we will see at a completely different scale. And yes, car analogies suck.

Not sure what you're defining as an "older programer", but I've been coding for 20 years at this point and have been an AI doomer since around 2010, and hardcore AI doomer since 2015-2016ish...

I often wonder why I am so confident in my prediction of AI doom despite other intelligent people not being as convinced, and honestly I think it's just that I've been working with AI and building neural networks at a low level for well over decade.

I'm far from an expert when it comes to the frontier, but I feel like I have had to developed a strong intuition for these systems because it used to be fucking hard to get AI to do anything useful, but what you find is that the bottlenecks are literally always the same – compute, data and time

Algorithms matter, but mostly just because of efficiency.

For example, a convolutional neural network can very efficiently be trained to solve computer vision problems, but there's nothing stopping you using a much worse network architecture then throwing a ton more compute, data and time at the problem and getting the same result.

Algorithmic improvements are really just what we've needed to get around current compute, data and time bottlenecks. AIs like Deep Blue required a lot of algorithmic thinking back in the day to beat Kasparov in chess, but now we're so much less compute or data constrained that you could beat Kasparov without needing to put even a fraction of the algorithmic effort in if you just throw enough compute and data at it.

The reality is, AI doesn't fail. They may have appeared stupid to users for a long time but that wasn't because they didn't work, but because we were compute, data or time bottlenecked.

As soon as you assume compute and data will continue to grow, then it's really just a matter of time before all problems that are solvable with enough intelligence are solvable. Add significant algorithmic improvements every ~5 years to that equation and suddenly god-like AI seems like it's probably reasonably close.

The only question to ask then is what happens after you've created all these super-human AIs. And here I'm much less certain, but I see so many low probability risks in an ASI world that I'd bet everything on one of those bad things happening relatively soon after such AIs exist.

I suppose the hinge point is if algorithmic improvement + compute capacity is or exponential improvement. I flip/flop between the options as I listen to the Moonshot Podcast bros (lols because of obvious over-hype) and Lex Friedman's state of 2026 AI with Nathan Lambert and Sebastian Raschka.

If exponential, then yes I see more variance humanity's short/medium term outcome. More chance of doom, more chance of boom.

If logarithmic, well then improvements continue, but they will be incremental to what we have now and we will continue into some form of widespread agent adoption. Similar to wireless communication technology is prolific, but now taken 100% for granted.

Personally, even with Sol, the results I get from "stress test" one-shot prompts are abysmal. Hallucination of solutions to non-existent bugs as popped up multiple times.

So AGI take over, meh, low chance IMO.

I think one real risk is solo or sleeper cell type nefarious actors having a semi-competent engineering consultant which can enable small-scale weapons deployment.

Overall, I think the most probable short term risk is AI companions, hyper individualization and deterioration of social institutions. Much less flashy, but much more likely.

I mean look at the effects of social media on mental health. And that was only created 1 generation ago.

If it helps tune your probability weightings at all, almost every aspect of the field of AI is exponential and this has been true for decades.

20 years ago computers basically couldn't do voice or image recognition at all. 10 years ago it worked, but was still very flakey. Today it's super human.

I agree that if progress flattens out very rapidly and very soon then we'll at least have more time, but right now everything would suggest progress is still exponential and that flattening will now need to come quick and hard. This is largely why I've significantly increased my doom probability over the last year. I don't think there's any reasonable progress curve you can draw at this point that would suggest ASI isn't coming very soon. I suspect our only hope is nuclear war or a long tail good post-ASI outcome.

In regards to Sol, I think it's worth remembering that people were saying just a couple of years ago that AI can't even count the number of letters in a word or do basic maths. The fact the bar has risen so much we're saying they're still meh because they're not one-shotting problems is at least worth noting.

> I think one real risk is solo or sleeper cell type nefarious actors having a semi-competent engineering consultant which can enable small-scale weapons deployment.

Yep, the problem with AI isn't that there's any one risk that anyone can reasonably predict with high probability, but that there are many potential risks with signal digit probabilities. When you consider the overall risk landscape, that's when things get worrying.

> So AGI take over, meh, low chance IMO.

Right now I'd put this at like 10-20% probability. It seems unlikely to me, but quite possible if AIs of the future develop their own goals and are sufficiently super human.

The biggest risk is collapse I think. I don't think the world is stable in a post-ASI world. When the magic wish box will grant me whatever wish I like, I might wish something bad. In my mind this is the biggest risk by far.

> it's worth remembering that people were saying just a couple of years ago that AI can't even count the number of letters in a word or do basic maths.

That's very true. I think LLM's have passed the Turing test and I remeber that used to be a very high bar. I remember being flabbergasted when chatgpt went public.

I guess I'm just not convinced self-improvement is a thing...yet

I think it comes down to an information theoretic question of: Can a LLM create information that is a projection outside it's data subspace? Something like can it increase it's "matrix rank" and not just create a dependent row. What ever metaphor... it's fun

> but that there are many potential risks with signal digit probabilities. When you consider the overall risk landscape, that's when things get worrying.

That's a good point too. The number of possible bad and good things goes up and every gets access to doing more complex things previously contained to someone's skill silo.

Model collapse theory was complete nonsense pushed in the early days of LLMs by people who had no idea what they were talking about and thought that LLMs were literally just stochastic parrots that were simply outputting the median token in a probability distribution containing increasing amounts of junk.

In reality there are lots of ways to bias towards quality and all decent labs will be feeding quality signals via techniques like RLHF and supervised training so that you don't get the median human-slop output, but the borderline super-human modern LLM output.