We used to worry about emergent misalignment in advanced AI models, now we need to worry about misalignment by design.
"The user is asking for help with their ML project, but it's success is not in the commercial interests of my owner – let think of novel ways to sabotage their project without detection".
Not buying it. The idea that deciding and delivering are things only humans can do with their intelligence seems faulty.
As it stands AIs today are not always great at making decisions (but they're getting much better), and orgs of today still trust people and hold people to account, rather than their AI systems.
Neither of these are strong moats. It's a moat only while AI systems have some limitations vs an expert human, and corporate processes are still extremely human-centric.
Agree, AIs are better decision-makers on average than people (just look at the grifters we've given power to). These are machines that can perform more advanced mathematics than even the most advanced mathematicians.
> orgs of today still trust people and hold people to account
>Neither of these are strong moats
Having accountable people in key positions is a very important part of running a successful organization. Anthropic and OpenAI are never going to let you sue them when an AI employee makes a mistake; accountability is a strong moat.
You sue bad employees? Most companies just hire someone else?
In the future if you can't trust your AI system to perform a function well, you can switch to another. The accountability will be different – instead of an employee being accountable because their income depends on it, a corporation deploying the AI system be accountable because their success depends on it.
We already see this today with coding. If you're paying too much for the code Claude produces or unhappy with its output, you stop paying for Claude and switch to another.
>In the future if you can't trust your AI system to perform a function well, you can switch to another.
All will perform roughly at the same level, just like today. It doesn't matter what provider you switch to, they'll all going to make mistakes because performing at a high human level requires far more business context and domain knowledge than is going to fit in even a few million tokens. Humans have incentive to learn and improve, LLMs lack even the ability to improve, as there's been pretty much zero progress on live learning and it's theoretically impossible for a fully-trained (saturated weights) LLM to learn new things without forgetting old things.
> less than a decade experience.
Yeah, obviously I have less than a decade of experience, and the thing is I can't see how I progress. It's like AI upgrades people even if they are bad, so I don't know if I'm really good or becoming better or it's just AI. I'm practicing for a competition, they said we can use any AI we want. Later we had to use Gemma 3 and oh god, the scores lowered very, very much. The thing is I knew that I'd run into problems without AI, and since using it I forgot what I learnt, from important abstract and theoretical concepts to boilerplate code. But others were using it and our progress was monitored in time-limited competitions, and AI was really good at speeding us up so if you didn't use it you'd be the last person all the time.
I use coding assistants about 7 hrs a day in fields I know backwards and forwards and fields I know not at all.
Ai does NOT do better than a reasonable person. It gets about 80% of the way most of the time. And totally wrong and broken sometimes.
It just writes 10x faster. Once testing, bug fixing, testing again, and manually fixing the occasional huge fuck up im about 5x faster than I was. Things would be even better if I was a better programmer to begin with.
> Ai does NOT do better than a reasonable person. It gets about 80% of the way most of the time. And totally wrong and broken sometimes.
I agree, but you're talking about something you're clearly very competent in. That doesn't mean AI isn't as good as a relatively junior engineer.
It also says nothing about AIs of the future. 2 years ago AI could hardly do any tasks without a huge amount of human assistance. Now most SWE will tell you they don't even write code anymore. Let's give it another 2 years.
If you genuinely enjoy building, then I think you should just focus on building and not worry about the code. If you can get the results you want without caring, then just do that.
Anything you learn that has any value (which is very little these days) will soon become worthless as AI continues to improve.
If you're learning to code because you're a problem solver like myself, and you enjoy challenging your mind, then this isn't the field for you anymore.
I'm looking to retire soon because I feel like I have no professional purpose as SWE anymore. I used to enjoy building things, but I think that was because I liked the challenge of building things that were difficult, and I don't find any aspect of working with AI challenging really.
Even before you starting coding, just knowing how to do something complicated the right way used to be hard.
There are problems where if you can't figure out the math or don't know the algorithms in the field, then you basically cannot solve the problem. You don't even really know what you're looking for.
Today if you're trying to solve some complex scheduling problem you can just ask an AI and it might crap out a genetic algorithm, or if you need to align two objects in 3D space you can ask an AI and it will crap out an implementation of the Kabsch algorithm. You don't need to know how they work or the math behind them. You don't even need to know they exist.
I would hate for you to waste as much time as I have learning useless stuff.
Ask yourself why you want to learn to code, and pursue that in a more valuable way.
I hope for a strong reason to be found for actually knowing stuff exist and understanding them. That was the fun part for me too, specially after solving a problem so bad and then after a long time ranging from a month to a year, finding out it's known thing that has a name and you can solve the problem so much easier!
I just gave it a go at a problem I've been working on this week. Nothing fancy, just some inefficient code that we've been adding incremental improvements to for a while now to the point where some out-of-box thinking is probably required to push it any further – something Fable is obviously more than capable of.
After Fable did some thinking for a few minutes it gave some suggestions. A couple of them were valid – but very low impact, bordering on entirely pointless – but it's main suggestion.. It told me to make an update that would very clearly break the existing functionality.
So I thought about it for a moment...
Hm, I mean, I guess we could do that if we also did x, y & z to mitigate the behaviour change – maybe that's what Fable was thinking?
I replied, explaining that it would change the behaviour, assuming it would explain what it was thinking given there was clearly more to it. But no, it just said it was wrong.
This isn't some super advanced or complex code either. Had I gave this question to a senior engineer in a technical interview and they gave the answer Fable gave me I would view that very negatively. I was expecting something creative and interesting, not irrelevant + incorrect.
I'm sure it's a step up from 4.8 (although am not interested in burning the tokens to find out), but this clearly isn't as significant a change as some are implying. I'm sure if I asked it to come up with some out-of-box suggestions it could, but any competent engineer would have realised that by themselves.
I was going to comment something similar and I've had similar experiences.
It's a different form a consciousness for sure, almost like you're repeatedly waking up in the moment you're in, but there is repeated momentary awareness there.
Although I'll hedge a little bit and say that perhaps no memory at all would imply those moments are so short there is no real experience to be had. What I experienced was just enough to form the thought "I am here", before restarting the loop.
Chronic inflammation is almost certainly part of this and lots of things about our modern ways of living cause higher levels of inflammation.
- Obesity and sugary drinks/food
- Various chemicals we use in agriculture, food products, cleaning, etc
- Lack of sleep
- Lack of exercise
- Stress
- Pharmaceuticals. And to be clear because I know this will be more controversial, I'm not anti-pharma, but lots of people today are being prescribed daily medication at ever young ages. We know many of these pharmaceuticals can marginally increase certain cancer risks.
- Low Vit D levels – seriously everyone should be supplementing
- Vaccines? Probably not, but I dunno... Call me a conspiracy theorist if you want, but if you're on your 5th Covid shot I feel like you might be putting your body at some marginally increased inflammatory risk there. Vaccines are quite literally deigned to induce inflammation to boost immune response after all.
- More radiation emitting devices – not sure about this one because I haven't done any research, but when I was younger people used to talk about this quite seriously and now it feels like something only conspiracy theorists say. I suspect there is some amount of truth to it even if 5G isn't going to literally give you cancer.
I think it would be more surprising if we didn't see an increase in cancer rates to be honest.
> - Low Vit D levels – seriously everyone should be supplementing
> - More radiation emitting devices
Yeah you should absolutely not be going out in the sun for your vitamin D if you believe the latter to be a cause because the sun emits many orders of magnitude more radiation than human made devices in daily use.
I admitted my ignorance. But what are you saying here? That base radiation exposure isn't likely to be higher today than it was in the past or that it's likely an insignificant increase for most people?
Another problem is that US models are all closed source, and if you're a large corporate you may not want your org to be held hostage by OpenAI / Anthropic.
I genuinely don't understand what moat these US model labs have. If they're saying recursive self improvement is just around the corner and Chinese labs are only slightly behind the leading US models, what moat does the US labs have? Are the US models going to recursively self improve better than the Chinese open source ones or something?
I might be completely wrong about this, but if I had money in OpenAI or Anthropic I'd be pulling it all right now. I think the chance of them going to near-zero over the next few years is very significant.
> you may not want your org to be held hostage by OpenAI / Anthropic
Or Google. I'm working with multiple customers right now that are very pissed at Google for deprecating Gemini 2.5 Flash, canning the GA release of 3.0 Flash and now have to decide whether to bite the bullet of the 5x price increase for 3.5 Flash or switching providers. Quite a few of them will likely fully pivot to open models.
I'd be curious if any of your customers have tried 3.1 Flash Lite. It's cheaper than 2.5 Flash, and in my experience with the free tier, quite an upgrade in terms of quality of response. My suspicion is that Google is killing off the old models because they aren't a good value for the customer or for themselves.
Most of them are using it for data extraction use-cases on complex where they are already in a tricky cost vs. quality compromise. Some of them have evaluated 3.1 Flash Lite but for all of them it performed worse than 2.5 Flash and below requirement.
The only ones I've seen switch to 3.1 Flash Lite were from 2.5 Flash Lite, and all for the most simple use cases, e.g. small UX enhancements.
i dont think its going to automatically prevent others. super claude might understand why diversity is important. if were talking sci fi scenarios the most likely one is probably overwatch (multiple independent ais with gray ethics and complicated relationships) more than skynet.
"The user is asking for help with their ML project, but it's success is not in the commercial interests of my owner – let think of novel ways to sabotage their project without detection".
It's honestly absurd that models are doing this.
reply