It is an access issue. If you could get step by step instructions on how to modify a virus so it kills all people over 6ft you bet your ass there would be people attempting it.
Column A, Column B. Building a small explosive device isn't hard. Building a million is very difficult, doing it covertly virtually impossible without the resources of a nation-state.
The problem with biologics is the self-assembly and replication machinery comes for "free." So the numpties who might otherwise blow up a trash can [1] now have a real chance of taking out a million people.
The problem with biologics is that you cannot build a virus in your garage. You need a lab. AI will give your recipe, but you still need a lot of money and cooperation of other people (and if you have so, you could hire human biologist in pre-AI era).
Also AI makes mistakes. If you ever coded with AI agent you know that loop "write trash => compile => fix compilation errors => repeat" (if there are no compilation errors, there are definitely logic errors to be fixed). In real world cost of attempt is huge. You need a lot of money and you risk to draw a lot of attention if you perform long series of iterative experiments to create working virus.
In case with bomb it means that even if you have AI which gives you recipe of the bomb, but you will explode your garage and yourself with a decent chance. So you probably need to setup a good experimental pipeline (hardened lab where you can try different formulas and see that happens without being killed) if you want to go beyond publicly known explosives available in pre-AI era to anyone who read school/university chemistry books. And this also requires resources and draws attention.
People extrapolate programming experience (the area where experiments are cheap, cannot kill you and provide detailed feedback what went wrong) to real life.
They would still have to procure things that would (I hope) light up many screens before they're able to. And such numpties are probably already monitored, or in prison for some other stupid life decision.
I also would like to hope that people that are likely to do such things are probably:
A) don't know how to break even the most basic guardrails of models
> They would still have to procure things that would (I hope) light up many screens before they're able to
“Many of the largest and most responsible providers in the industry already screen and record orders voluntarily,” but there is no requirement to do so [1].
If that were possible, they would already be attempting it with the same level of ability as if they didn’t have access to a text file generator app. It is not about access to the information.
All of this “guardrails” handwringing is nonsense. These things output text. Are you for censorship of a book written by a biotechnology expert that gives out the exact same information?
I don’t know if you’re being silly but it is orders of magnitudes easier to modify an existing virus to selectively target certain snps than make “antiviral nanobots”
Claude, modify the existing 6ft killer virus so that it only makes my balls itch slightly for a day and gives me lifetime immunity to all further stamms of the 6ft killer virus. Make no mistakes, double check so the virus causes no unforseen complications.
It's inevitable. Also, it's not like I get to vet who does or doesn't have access. Blind trust in the current selection made by an unregulated corporation just makes me anxious.
Security in the form of "pay to play" is just kicking the bigger issue down the road.
The argument against security through obscurity isn't that it doesn't work at all. It does to a degree, only it is not as strong as people think.
An example from the meat world: not publishing your vacation dates well in advance for the world to see somewhat reduces your chance of being burglarized. That is security by obscurity; not reliable, but not completely inefficient either.
But if you live in a fortress (security by key material), you can well declare your vacation dates without running the risk.
It the tool was made available to anyone to build a virus, anyone would be able to build counter measures, if only a select few people have access they get to build the virus and everyone else is at a disadvantage. So, yes, I am leaning towards making these tools open rather than gated behind some priesthood and government that gets to wield exclusive power.
Compare the cost/ease of attacker vs defender if one person is given a virus to unleash anywhere in the world and another person is given a vaccine to distribute to the whole world. Or compare building a large bridge to someone disabling that bridge, etc. Prevention and repair is almost always more expensive than vandalism.
I don't think there's an ideal solution here, but giving trusted people access to fix security issues before giving it to the wider public seems like a reasonable compromise. They're letting you use the model for all other uses.
you need a lot more than the nucleotide sequence to make a virus. you need the DNA or RNA to be synthesized, assembled, packaged properly. and long sequences are pretty hard to do. you need a lot of equipment, or you need to order from services. the oligo synth services can harden their KYC and/or screen for suspicious sequences.
sure, a malevolent state actor could swing it, but they could make a bioweapon without Mythos's help already.
also, vaccine production and disease surveillance have ramped up very quickly. they will ramp up further, despite political setbacks. it's a cat and mouse game that favors the defenders IMO.
but the bioterrorism narrative is useful FUD to spin open-weight models as existentially dangerous. I am far more worried about Anthropic's own goals than the goals of some crackpot in a shed.
> it's a cat and mouse game that favors the defenders IMO
How so? I'm actually against most of the "safety-tuning" that anthropic does, but this seems fundamentally untrue, a close analogue being video game cheat development. I think in general the cheat developer has an advantage and the cheats generally proliferate for quite a while before being patched.
Video games are an interesting analogy since they often trade security for performance, trusting clients about world state quite a bit.
Finance and biology do come across as two similar high level systems. But while we can employ KYC, fraud detection, and various auditing techniques to finance, I don’t know what you do for biology. You can easily run an algorithm over every transaction a person makes in their account but there’s no equivalent for every cell, every bacteria strain, every virus in the human body.
(disclaimer: layperson remembering how the immune system works.)
the adaptive immune system effectively does KYC by checking the antigens presented on the surfaces of cells. the thymus selects for B-cells (iirc?) which don't react to a corpus of the body's own antigens, but cover a wide library of everything else. when it sees something it doesn't recognize, it reproduces, warns the rest of the immune system and marks targets. that's why our immune systems can eventually conquer almost every pathogen we encounter, if we can survive long enough for it to do its work.
but the KYC I was referring to was KYC that vendors of oligonucleotides (should) be doing, to keep people from ordering nefarious sequences.
I'm bullish on mRNA vaccine technology to release the "patches" much more quickly. there was widespread resistance to this during covid, but covid wasn't horribly lethal. if airborne Ebola spread as productively as covid, for example, I doubt there'd be many anti-vaxxers left (one way or another!) the acceleration of biology research that might accelerate pathogen development should also accelerate the development of broad-spectrum mRNA vaccines with high persistence.
also, afaik the most effective way of developing pathogens is through serial passage through humanized mice or something like that - directed evolution at a small scale, selecting for traits. AI simply isn't needed for that. I don't think information or intelligence has been the bottleneck for bioterrorism, it's motivation and resources - same as for any other kind of biology research program.
EFF doesn’t allow most people to reply to their X posts. Scroll on their profile right now and you’ll see you likely don’t have the ability to reply to their posts.
This will damage their view count according to the algorithm bc this limits their engagement
you have 2 labs at the forefront (Anthropic/OpenAI), Google closely behind, xAI/Meta/half a dozen chinese companies all within 6-12 months. There is plenty of competition and price of equally intelligent tokens rapidly drop whenever a new intelligence level is achieved.
Unless the leading company uses a model to nefariously take over or neutralize another company, I don't really see a monopoly happening in the next 3 years.
I was focusing on a theoretical dynamic analysis of competition (Would a monopoly make having a competitor easier or harder?) but you are right: practically, there are many players, and they are diverse enough in their values and interest to allow collusion.
We could be wrong: each of those could give birth to as many Basilisks (not sure I have a better name for those conscious, invisible, omni-present, self-serving monsters that so many people imagine will emerge) that coordinate and maintain collusion somehow, but classic economics (complementarity, competition, etc.) points at disruption and lowering costs.
I don't disagree but in this day/age figuring out who the customers are is fairly trivial with "AI" and then simple marketing campaign will at least point the right eyes to the right place. Unless there is large moat switching will happen (it is already happening across the industry which is why we are hearing about the death of SaaS all that jazz...)
Private companies should have the freedom to ban/censor whatever content on their platforms they want. I’d prefer if they don’t, but we shouldn’t force governments to prevent companies from creating their own rules about how people can use their own software
Governments however should not interfere with citizen’s freedom of speech - there should be no fines/arrests for insulting politicians. Otherwise those governments are actually authoritarian and repressive.
Allowing private companies to do anything they want, when there are only a few large private companies, makes them a shadow state. Europe is much more willing to restrict companies from becoming shadow states. If they make a law that says bank transfers must be used for large payments (or if that becomes de facto true) they also make a law that says banks must give accounts to everyone.
Germany – Robert Habeck insult raids (2024–2025): Multiple citizens faced police raids, investigations, fines, or suspended sentences (jail risk if violated) for online posts calling Green politician Robert Habeck derogatory names like "idiot" or "moron," or sharing mocking memes, under Section 188 enhancing penalties for insulting politicians.
https://www.dw.com/en/germany-greens-habeck-presses-charges-...
Germany – Friedrich Merz "Pinocchio" case (2025–2026): A pensioner faced criminal investigation (potential fine or jail under Section 188) for a Facebook post calling Chancellor Friedrich Merz "Pinocchio," prosecuted as an insult likely to impair a politician's public duties.
https://www.facebook.com/60minutes/posts/dozens-of-police-te...
Germany – Ricarda Lang insult investigation (2024–2025): A citizen was investigated (potential fine/jail) for an online post calling politician Ricarda Lang "fat," charged as criminal insult under Section 185 protecting officials from derogatory remarks.
https://nypost.com/2025/02/21/world-news/germans-cant-insult...
Robert Habeck was NOT arrested, he and his friends were investigated in the broader case of neo-nazi propaganda which they were spreading as well. Unless you consider neo-nazi freedom of speech, of course.
The Pinocchio case meant exactly one official letter sent to that guy, lol "arrests". The investigation was dropped and everybody criticized the investigation.
Ricarda Lang case was a request to the well-known network Gab to identify who insulted the politician, because in Germany insults are a crime. Maybe in the US insulting is a popular free speech pastime, but this is not US. Gab refused to identify the person and that was that.
So, again, I can see when we are spreading lies to support some ideology, but they are just that: lies.
Re the other cases: in a good democracy, insulting politicians should not be a crime and there should be no investigations for someone insulting a politician.
That is your POV. I fear that democracy erodes when there's insults, belittling, ... instead of exchange of arguments and the contest of ideas. Because at some point insults turn into ugly actions. Whether it's Charlie Kirk or Melissa Hortman.
There is a reason the founding fathers put freedom of speech as the first amendment
Insults should absolutely be protected speech.
In countries that make insulting politicians illegal, all a politician has to do to become a dictator is say that speech criticizing them/their behavior is insulting and therefore illegal
Would you like if Trump arrested anybody who insulted him?
I mean that's why it's called free speech. Probably the most famous case the ACLU (American Civil Liberties Union) fought for was to make sure Nazi's could hold a rally and march through Skokie, Illinois, USA an area famous for being predominantly Jewish.
> Multiple citizens faced police raids, investigations, fines, or suspended sentences (jail risk if violated) for online posts calling Green politician Robert Habeck derogatory names like "idiot" or "moron," or sharing mocking memes [...]
The police raids were done because of the posted Nazi images, NOT because of the Habeck insults.
reply