This happens in any industry where value/status are at a premium.
Finance, Law, VC guys were good too in the beginning but when the value/status change happens it attracts certain kind of guys who are average in talent but excel in demonstrating value and social management of the value/status.
Another change which has happened recently is that the economics of engagement farming have become common place wisdom as already proven effective for everything from selling books, personal brand, career skill/virtue signalling, staying relevant.
Due to this everyone is talking more without restraint and not keeping in their own lane of earned expertise.
Academia is the most relevant, toxic example that I can think of. Be horrible to others on a short term contract (grad students, postdocs) and break them whilst extracting maximum value -- get more papers, more grants written -- more money -- success.
Be nice, think about hard problems for a long period of time, only speak up when you have something positive to contribute -- be labelled an underperforming academic and managed into obscurity.
A great example of this is Peter Higgs, who famously said that he'd be unemployed pretty quickly in the academia of 2013. [0]
Academic pursuits used to be self-funded by gentleman scientists, not governments. Whenever something gets subsidized by the government it inevitably becomes dominated by exploitive petty politics.
All of the low hanging fruit that could be discovered by self-funded gentlemen scientists has been picked. That doesn't scale to a supercollider or a large RCT. Funding at the whims of rich benefactors is very susceptible to petty politics.
Politics is irreducible from human affairs, privatization doesn't eliminate politics. It relocates it to a different set of actors. That could be a better set, but when it is it's because it's a more local and hands on group of people, not because those people happen not to work for the government. Governments are awkward because they are deep bureaucracies, and deep bureaucracies divorce the decision makers from the impact of their decisions. Weaker feedback leads to worse decision making. Not because there is a magic property of government that makes it uniquely bad. Large corporations, universities, and other deep non-governmental bureaucracies have similar pathologies.
That's something of an exaggeration, they are empowered to do violence and collect taxes and other things that are more problematic when abused, but still, privatization isn't a silver bullet.
>Politics is irreducible from human affairs, privatization doesn't eliminate politics. It relocates it to a different set of actors.
We ideologically privatised the water sector into regional private monopolies in the UK, and anyone who's had experience with the water monopolies knows this is the truth.
> Whenever something gets subsidized by the government it inevitably becomes dominated by exploitive petty politics.
When younger I've had job in groceries stores and saw petty politics.
There's nothing particular to being subsidized or not: politics is something humans do, and the pettiness is simply a reflection of the people involved.
"Whenever something gets subsidized, it inevitably becomes dominated by exploitive petty politics."
I think it's just limited resources + the single most natural way for humans to compete for limited resources. This isn't actually an inevitable outcome - just the most likely one.
The "self-funding" regime requires people who are both rich enough to afford to fund science and sharp and driven enough to advance science to exist. That's a high bar. And while there is some correlation between intelligence and wealth, the tails come apart hard. People driven to pursuit wealth above all may not be driven to pursue scientific discovery.
We have plenty of billionaires, and preciously few of them actively pursue pushing the frontiers of science and technology. Even by funding the endeavors - let alone by being in the trenches themselves.
> Whenever something gets subsidized by the government it inevitably becomes dominated by exploitive petty politics.
Your US-blend of anti-state brainwashing is showing. There is nothing inherently different in the for-profit status of an organization that prevents the occurrence of "exploitive petty politics". You see those from any organization from homeowners organization to full blown FANGs. I mean, have you ever paid attention to the crap being pushed by the likes of Tesla/SpaceX/Twitter?
public defender lawyers who fought for workers rights and against items like company towns, abolitionists, etc. many good lawyers
finance people who invented life insurance, health insurance, car insurance, friendly societies. as much as we complain about insurance here in the US, life was immeasurably worse when there was none. there was no such thing as state health care or social security in those days
you would be surprised to find that there are many people in finance who never tried to make a quick buck, and are pretty altruistic. this is evidenced by the large amount of family owned banks
tech now going through what finance did in the 1980s, shift to greed and excess
The "good lawyer" is/was a major archetype in modern Western culture. See "The Devil and Daniel Webster", "To Kill a Mockingbird", "Paths of Glory", just to name a few examples.
“ In 1992, an Alabama editorial called for the death of Atticus, saying that as liberal as Atticus was, he still worked within a system of institutionalized racism and sexism and should not be revered. […] Critics of Atticus maintain he is morally ambiguous and does not use his legal skills to challenge the racist status quo
[…]
despite the novel's thematic focus on racial injustice, its black characters are not fully examined.[79] In its use of racial epithets, stereotyped depictions of superstitious blacks, and Calpurnia, who to some critics is an updated version of the "contented slave" motif and to others simply unexplored, the book is viewed as marginalizing black characters.[130][131] One writer asserts that the use of Scout's narration serves as a convenient mechanism for readers to be innocent and detached from the racial conflict. Scout's voice "functions as the not-me which allows the rest of us—black and white, male and female—to find our relative position in society".[79] A teaching guide for the novel published by The English Journal cautions, "what seems wonderful or powerful to one group of students may seem degrading to another".[132] A Canadian language arts consultant found that the novel resonated well with white students, but that black students found it "demoralizing".[133] With racism told from a white perspective with a focus on white courage and morality, some have labeled the novel as having a "white savior complex",[134] a criticism also leveled at the film adaptation with its white savior narrative.[135] Another criticism, articulated by Michael Lind, is that the novel indulges in classist stereotyping and demonization of poor rural "white trash".
Totally. I feel the author, we just used to nerds, but now the space is occupied by social media and false narrative that revolves around founders. No ego hurt here of course - but it is hard to imagine Woz or Stallman to ask for a mass surveillance program or ads in AI or pushed AI search in internet search. I believe this article actually went to this realm - tech for tech, having fun…but all we get is maxxxx enshitification.
Yeah, for law I imagine these "nice" beginnings were 2000 years ago at best. If they even existed at all. But all these jobs where talking to other humans is paramount will be dominated by extroverted quacks by default. Same goes for the capital raising college dropout pseudo-tech-bros. They were never nice, they just weren't so engaged in public discourse before, when billion dollar net worths still meant you actually had revenue and not just a vague trendy idea.
Not that far. Lawyers had a great deal of influence in creation of all modern nation-states, human rights, international law and maintenance of the core social contract in the modern society.
Similarly lawyers/bankers were the ones who built in trust in capital, contracts, businesses and protection of investor rights. Delaware c corp is not an outcome of bad guys.
I got into programming and computers due to their intellectual depth, and the exciting opportunities they opened to explore everything from electronics to obscure areas of mathematics... through to theory of mind and the dream of making silicon think.
The combination of endless trend-chasing, software churn, and techbro culture made me hate everything about software, so I jumped ship to biology.
Well, and now with the push towards AI slop and letting agents do work for you, it is even less about creativity and talent. You can't even chase trends in libraries while still being clever about it any longer, you gotta chase more and more braindead ways of getting code generated based on tons and tons of mediocre code found online, gobbled up by big tech without the original creator s' consent.
I think that plenty of 'nerds' would very much disagree about that. Steering agents effectively is something that can take massive amounts of creativity and talent and be quite helpful wrt. the final result. There's no real analogue of this in traditional programming.
It's being the tech lead of a team of junior to mid level developers. You design roughly what the solution should look like, split it into reasonable sized tasks so they don't go off the deep end, advise them on some of the details, then assign them the tasks and let them get on with it, keeping an eye on what they're doing, reviewing their output, and course correcting them when they go wrong.
Just like with a team of humans, you have to use your judgement as to how much supervision they need individually and how large a task you can give them without them going off the rails.
Thats my thought as well, LLM agents put you in the role of a (often micromanaging) tech/team lead of a small team, but the speed and fast feedback loop makes it look different.
> Finance, Law, VC guys were good too in the beginning but when the value/status change happens it attracts certain kind of guys who are average in talent but excel in demonstrating value and social management of the value/status.
Those career paths were always crooked. We see that going back to my great grandparents time with Black friday of 1929. They fucked around with unrestricted capitalism, and found out. Quite a few killed themselves by throwing themselves off of buildings.
It was only when FDR took office and worked with Congress to make tons of rules keeping the money hoarders from destroying the economy yet again. And it bloody worked. For those of you who say FDR was a communist, absolutely not. He was fighting against a large contingent of the population who were socialists and communists. He did appease some of their demands, bit not many.
FDR led us into our most glorious 20 years, the 1950's to 1960's. Cheap education, cheap homes, plentiful well paying jobs, only needed 1 worker per house. Thats what the boomers remember and want.
And it was systematically dismantled piece by piece.
'VC guys were good too'?!?! I take it you do t remember the 1980's Mergers and Acquisitions crisis? Thats when enough data was available for a company, that mergers, acquisitions, and liquidations coukd make a handful of people scads of money, and destroy the economy to boot.
And i also scarce remember a time when 'Finance' was good. Their slur was beancounters. Something costs $20 but saves $1000? Nope, its -20$. The loss is never analyzed. Every job Ive woeked in has had this perverse logic.
And especially with money, Goodharts Law comes to mind. "When a measure becomes a target, it ceases to be a good measure".
"Men living in democratic ages have many passions but most of their passions either end in the love of riches or proceed from it." Alexis de Toqueville
Everything is extractive. Farmer plants seeds, partially sets the environment. The work is done by the seed/sun/soil/water. And so is every profession: labour or not. Most of the business are structured in such a way that someone can exploit them to make even more money. The whole vendors and b2b system is mutual extraction.
Looking through wages and trying to find a ceiling(by time/effort) on the value creation by a human is one dimensional at best.
The point of farming is literally to "extract the value" that something else creates.
> The work is done by the seed/sun/soil/water.
and the farmer collects. It's not that the farmer does nothing or deserves nothing. But it is precisely the same as the capitalist model: the capitalist sets the stage for labor to do the work, and then collects.
As others have noted, the central question is who gets to benefit from what is created and why.
From the perspective of the farmer re: natural resources and the capitalist re: human labor, they are precisely the same: an existing capability in the world that can be used to produce value that can be sold for more than that production costs.
Obviously when viewed from other perspectives, they differ significantly.
Understood, but you're removing the moral aspect of the capitalist's exploitation of labor from a discussion on why the capitalist gets to make a billion dollars and the farmer doesn't.
There is a difference between the work most do: working for an hourly wage, and effectively getting rich through capital gains.
This is what aoc is referring to essentially. It's practically impossible to become a billionaire through "regular" work alone that pays you a salary.
I'd argue that for all super wealthy people, their salary isn't the major factor in how they gained their net worth. Lets take Googles CEO, he makes 2 mict llion per year (the exanumber isn't that relevant here). With this salary it'd take him 500 years to earn his net worth. Again, completely different proportions to "normal" people earning their net worth through their job. And I'd argue you can do this for everybody with more than 100 mil. dollars.
Documentation is worth it only if it is read. If your coworkers don't read/remember/respect the documentation process then people tend to not keep the docs up to date. Unless the docs are for users who you don't want to come to you at all for support.
Claude is a better reader. I have to just tell it to read the docs/specs sometimes.
This is so true. I hate it. I document the same way for myself as I do with claude/codex and it's great it wasn't to much of a difference to be more verbose outside maximizing tokens.
The original did not come out of a vacuum. It was done on multiple generations of meat. Even though this one uses a little bit of silicon, it is still standing on the same shoulders.
I genuinely thought this one was a satirical take on the narrow-mindedness of the aliens in the original, even though the story tries to paint humans as narrow minded. I guess this fundamental human trait to believe that their cognition is the ultimate way to think in the universe ironically leaked into all these stories as well. Real spacefaring civilisations would probably have seen all kinds of intelligence rise from sufficiently complex systems.
The weights start with a random manifold.
The training takes data and shapes the manifold, weight by weight, in many cycles.
Once the training is the done manifold is fixed.
When a new inference has to be done the query(q) is projected in the manifold space.
This projection is dropped on the manifold and the gravity of the manifold gives an answer of q+1 length.
Which(qw+i) is dropped qw+n times to output a final response of n length.
The gravity is created by repeated multiplication(of the weights/input) to find out how the projected embeddings should fall according to the manifold in the GPU.
It's like a giant plinko board where the shape of the original disc guides how it falls through the apparatus, and the apparatus has been tuned so that different discs end up in the exits we want them to
Compression is the reason why these Models are able to learn and understand.
My brain is doing the exact same thing.
I learned enough to compress concepts like a bike and what a bike does and for what i can use a bike.
Ask a LLM and it will answer you similiar to humans.
Blind people learn concepts of bikes too and in a smiliar way: by description.
LLMs just have so much data in form of text available and are able to ingest all of this, that the LLM compression algorithm doesn't has to be that good/finetuned than ours.
But I would assume that Yann LeCun's JEPA or other breakthroughs in the next few years will get us there.
The man posits that clicking is instinctual for blind people but they are told to quiet down in class and most never develop their echolocation abilities
A blind person has touched warm and hot things and gotten burned before, and then they are told lava is this molten liquid that is even hotter than anything they have touched. That is enough for them to understand.
A blind person that never touched a hot object wouldn't really know though, there is a reason we dismiss talk from people who lack experience.
You don't know that. Yo don't know what someone would think if you tell them the general concept of cold and warm.
The reaction you should have, the feeling etc.
I asked chatgpt how it would describe a scene without mentioning temperature. It was very good in describing what a human would describe.
I'm aware of the bias we have against LLMs but I think people just underestimate how much data is there.
I'm not saying a robot wouldn't be better with this information or an LLM and they actually use temperature sensors for robots so they can control movement speed and dexterity with overheating elements but the gap is small.
think of it like this: the goal of LLMs (im saying LLM as shorthand for all of the AI algs) is to replicate human output / work, not so much to capture the human experience. so in order to do things, like communicate concepts, play a role in a scene, make choices, we project our human experience down into a lower resolution / constrained space to generate an output.
the question then is whether or not LLMs can mimic the projection of human experience as well as the real thing or not. My hypothesis is no for total general replication, but in certain fixed problem spaces I think its yes and the set of these spaces is growing. will it grow enough to encompass all practical work? not sure yet
AI tools are more like an author writing a character than a human revealing truths about their experiences. the difference between the LLM and an actual author is that the LLM kind of starts off with a 'flanderized' version of the character while an author can sort of blend personal experience and character to get at real human relatable decisions for the character. the result can be indistinguishable, because if the LLM is too predictable, too flanderized, we can inject artificial randomness to it to simulate personality.
you end up with unsolvable debates like movie critics have. "This character wouldnt make that decision in that situation if they were a real person, it doesnt make sense" vs real humans making irrational decisions.
ultimately i think the human experience is something we learn about with each other as we live it, and to think were any good at identifying it from sufficiently good imposters is putting the cart before the horse. once i know something is done by an actual human, my understanding of the world can shift. we dont decide what it means to be human, we accept what humanity is and try to work with it. So the philosophical debate on AI experience is moot to me, its not human therefor it teaches me nothing about humans and can replace exactly 0 human activity for me. Still can be a useful tool though
I don’t think this analogy holds. The whole way through the processing pipeline in the brain, different sensory data is ingested separately and processed separately; and we still don’t understand how that data is then integrated into a cohesive experience.
LLMs have the same fundamental input regardless of modality, tokens. There is a preprocessing step before the “brain”, which is more akin to some super-synesthesia where all senses are translated into sound before becoming experience.
Can't you say the same about the connectivity between the brain and your senses? Your eyes do 'preprocessing', but in the end the connection to your brain is just through electrical impulses in the end. All senses get translated to some sort of electrical signal, just like in an LLM with tokens.
In what way is that different from any other model of reality that you'd use to winnow a dataset into an answer to a question? The only major difference I see is that beyond a certain number of transformations, people are willing to treat it as some sort of miracle, and too tired to figure out why it came up with the answer it came up with. It's almost like people desperately want to give up their agency and creativity to black boxes, whether those weights produce answers that are right or wrong. Factor in that psychology and it looks a lot less like we have invented something useful, and a lot more like we as a species are choosing to quit life en masse.
> The only major difference I see is that beyond a certain number of transformations, people are willing to treat it as some sort of miracle, and too tired to figure out why it came up with the answer it came up with.
It’s funny, because I thought you were talking about humans here when you wrote this. We know some things about how our bodies encode information that is sent to the brain, and we know some things about how neurons receive information and act on it, but after that we get too tired and give up on how the brain works and treat it like a miracle.
It’s like we desperately want to believe our consciousness is not just electrical impulses in our brain, and we want to ascribe agency and uniqueness to the physical processes going on in our head.
There's definitely a sizable contingent of people who desperately want to believe consciousness is just electrical impulses in our brain. Because "what else could it be"? The fact is that we just don't know, and "abiding in the not-knowing" is for many the most uncomfortable thing ever. Especially for the curious- and rational-minded people this forum tends to attract. I'm one of them, too.
It is basically reductionism, or in the extreme atomism, and (as I understand it) it has largely fallen out of fashion both in the philosophy of science and in the philosophy of mind, since it heyday in the 1970s. And I don’t see it coming back into fashion any time soon.
I can definitely see why reductionism would appeal to an educated public. We learn about the periodic table; we are able to break sentences down to words, down to letters; to break an executable down to binary code, to machine instruction, to electrical currents flowing through semiconductors. Why shouldn’t we be able to do the same with conscious thoughts? It is certainly an appealing thought process.
As I understand it, reductionism started to fall out of favor because of the rise of quantum mechanics and chaos theory, where we have a lot of weird phenomena which cannot be explained by reducing the particles down to the sub-sub atomic (or rather they are better explained by describing the interactions directly).
Quantum Mechanics and Chaos Theory don't preclude reductionism, and I wouldn't say it has fallen out of favor as a whole. Certain types certainly have, but not the overall idea.
Also, nothing about the idea of the mind only being made up of physical processes means things have to be deterministic.
It doesn’t preclude it correct, however it provides a pretty strong examples of where reductionism is lacking, which what I believe has turned a lot of philosophers of science against pure reductionism (I am probably oversimplifying here. An expert science historian [which I am not] could probably write a whole book about why reductionism is not as popular today as it was in the 1970s).
There are a whole lot more physical processes going on in our bodies then just neural activity. And my best guess is that is exactly where reductionism fails. It is possible that neural activity is a necessary but not sufficient condition for consciousness. It is also possible that we are looking in the wrong direction, that consciousness arises via interactions with the world. In either case (of which I find the former quite convincing) we will never be able to describe the mind by just looking at neural activity.
I am actually of the opinion that cognitive scientists are doing an excellent job describing the mind with our current theoretical models which excludes the tough questions of consciousness.
I feel like these are separate things... neural activity being necessary but not sufficient for consciousness does not mean reductionism is wrong, it just means the fundamental building block is not a neuron.
It might not even be possible to fully understand the physical mechanisms that underlie consciousness, but that doesn't mean there has to be something more than physical mechanisms.
Yes. Even to this reductionist, “neural activity” is insufficient to describe to consciousness in the same way that “it’s physics” is insufficient to describe how a car works.
I could put a bunch of metal and rubber and gasoline in a pile and light it on fire — all the necessary ingredients for a car — but it wouldn’t create a working automobile. The arrangement of the objects and processes matters.
In the same way, if you put a bunch of brain cells together in a Petri dish, but their connections or firings were disordered, I wouldn’t expect consciousness. “Neural activity” is thus insufficient on its own, but this I doesn’t mean reductionism is incorrect. It just means you didn’t correctly reduce the problem to the correct constituent parts. You left some out.
Reductionism is a theoretical framework. It is neither right nor wrong, Sometimes a theory based on reductionism is wrong, but reductionism it self is never wrong.
Reductionism usually includes interactions of the lower parts (unless you are an atomist; in which case go back to ancient Greece), I never denied this. However even with the interactions, reductionism is still a lacking framework to describe consciousness. If it wasn’t so lacking it would be more popular among the people who actually study the mind.
I didn’t say it was wrong, I said it was lacking and unpopular among modern philosophers of science. If you want to explain consciousness as arising from interaction with the environment (like Ted Chiang did in yesterday’s article) holism is a much better approach, same if you want to use evolutionary explanations, like Daniel Dennett did at the turn of the century.
I think reductionism is simply to limited of a philosophical framework for modern science and philosophy.
If the mind is made up only of physical processes, then the only way it could be non-deterministic is if the physical processes themselves were non-deterministic. In that case neither the mind nor the physics can be reduced to a deterministic model in any meaningful sense where the same inputs would generate the same outputs, so reductionism falls apart if you introduce non-deterministic physics as the base.
Nit: A truly educated public would not take reason (empirical evidence, denial of the magical, unseen, or superstitious) to mean that we must assert that we live in a clockwork universe or assert an explanation of the mind based on observable Newtonian physics and electrical phenomena. Confusing a clockwork model of the universe with reason, or thinking that the choice between that and superstition is binary, is actually a pre modern and uneducated way of framing the problem of how the universe works, and if it's the recourse of the "educated" shows a dangerous regression from how educated they were 50 years ago.
Maybe we do. I think it's a human tendency at large to ascribe pattern or intelligence or spirit where there is only noise. If we can't even prove our own intelligence, doesn't that reinforce the idea that we're in no position to claim intelligence has emerged by running our own intellectual output through a fixed set of weights, the training of which we also designed? At best, any such intelligence would be entirely self-refential and exposed to the question of whether we ourselves are intelligent. If your position is that we are not, then there's no way an LLM could be.
> but after that we get too tired and give up on how the brain works and treat it like a miracle.
I disagree. We know very well how neurons work, and we have a pretty good idea of how neural activity translates to behavior. In other words, we have a pretty good idea on how the brain works. We stop at consciousness because as of yet it is in the realm of philosophy, not science. We don‘t know what consciousness is or even whether or not it is useful for science and we are simply waiting for the philosophers guides us out of that situation.
Note that both cognitive psychology and behavioral psychology has done fine without tackling consciousness. When neuropsychology emerged in the 1980s it complemented both these fields perfectly. The situation is the opposite with the philosophy of mind which grew significantly around the same time.
There have been some attempts to describe consciousness as an emerging phenomena out of neural activity, but so far all of these attempts have failed, or at least failed to turn consciousness into a useful term in psychology (the way gravity is a useful term in physics). I think it is equally likely that these attempts have failed because consciousness may simply not be a useful term in psychology, that is as likely as it is that we simply don‘t understand it well enough.
Saying we have a good idea of how the brain works massively overstates the case...
We know how neurons fire. We do not know how a brain turns that into thought, meaning, intention, experience and on and on. That is not "pretty well understanding the brain", it's understanding some components and hand waving the thing we actually care about.
What I actually care about is how neural activity translates to behavior. And we have a good enough idea of that that we can design SSRI medicine to treat depression, or neurological tests to detect Alzheimer. As for experience we do know something and we are learning more with cognitive psychology, in e.g. priming experiments etc.
I feel like the search for consciousness is to psychology what the search for the Aether was for physics and chemistry. I think it is a worthwhile search, and maybe we will discover something important during that search, but we should also be prepared to find out that the thing might not exist, or it’s presumed properties are better explained with a different model.
SSRIs are not evidence that we understand how neural activity becomes behavior. They are evidence that you can perturb a system usefully without understanding it very well. That is exactly my point.
Respectfully, you are miles out of your depth here.
I don‘t see why you felt the need to insult me here. We are having a very common disagreement here, one which philosophers of science have been actively debating for several decades.
My point with the SSRI is that we know that serotonin is a chemical which incites certain neurons, and we know that a lack of activity of neurons in that general area in the brain is correlated with depression, so scientists were able to accurately predict that keeping the serotonin in that brain area for longer would increase brain activity there and decrease the level of depression.
This counts as pretty good understanding in my books at least. It teaches us very little about consciousness but my point is that it doesn’t have to. Just like Newton’s theory of gravity did not have to teach us about some deeper cosmological truth.
It's not an insult to suggest one is out of the depth on a topic, especially when it isn't one's field of expertise. You are giving the pop science explanation of various things.
> When disagreeing, please reply to the argument instead of calling names. "That is idiotic; 1 + 1 is 2, not 3" can be shortened to "1 + 1 is 2, not 3."
Just as a reader with no particular dog in the philosophical (or semantic) fight over how well we do or don't understand the brain: That rude remark lowered rather than increased my estimation of your knowledge or authority on any subject you would be discussing. Generally, people who are highly knowledgeable and confident on a subject don't resort to telling others they are out of their depth, because they don't need to. At the very least, it's suspicious to throw an ad hominem into your rebuttal.
Winning a debate is about convincing the audience, and I found that an unconvincing statement, apart from it being an obnoxious rhetorical tactic.
But it did make me think of The Big Lebowski. "You're out of your depth, Donnie!"
Some schools of the philosophy of science would argue that you do. However you are describing is a very different acquisition of knowledge then what scientists did when developing SSRI medicine. We had to:
1. take pictures of brain activity under different conditions to see which regions were active during different moods,
2. sacrifice a bunch of mice to see which neuro-chemical activated which neurons,
3. predict that inhibiting the re-uptake of a specific neuro-chemical would activate that region,
4. predict that activating that region would decrease the level of depression
In your solar example you would have discovered melanin and its relation to your skin tone, and you would have studied the effects ultra-violate radiation has on your melanin levels. Then you would have predicted that staying out of the sun will not give you a tan.
Yes, but our friend's apt analogy shows the danger of absorbing Plato's cave as the one thing you learned in Uni. If everything is a shadow on the wall then, of course, every type of study you just mentioned is merely another set of shadows. Nothing can be proven, and the coin of the realm is not to disprove anything but merely to signal your disbelief. Arguing with data for the power of reason against such a philosophy is pointless, as sincere as your response was (and I did appreciate it).
> beyond a certain number of transformations, people are willing to treat it as some sort of miracle, and too tired to figure out why it came up with the answer it came up with
It’s less about being too tired and more about being realistic about the limits of understanding.
Consider mass and energy flows in planet-scale systems: At some point we call these “weather” and change the tools with which we study them, but we never stopped trying to understand the phenomenon.
If you're going to make something smarter than a person, you got to be convinced that you're only going to be able to understand it on the single training step level and then inductively trust that the rest of it works. We do empirical testing of course with evals, but there's sort of an art to figuring out what is theoretically going to improve eval performance. Trying to fit the meaning of all those weights in your little human brain and working back from there isn't going to work for more than a little slice of the dataset at a time because that's all we can fit in our understanding.
When we attempt to recreate those complex, planetary atmospheric phenomena in a box, we're doing so in order to measure and study them.
Making random turbulence in a box until it resembles the outside world, and calling it weather and extrapolating some predictive meaning from the result, is the total antithesis of what you're describing about why we come up with simplified models for impossibly complex systems. The purpose of [mathematical] models that are built thoughtfully is to explain why complex systems are the way they are, with data and algorithms, however imperfectly. [Whereas] The purpose of LLM models is to give the illusion of answering questions while never answering why the answer was given. The difference is the difference between a scientist and a tarot card reader, an equation and an oracle.
People have a well known tendency to gravitate toward the shamanistic, oracular, and superstitous. Listen, I ran a casino for 6 years, I know. The impossibility of knowing how 80 layers of matrix multiplication led to a particular answer is in itself a psychological factor in choosing whether to accept the answer or to question it. People tend to err on the side of the over, in sports betting terms... or on the lazy side in general... and they will make whatever excuses they need to after the fact to justify their decisions. So now we have a machine that can act like an oracle and which you can also blame, but the blame goes into a void because this machine is stateless and is only a reflection of information, not an intentional refinery of data.
Sit next to a bank of slot machines for an hour and listen to the absolutely ridiculous shit most people will come up with to explain how they "know" if a machine is going to pay out soon, and then tell me if you think it's a good idea to give them an LLM in their pocket to answer their questions in whatever way they frame them.
> The purpose of [mathematical] models that are built thoughtfully is to explain why complex systems are the way they are, with data and algorithms, however imperfectly.
Nope. The main purpose of the whole endeavor is usually to predict the behavior of a complex system, because that's actually what we care about. If we can predict it, we can adapt to it, and eventually use it to our advantage.
Explaining why a complex system is the way it is, is merely nice-to-have. Models are opinions. All of them are wrong, but some are useful, and we rank them by how useful they are. The models and explanations are important because, beyond their elegance and convenience, it's also the case that more accurate models give you better predictions across larger domains, meaning we get better at getting something useful out of the complex system.
People get fixated on modern theoretical science, with bottom-up mathematical explanations traced through seas of empirical data, with whole magical rituals of peer review and double-blind studies and statistical significance around them. But they forget that the core of empirical science is literally throwing shit at a wall to see what sticks. That is the guiding principle, everything else is just making the process more efficient.
Understanding complex natural systems (or even engineered ones that got too complex) always starts with tests - tests on the real thing, then on approximate models that we poke and prod and bash into shape until they start acting similarly to the real thing. It's through the poking and bashing, and how they affect our proxy model, that we glean insights into nature of the simulated phenomena, and eventually formulate general theories - but more importantly, the models give us useful predictions from the start, before we have any theories explaining why.
I don't know - this is a highly specific interpretation of both what science is and why people choose to do it.
I'm a scientist. Believe it or not, I believe in substantially more than prediction and I think its rather trivial to come up with examples where mere prediction is insufficient to meet a normal person's notion of an account of a thing (eg, pre-copernican planetary motion). I'm not saying you are wrong, per se, just that the idea that "it was prediction all along" is a very specific idea of what human beings are interested in and what we are up to.
> that we glean insights into nature of the simulated phenomena
That is right - most people believe that there is a simulated phenomenon "out there" that we learn about. I think there are strong reasons to believe this having to do with how models are related to predictions. The wrong ontology can make prediction very hard and the right one can make prediction substantially easier. Arguably, we are in that situation right now with language models - we just threw a lot of parameters at the problem and now we are able to predict but we still don't really understand. This is perhaps inevitable in the case of language, but I don't think we should look at models with tons of degrees of freedom and the ability to predict things as a death knell for the very idea of deeper understanding.
Great post. And that's exactly where I think we are with language models... we as a civilization are hypnotized and enchanted by the overfitting of models whose parameters are beyond our understanding, but whose mistakes we are more likely to forget than its accuracies, which again is a central human characteristic that explains our attraction to both psychics and slot machines.
Heck, it even explains my own attraction to overfit sports betting algorithms. No one is immune.
What's dangerous is when something like that replaces independent thought and becomes societally pervasive. That's an "oracle" the likes of which ancient civilizations warned that believing would lead to tragedy (or at a minimum, accidentally boning your own mother).
I'm an atheist, but raised Jewish. I read the Torah as a series of specific warnings and prohibitions against every type of shamanism, magic, witchcraft, prognostication, and deification of systems which predict (as well as systems which attempt to turn language into machinery, and worship the machine they've built ... see also, "Sound of Silence" by Paul Simon and "The Future" by Leonard Cohen, which both express this theme well). The framework requiring proof and disavowing illusion or the belief that all is illusory is notably different from a Buddhist perspective, for example.
We as a culture, right now, are not handling well the rise of a golden idol or an oracle in our midst. The right response is to try to trace the output back to ground truth and figure out why your model made a prediction... or else to build a model from ground truth and see how it performs against the oracle. We are doing neither. We're diving headlong into our own confirmation biases.
[Edit] I just wanted to add, because I got off track, that your conclusion about what's going on with human curiosity in cases where prediction is not the issue seems right to me. Barring some edge cases like predicting an eclipse and using it to slaughter your enemies, I think a lot of us do simply want to understand how things work, because figuring them out is enormously gratifying and is the work of lifetimes of incredible people who came before us. Using that knowledge or those techniques to predict things is technology, not science, and while I'm a fan of both, the former is only ever a practical test of the latter. Moreover, the sense of accomplishment of randomly walking your way to a profitable model is ephemeral and in a way earthbound, limited to the plane of one's own brief existence. Even if it were platonically perfect, a model is only saying how something behaves, not how it works. That's nothing compared to the joy of figuring out even the most trivial or axiomatic thing about how a cell or a compound or a physical structure or anything works, about how the universe actually works. And I think our better angels tell us to seek those answers, because our own life is fleeting, and predicting behavior is, like wealth, something you can't take with you. And not something you'll be remembered for anyway.
I think we're talking about different kinds of models. I was referring to things like fluid dynamics equations that explain why gases and liquids move and how they act when changing states, as a basis for building weather models that predict how things will unfold in the future.
I'm also a fan of going the other direction: I've had a sideline working on code to evolve genetic algorithms for the past 20 years, and while the goal of that is to be predictive and profitable, it's often the underlying real-world dynamics my little mutants surface which are the most interesting and applicable in the long run. So I'm not saying there isn't a place for throwing everything at the wall until you see what sticks and then deriving a hypothesis from that (whether your interest is to predict the future, or merely academic, to explain the past). What I am saying is similar to you: We should not treat any model as an oracle. But I'm also saying that models can be built or they can be evolved, and if we only evolve them without understanding how they work, we are missing a crucial ingredient to knowing how well we should rank them. Overfitting and sample bias and data leakage are not problems when you want an equation to calculate airflow over a wing. If you began with an evolved equation which derived the results and didn't start from the base reality, you couldn't trust that equation to be airworthy even if it were right 99.99% of the time against the data it was trained on.
> The purpose of LLM models is to give the illusion of answering questions while never answering why the answer was given.
This is just your own idiosyncratic and biased belief. You're not describing anything objective about LLMs, you're describing your personal attitude to them. This colors your understanding in a way that can't really be reasoned with until you let go of the artificial constraints you're imposing on your own understanding.
> Sit next to a bank of slot machines for an hour and listen to the absolutely ridiculous shit most people will come up with to explain how they "know" if a machine is going to pay out soon, and then tell me if you think it's a good idea to give them an LLM in their pocket to answer their questions in whatever way they frame them.
If the LLM in their pocket has a more robust world model than they themselves and is e.g. able to refute their irrational convictions, it actually seems like a very good idea. (Big if, of course.)
I actually wish I could upvote this more. It's a great point.
Yes, if the LLMs didn't amplify whatever people already thought and feed it back to them as sycophantic praise, and instead scolded them into realizing they were doing something dumb, then maybe we would be having a totally different conversation.
But then the conversation might be about an LLM scolding someone into committing suicide instead of helping them commit suicide.
I want freedom to do what i want and not sitting in front of a computer and coding for some company.
Please AI lets burn down knowledge work and labor work. Lets create so much stress to our society that we start rethinking what works mean.
Lets redefine work into discovering the world again. Let people do old handcraft jobs, let them do more sports, let them read more, let them write and make more. Let them enjoy nature.
Work has never been about "discovering the world". There have been a handful of privileged folks who had the time to "discover the world". Work has traditionally been "let's find enough food for my family". If you want to think of a future of abundance then perhaps we can discover the world.
It’s not a capitalism problem, it’s an ecosystem. Wherever living things compete for finite resources and opportunities, certain properties emerge from the system. And make no mistake we live in a crowded world full of fierce competition.
Among these properties are optimum behaviours such as hoarding the maximum you can defend (rather than the minimum you comfortably need) and using your power to forestall the growth of others. These behaviours are repeated in nature from microorganisms to apes to humans.
There is no social order which can prevent us from living in an ecosystem, and from these properties and behaviours emerging.
The US American system is more capitalistic than the german one and the scissor between poor and rich is higher.
Thats definitly an indication that this plays a role in it.
And we do not have a problem of resources today, we have so much food, that its sometimes just spoils and no one cares and sometimes it even spoils in huge warehouses due to neglect or because a person decided not to sell it under a certain price.
Capitalism also sets priorities for resources. So instead of making sure everyone is fed properly, we also spend time and energy to diverge resource types like different plants, exotic plants etc.
> The US American system is more capitalistic than the german one and the scissor between poor and rich is higher.
And yet the US is a dominant player and Germany is not.
> And we do not have a problem of resources today, we have so much food
I never said there needs to be scarcity for an ecosystem to emerge. These properties emerge wherever living things compete for finite resources. Imagine a pond where you introduce a few small catfish. At first, food is plentiful. Yet the fish will grow and reproduce until it is scare. It's the same with resources. Individuals/companies/societies will grow to consume the maximum they can sustain, because this is the optimum behaviour in an ecosystem.
The point is that organisms which avoid these behaviours tend to lose, in every sense of the word, to organisms that do. The behaviours are emergent from the constraints of the system.
> that its sometimes just spoils and no one cares
Transportation costs are a major driver of food shortage. It's not at all free to get large quantities of food from where they are produced to where they are consumed. People are generally not willing to transport food (or do much of anything else) at a loss.
This seems to be a little naive about how humans consume the benefits we create in society.
"Let people do old handcraft jobs, let them do more sports, let them read more, let them write and make more. Let them enjoy nature."
Very nice thoughts. You know we all could do this today without "burning it down"? Get in your pod, eat your slop, and watch your screen is where this is headed.
"I want freedom to do what i want and not sitting in front of a computer and coding for some company."
You get that it's you creating the misery here? Then stop? Don't do it. Go start a farm or whatever you think will solve your problems. At some point this all boils down to "chop wood and fetch water" so if the modern way of doing that is so terrible then stop. Go fetch water the old fashioned way and be free.
> Lets redefine work into discovering the world again. Let people do old handcraft jobs, let them do more sports, let them read more, let them write and make more. Let them enjoy nature.
Why leave something so important up to what AI does or doesn't do?
"I want freedom to do what i want and not sitting in front of a computer and coding for some company."
"Please AI lets burn down knowledge work and labor work"
"Let people do old handcraft jobs."
So many presuppositions about what people want to do.
As a child I spent a lot of time programming and doing "knowledge work" because it's fun - I don't enjoy "old hand-crafted jobs".
Sure, let's definitely destroy capitalism in it's current state I suppose. But I find people like you who hate knowledge-work/coding and think everyone else must feel the same and only do it for the money a bit out-of-touch.
right, these knowledge work and coding jobs are, by my lights, about the best possible job. From my perspective we've invented a machine that does the fun parts while leaving me the less fun parts (review, various hard-to-claude janitorial tasks, etc).
I might like woodworking as a hobby (for example), but I sure as heck don't want to be a carpenter or to depend on my ability to hand craft enough widgets people like to survive
Be more critical, we do our jobs because of capitalism, not because every single company needs an hr department or the next crud application or the 1000th webshop.
You have in several places repeated that "no one is working on the field to get food", but that's not actually true. While most people in modern society don't work on fields to feed themselves (and others), a small fraction actually do! They feed not only themselves, but all of us.
In this future utopia where everyone gets to be a mediocre handmade-furniture maker, who exactly will be the 1-2% needed to work the fields to feed us all?
Or the 1% of people will be allowed to live in the city centers or on the beach or we share the load across peple. Everyone has to work hard and good for 5 years.
who's choosing this lucky 1%? Who's choosing what this 5 years of "hard and good" work looks like? Is 5 years of work a person really enough? Are you aware that farming is predominantly done by machine, and that's why we're down to so few people working in it?
Sure, the idea of a life of leisure and choice sounds great, sign me up. But I think resources are not distributed evenly, the folks with the most power distribute resources have little inclination to distribute them evenly, and even if we did distribute resources as evenly as possible we would still have scarcity, as with your city and beach examples. We will still need people to deal with toilets, to deal with food and so on.
If we've invented the magical cybersyn dream, and we can have central planning done for us, so everything is efficiently allocated and automated, how can you be so sure your personal allocation will be leisure and not ditch digging or bum wiping? I will bet a jelly donut that what you have described will not come to pass in my (or your) lifetime.
The solution we've come up with is move all the unpleasant work stuff to China where people don't complain about doing it because they already have communism, and therefore everything is of course effortlessly perfect there.
> The only major difference I see is that beyond a certain number of transformations, people are willing to treat it as some sort of miracle
Like with every invention ever? Cause that's the literal goal and idea?
You take things and then combine them until the ensemble performs a desired abstract function the individual parts alone could not. The end result then is a new thing of its own, arguably indeed a miracle (not in a religious sense of the word).
> and too tired to figure out why it came up with the answer it came up with
There are people whose entire career is this. They work at these companies.
> Factor in that psychology and it looks a lot less like we have invented something useful, and a lot more like we as a species are choosing to quit life en masse.
You're saying this as if people haven't been historically the masters of optimizing out the enjoyment from things. It's what we do. Provide an ultimate solution, and of course you'll extinguish a whole lot of motivation across the board.
It would be interesting (if unlikely) that we found our reality worked that was as well. Mostly just super granular gravity nodes made up purely of clumpy probability distributions.
Also I describe this more simply to others as a tree walk where the manifold is mapped along a walk through it (even if it’s not linear one) where you choose the next jump based on the most likely future mode relative to the nodes already created and the ones from the prompt.
This helps some understand attention layers rudimentary. Even tho it leaves out that multiple layers sort of prune the overall manifold in successive passes.
The data is the code is the data. Reality has no distinction between "data" and "code". These terms are categories we impose on systems we design, to make it easier for us to build and reason about them, but they're nothing but mere opinions, and depend less on the system structure, and more on the perspective of the person asking which is which.
This is related to, and possibly equivalent with, the core point of both this story and the original one: computation is independent from substrate.
You can build a computer out of anything, whether it's semiconductors or lasers or meat or magnetic fields or water flowing downhill or abstract thought, and that computer will happily perform the same computation as every other equivalent construct from whatever substrate. That's because computers are ultimately made of math, and we design "real ones" by finding ways to approximate the mathematical constraints with physical systems. But the choice of how to map the math to physical systems is completely arbitrary, and any such mappings are equivalent from POV of information processing ability.
(Of course substrate is not arbitrary from economic POV, which is why we build most of our computers out of silicon and plastic, and make it work with electric current and lasers.)
One of the best thing I done for my career (as a self taught software developer, but with a degree in electronic engineering) is to learn computation theory.
Computation is math (and a very restricted subset of math). It’s mostly specific sequences of sets manipulation. What sets and what manipulations are defined by people, not by the idea of computation.
The best thing is that as soon as you specify the sequences of manipulation, it become a a set that you can manipulate. That can be a difficult concept to grasp, but that’s what helps in designing notation that are more appropriate for the human mind to describe a solution for a specific problem.
That's why encountering something like LISP for the first time (by writing a LISP interpreter, for example) creates a big bang event in form of an imminent intellectual catharsis. People who encountered it just once, will never be able to see the world through the old "meaty" lenses afterwards.
This is a good model. If you take an old ROM dump from a video game, it's just a pile of bits. You don't know what bits represent code, what represent an image, what represent text, etc. You have to analyze them contextually to actually figure out what is code and what is "data" in context, because without context they are truly one and the same.
> a ray of light has to know where it will ultimately end up before it can choose the direction to begin moving in
A ray of light doesn't know or choose because it has no agency, just like an apple doesn't know or decide to fall because of gravity. It's an anthropomorphization.
> a ray of light has to know where it will ultimately end up before it can choose the direction to begin moving in
I'm no physicists, so I guess I'll ask it: Why?
Also related, why do some ray of light then "see" a black hole yet decide to head into them anyways, if they saw it before they went in that direction? Seems like a dumb move :)
Its future isn't over there because it moves in that direction, instead it moves in that direction because its future lies over there.
Relatedly:
> [General Relativity] basically says that the reason you are sticking to the floor right now is that the shortest distance between today and tomorrow is through the center of the Earth.
Yes, yes, but what fertile fallacies and common misunderstanding can politicians use to acquire more power via exploiting the difference between the common person's flawed understanding due to cargo culting, cognitive biases, and/or outdated or inappropriate analogies vs actual reality? Is there any way we can get the AI to say give all political power to narrator is the solution to all problems and use the common person's mistaken worship of AI as a spiritual all knowing conscious being with unusual sensitivity and caring about everyone to cement that power? Certainly one of you eggheads can tweak that for me? What? It's against your ethics? We're trying to save the world here. Here, let me call up Bernie Sanders to propose nationalizing half your companies so we can do that.
so can we all agree that LLM models/agents are bad at BFS for exploring a problem space but are good at DFS to implement a solution if the context/requirements are rich enough.
The problem is because of the RL and system prompts by the providers which tend to placate the user using certain language tones and register for response. This objectively messes up the generation while steering it into acceptable responses.
Most of the conversational skill and perceived intelligence of these models in hidden in RL/system prompts.
But the general use case is not the most efficient for a greenfield to-be fully managed by an agentic system code-base. It is built to be good around the scaffold(programming like humans) and not the actual problem space.
Anthropic's target should be a codebase designed for agentic comprehension from the first commit. Here the codebase adapts to the agent. You can enforce conventions, structured metadata, semantic indexing, explicit dependency graphs. Whatever makes the agent's job trivial rather than heroic.
The large majority of coding is maintenance work, not greenfield development. Even if you are doing greenfield development, it won't be long before it is maintenance.
But once a human learns a function their errors are more predictable. And they can predict their own error before an operation and escalate or seek outside review/advice.
For e.g. ask any model "which class of problems and domains do you have a high error rate in?".
Finance, Law, VC guys were good too in the beginning but when the value/status change happens it attracts certain kind of guys who are average in talent but excel in demonstrating value and social management of the value/status.
Another change which has happened recently is that the economics of engagement farming have become common place wisdom as already proven effective for everything from selling books, personal brand, career skill/virtue signalling, staying relevant.
Due to this everyone is talking more without restraint and not keeping in their own lane of earned expertise.
reply