In all honesty, this is one of the less abused quotes, and I have seen more benefit from it than harm.
Like you, I've seen people produce a lot of slow code, but it's mostly been from people who would have a really hard time writing faster code that's less wrong.
I hate slow software, but I'd pick it anytime over bogus software. Also, generally, it's easier to fix performance problems than incorrect behavior, especially so when the error has created data that's stored somewhere we might not have access to. But even more so, when the harm has reached the real world.
User-facing, sure, nothing stopping us from doing "simple and fast" software. But when it comes to the code, design and architecture, "simple" is often at odds with "fast", and also "secure". Once you need something to be fast and secure, it often leads to a less simple design, because now you care about more things, it's kind of hard to avoid.
IME doing application servers and firmware my whole career, simple and fast are usually the same thing, and "simple secure" is usually better security posture than "complex secure".
Interesting, never done firmware, but plenty of backends and frontends. Besides the whole "do less and things get faster", I can't think of a single case where "simple" and "fast" is the same thing.
And I'd agree that "simple secure" is better than "complex secure" but you're kind of side-stepping what I said, what about "not secure at all", wouldn't that lead to simpler code? Usually does for me, especially if you have to pile it on top of something that is already not so secure, but even when taking it into account when designing from ground up.
Not really. `return 0` is the simplest program you could write, but it's not terribly useful. There's an underlying assumption that there's some purpose/requirement for the program to exist. Through that lens "secure" is just assumed as a requirement, and the simplest way to meet your requirements will usually still give you the fastest program too.
"Do less and things get faster" is a very wide class of fixes. e.g. you could do tons of per-packet decision making millions of times per second for routing and security policies, or you could realize the answer changes slowly in time, and move that to upfront work, separating your control vs data processing, and generally making it easier to understand. Or you could build your logic into your addressing/subnets and turn it into a simple mask and small table lookup. So your entire logic gets boiled down to a table (incidentally why I can't understand why people say ipv6 is complex. Try using ipv4! Having more bits for addresses is awesome!).
Sort of. But if you keep the software simple, then it is easier to optimize the bottlenecks. You don't really need to make everything complicated to make it faster, just a few well selected places need to be refactored.
Same. I, too, am sick of bloated code. But I use the quote as a reminder to myself: "look, the fact that you could spend the rest of the workday making this function run in linear instead of quadratic time doesn't mean you should – you have so many other tasks to tackle that it's better that you leave the suboptimal-but-obviously-correct implementation of this one little piece as-is for now, and return to it later if you need to".
That is a great example of when the original quote works as intended.
I'm reacting to experiences where the software that emerges from a relatively large team effort can't really be made meaningfully faster because there are millions of tiny performance cuts - from the root to the fruit.
/* If we can cache this partial result, and guarantee that the cache stays coherent across updates, then average response time will converge on O(log N) instead of O(N). But first make the response pass all the unit tests today */
Still halfway through reading, but what you've made can unlock a lot of use cases.
> I tried SQLite first, but its extension API is limited and write performance with custom storage was painfully slow
For many use cases, write performance does not matter much. Other than the initial import, in many cases we don't change text that fast. But the simpler logistics of having a sqlite database, with the dual (git+SQL) access to text is huge.
That said, for the specific use case I have in mind, postgres is perfectly fine
SQLite is fine right up until you want concurrent writers. Once you need multiple users, cross-host access, or anything that looks like shared infra instead of a local cache, the file-locking model stops being cute and starts setting the rules for the whole design. For collaborative versioning, Postgres makes more sense.
For a distributed VCS, what would be the need for such things? Even if it were a really big project, how many writes could be going on that this becomes a bottleneck? I don't see it but maybe you have a situation in mind.
The single-file simplicity of SQLite is a huge win for self-hosted apps. I've been using SQLite in WAL mode for a single-user app and it handles concurrent reads from the API while background workers write without issues. Backup is just cp. For anything that doesn't need multi-user concurrent writes, it's hard to justify the operational overhead of Postgres.
ko
Yeah, I get that, and I'm fully on your side. SQLite would have been a nice fit. The only downside is the delta compression problem. Creating an extension for SQLite works, but it's slow. I had two options:
1) Do the delta compression and caching and so on on the pgit side and lose SQL queryability (or I need to do my own), or
We are dealing with a hype, but the reality is that AI would change everything we do. Local models will start being helpful in [more] unobtrusive ways. Machines with decent local NPUs would be usable for longer before they feel too slow.
> the reality is that AI would change everything we do
Your true believer convictions don't matter here. Those AI accelerators are merely just marketing stunts. They won't help your local inference because they are not general purpose enough for that, they are too weak to be impactful, most people won't ever run local inference because it sucks and is a resource hog most can't afford, and it goes against the interests of those scammy unprofitable corporations who are selling us LLMs as AI as the silver bullet to every problem and got us there in the first place (they are already successful in that, by making computing unaffordable). There's little to no economical and functional meaning to those NPUs.
> most people won't ever run local inference because it sucks and is a resource hog most can't afford
a) Local inference for chats sucks. Using LLMs for chatting is stupid though.
b) Local inference is cheap if you're not selling a general-purpose chatbot.
There's lots of fun stuff you can get with a local LLM that previously wasn't economically possible.
Two big ones are gaming (for example, text adventure games or complex board games like Magic the Gathering) and office automation (word processors, excel tables).
It surprises me that semantic search never gets mentioned here.
If you can use the NPU to process embeddings quickly, you get some incredible functionality — from photo search by subject to near match email search.
For consumer applications that’s what I’m most excited for. It takes something that used to require large teams, data, and bespoke models into commodity that any app can use.
> Ask your friends or a small business owner if they are going to spend $1k on a new laptop because "there's lots of fun stuff".
Do people not buy gaming PCs and game consoles? Isn't that buying something because "there's lots of fun stuff?"
And while sure a business owner wouldn't be buying it for "fun stuff", if it was about being able to run the AI tools they want without the business risk of sending your most important data and intellectual property to an AI provider wouldn't some think about it?
Are you kidding? A good ratio of ST folks run finetunes of Mistral Nemo (if it tells you anything). Anyway your core statement is simply wrong ("local chat sucks").
> If you intend to do LLM inference on your local machine, we recommend a 3000-series NVIDIA graphics card with at least 6GB of VRAM, but actual requirements may vary depending on the model and backend you choose to use.
Also, please be respectful when discussing technical matters.
> we recommend a 3000-series NVIDIA graphics card with at least 6GB of VRAM
...which is not by any means a powerful GPU, and besides the AMD Ryzen AI CPUs in question have a plenty enough capacity to run local LLMs esp. MoE ones; with 3b active MoE parameters miniPC equipped with these CPUs dramatically outperform any "3000-series NVIDIA graphics card with at least 6GB of VRAM".
> please be respectful when discussing technical matters.
That is more applicable to your inappropriately righteous attitude than to mine.
> You have fallen headfirst into the "Not now, so never" fallacy.
Perhaps. Though we have empirical evidence of how much we can quantize and distillate models to the point of practical uselessness. That sets a bar for how large a local model needs to be for general-use as to compete with the could ones. We are talking in the area of 60GB for GPT-OSS/Qwen3.5, which is what enthusiasts are running on 32GB DDR5 + 24GB VRAM RTX 3090.
> As if consumer hardware won't get more powerful
Now I will let you, with that last fact in hand, plot a chart of how much it's been costing to provision that over the past 2 years and use it to prove me wrong about the affordability of local models.
> Parent comment is fair and technically accurate.
In what way precisely? That local LLMs "suck"? Is that a technical argument?
Or this statement "there's little to no economical and functional meaning to those NPUs." - is that actual factual statement or a emotionally charged verbal flatulence? and what "they won't help your local inference because they are not general purpose enough for that" even means? People succesfully run largeish MoE llms on AMD Ryzen AI miniPCs.
> Do you have a real argument, especially a technical one, that you can contribute?
What kind of argument do you want me to "contribute" wrt the ideological rant the "parent comment" had managed to produce?
Hey, OP here with their (apparently) controversial views. I stand firmly within those lines:
- I shouldn't be paying more for my next CPU because it has a NPU that I won't ever use. Give me the freedom of choice.
- Given that freedom of choice, it would seem that a majority would opt-out (as seen recently by Dell), so the morals of all that are dubious.
- NPUs may not be completely stupid as a concept, in theory, but at this point in time they are proprietary black-boxes purpose-built for marketing and micro-benchmarks. Give me something more general-purpose and open, and I will change my mind
- …but the problem is, you can only build so much general-purpose computing in bespoke processor. That's kind of its defining trait. So I won't hold my breath.
- Re: local-inference for the masses, putting aside the NPU shortcomings from above: how large do you think a LLM needs to be so it's deemed useful by your average laptop user? How would the inference story be like, in your honest opinion (in terms of downloading the model, loading it in memory, roundtrip times)? And how often would the user realistically want to suffer through all that, versus, just hopping to ${favorlite_llm.ai} from their browser?
Anyhow, if that makes me "antiai", please, sign me up!
> I shouldn't be paying more for my next CPU because it has a NPU that I won't ever use. Give me the freedom of choice.
There is a plenty to choose from.
> - NPUs may not be completely stupid as a concept, in theory, but at this point in time they are proprietary black-boxes purpose-built for marketing and micro-benchmarks. Give me something more general-purpose and open, and I will change my mind
In fact the linked article is not talking about NPUs in particular, but about Ryzen AI cpus. These have unified memory and more compute compared to normal ones which make them very useful for inference.
> how large do you think a LLM needs to be so it's deemed useful by your average laptop user?
Depend what they need it for. Useful autcomplete in IDE starts at around 4b weights.
> loading it in memory
Happens only once, usually takes around 10sec.
> roundtrip times
Negligible? it is loca after all.
> And how often would the user realistically want to suffer
For some people maybe. I don't want to use local AI and NPU will be dead weight for me. Can't imagine a single task in my workflow that would benefit from AI.
It's similar to performance/effiency cores. I don't need power efficiency and I'd actually buy CPU that doesn't make that distinction.
> Can't imagine a single task in my workflow that would benefit from AI.
You don't do anything involving realtime image, video, or sound processing? You don't want ML-powered denoising and other enhancements for your webcam, live captions/transcription for video, OCR allowing you to select and copy text out of any image, object and face recognition for your photo library enabling semantic search? I can agree that local LLMs aren't for everybody—especially the kind of models you can fit on a consumer machine that isn't very high-end—but NPUs aren't really meant for LLMs, anyways, and there are still other kinds of ML tasks.
> It's similar to performance/effiency cores. I don't need power efficiency and I'd actually buy CPU that doesn't make that distinction.
Do you insist that your CPU cores must be completely homogeneous? AMD, Intel, Qualcomm and Apple are all making at least some processors where the smaller CPU cores aren't optimized for power efficiency so much as maximizing total multi-core throughput with the available die area. It's a pretty straightforward consequence of Amdahl's Law that only a few of your CPU cores need the absolute highest single-thread performance, and if you have the option of replacing the rest with a significantly larger number of smaller cores that individually have most of the performance of the larger cores, you'll come out ahead.
* You don't do anything involving realtime image, video, or sound processing?
I don't
* You don't want ML-powered denoising and other enhancements for your webcam,
Maybe, but I don't care. The webcam is good or bad as it stands.
* live captions/transcription for video,
YouTube has them. I don't need it for live calls.
* OCR allowing you to select and copy text out of any image,
Maybe
* object and face recognition for your photo library enabling semantic search?
Maybe but I think that most people have their photo library on a cloud service that does AI in the cloud. My photos are on a SSD attached to a little single board ARM machine at home, so no AI.
What I would like to be able to do is running the latest Sonnet locally.
In general I think that every provider will do their best to centralize AI in their servers, much like Adobe did for their suite and Microsoft did for Office so local AI will be marginal, maybe OCR, maybe not even blurring the room behind my back (the server could do it.)
There are alternatives to Adobe and Office, because I don't care about more than the very basic functionality: running Gimp and Libreoffice on my laptop costs zero. How much would it cost to run Sonnet with the same performances of Claude's free tier? Start with a new machine and add every piece of hardware. I bet that's not a trivial amount of money.
None of what I listed was in any way specific to "content creators". They're not the only ones who participate in video calls or take photos.
And on the platforms that have a NPU with a usable programming model and good vendor support, the NPU absolutely does get used for those tasks. More fragmented platforms like Windows PCs are least likely to make good use of their NPUs, but it's still common to see laptop OEMs shipping the right software components to get some of those tasks running on the NPU. (And Microsoft does still seem to want to promote that; their AI PC branding efforts aren't pure marketing BS.)
The issue is that the consumer strongly associates "AI" with LLMs specifically. The fact that machine learning is used to blur your background in a video call, for example, is irrelevant to the consumer and isn't thought of as AI.
Never wanted to do high quality voice recognition? No need for face/object detection in near instant speed for your photos, embedding based indexing and RAG for your local documents with free text search where synonyms also work? All locally, real-time, with minimal energy use.
That is fine. Most ordinary users can benefit from these very basic use cases which can be accelerated.
Guess people also said this for video encoding acceleration, and now they use it on a daily basis for video conferencing, for example.
Those usecases are at least 5 years if not 10 years out. They require software support which won't come until a significant part of the pc market has the necessary hardware for it. Until then, paying extra for the hardware is foolish.
This will only come if Windows 12 requires a TPU and most of the old hardware is decommissioned.
Also similar to GPU + CPU on the same die, yet here we are. In a sense, AI is already in every x86 CPU for many years, and you already benefit from using it locally (branch prediction in modern processors is ML-based).
> Also similar to GPU + CPU on the same die, yet here we are.
I think the overall trend is now moving somewhat away from having the CPU and GPU on one die. Intel's been splitting things up into several chiplets for most of their recent generations of processors, AMD's desktop processors have been putting the iGPU on a different die than the CPU cores for both of the generations that have an iGPU, their high-end mobile part does the same, even NVIDIA has done it that way.
Where we still see monolithic SoCs as a single die is mostly smaller, low-power parts used in devices that wouldn't have the power budget for a discrete GPU. But as this article shows, sometimes those mobile parts get packaged for a desktop socket to fill a hole in the product line without designing an entirely new piece of silicon.
Your comment is almost completely irrelevant to what the parent is saying. "AI would change everything we do" has nothing to do with "This new chip along with bloat from Windows enables new workflows for you". If you have been paying attention, you'd know that NPUs from these new CPUs barely made any difference from a consumer's perspective.
I am bullish on AI being used in all sorts of useful and discreet and non-discreet ways in the present and future. However I am exceedingly skeptical of NPUs being some winning bet.
No one is running LLMs on current gen NPUs so if we will in the future its a long time coming. Unless they can demonstrate some real (and not marketing) wins I remain skeptical that a large NPU for LLMs is the future.
I can totally see NPU accelerating simple tasks, but to be worth the silicon they have a ways to go imo.
99% of people don't need or want a dev workstation. My travel laptop is 7+ years old and I couldn't tell you the difference between it and a current flagship in terms of browsing and everyday tasks.
I will not lie, I find LLMs useful but the desktop experience is pretty polished already. NPUs seem to be an attempt to ride the AI bandwagon with very little to show for it so far.
I see a lot of people put so much stock in a future of local models, but I am not convinced we’re on track for a world where most people are running local models. Especially when you have companies like Microsoft explicitly trying to make a world people don’t even own their software/hardware anymore and do everything on the cloud.
So I’ve got a lot warmer to believing that AI can be a better programmer than most programmers these days. That is a low bar :). The current approach to AI can definitely change how effective a programmer is: but then it is up to the market to decide if we need so many programmers. The talk about how each company is going to keep all the existing programmers and just expect productivity multipliers is just what execs are currently telling programmers; that might change when the same is execs are talking to shareholders etc.
But does this extrapolate to the current way of doing AI being in normal life in a good way that ends up being popular? The way Microsoft etc is trying to put AI in everything is kinda saying no it isn’t actually what users want.
I’d like voice control in my PC or phone. That’s a use for these NPUs. But I imagine it is like AR- what we all want until it arrives and it’s meh.
People are not uniform fungible programmers. I want someone who is a specialist and above average at the stuff I'm trying to do. If that specialization commands a higher market rate than average, then yes, that's what they get paid.
In my team, I've been working to help everyone do a task that's necessary, but because it was too difficult, people bypassed it. Over time, I made it simpler, others are joining to make it even simpler, but in the process, I'm not doing as many "feature tasks" as I could. I joke that people are mad at me every day, but grateful every week and month.
I had to stop trying to prove myself to the company. I have already done that when y'all interviewed me. Now I do what's best for everyone, and I want the company to prove to me that it deserves people who do the right thing despite the processes not valuing it. If it does not, I have enough resources to spend some time on the projects I cared about most.
This mentality gave me peace of mind and helped many people in partner teams go faster and with higher quality.
Management still does not openly appreciate it, but it shows in how they interact with me. Like when you learn to talk to your parents as equals. It's unexpected for them, but they quickly embrace the new interaction and they love it much more than the one before.
My iPad wins by a large margin the lowest bang for bucks of any competing device I ever purchased
A locked down device with no multitasking, even the browser download is interrupted if I switch away.
The true embodiment of paternalism, it protects you from all the scam in the world, except the noble ones who are willing to pay the 30% share.
It has some multitasking on the latest versions of iOS. You can drag a YouTube window up by the bottom right corner and open a Reddit window on another part or the screen if you truly need to be distracted from distraction by distraction.
Like you, I've seen people produce a lot of slow code, but it's mostly been from people who would have a really hard time writing faster code that's less wrong.
I hate slow software, but I'd pick it anytime over bogus software. Also, generally, it's easier to fix performance problems than incorrect behavior, especially so when the error has created data that's stored somewhere we might not have access to. But even more so, when the harm has reached the real world.
reply