I can understand the joy of running things yourself, and can also see the privacy aspect. However, I pay ~3$ per 1M/tokens for that model on Openrouter, and it's not even quantized. A refurbished 3090 and a 5080 will set you back well over 2k, not to mention the electricity to run them...
> I pay ~3$ per 1M/tokens for that model on Openrouter
I think the thing is, there's an unspoken "for now" at the end of that sentence and people running this locally are hedging against that "for now". Some people prefer to feel that they own the means rather than rent the means, even if the one they own is worse than the one they can rent. Especially with today's Fable news and the harsh realisation that the "for now" is dependent on very many unpredictable factors, where the one you have locally costs you capital today and a relatively predictable run-rate (made more predictable with on-prem solar for example), but should otherwise work predictably forever.
I'm not saying that you're wrong to do what you're doing, just that many people have their own lines in the sand where renting vs buying makes sense, and it doesn't only boil down to a rational (or irrational) financial decision.
You're treating open weight inference providers the same as proprietary ones. They're fundamentally different business models. Proprietary companies have an incentive to subsidize actual inference and training costs in order to gain market share. The few dozen or so companies selling Qwen models by the token on openrouter are in a commodities market.
If suddenly the CCP declared a total digital embargo on Alibaba's Qwen models or even if for some reason all of mainland China (and Singapore) was completely unreachable from the rest of the world, the dozen or so companies selling Qwen by the token elsewhere in the world could continue business as usual.
I don’t know anything about the open weight host business model. Do we know for certain that the folks selling inference by the token are really selling them in an upfront and profitable way? No subsidies from harvesting the info, to sell to the model trainers or anything like that?
Or subsidies from hopeful investors sweet-talked into not understanding the commodity nature of the business they are investing in. But that does not change much about the general assessment.
Chances are the typical story goes founders start fully believing that they would succeed with their own innovation but slip down a gradient towards commodity provider without really noticing themselves.
I was thinking of user-side regulations as well, not only provider-side ones. I could imagine a world where a government rules that you may not use LLMs for anything, which would be much easier to get around if you have local means.
I've spent the past week trying to scheme a way to get affordable local inference of something useful (Qwen3.6-36B-A3B) for ~$500 and have come to the conclusion that it simply isn't viable. A pair of power-restricted P100s in a workstation gets close but the workstations themselves are expensive and rare as hen's teeth (not to mention loud and large). I think early '27 will be when things open up as the hardware market unclenches and further strides are made in small capable models.
I'm running Qwen3.6-35B-A3B on a very ordinary desktop PC (32GB DDR5, 8GB Radeon 6600XT) and getting a useful 15-20 tok/sec out of it. The MoE architecture and auto offloading from system to VRAM is just fantastic. Unsloth Q4_K_XL.
The Qwen3.6-27B is unbearably slow as it doesn't fit in VRAM, though, i think the MoE is very easy to run.
It is also extremely nice that you can just `apt install llama.cpp libggml0-backend-vulkan` now too.
I wonder what parent poster means with „useful” and what he actually tried? Feels like he was just comparing some benchmarks.
Yesterday I downloaded Gemma4-26B with Ollama on quite rusty desktop with 1070 8gb and 32gb of ram and Core i5-9400.
I drop photo of my water meter and tell it to read the value and serial number. It was far from instant but it was also easily under 3 minutes and result was correct.
Earlier like in February I was trying the same photo with Gemma3 on the same hardware and results were bad.
An R9700 is $1350 and can get 100 TPS running Qwen3.6-35B-A3B Q5 with 130k context window (with room to spare) with a bit of fine tuning llamacpp-vulkan, but llamacpp's repository instability and lack of real versioning frustrates me.
In terms of electricity, if you aren't using it, even with all the vram loaded, at most your wasting about 30 watts or so.
Prompt processing a large uncached context is annoying, which is why I forced a lower context window, but I don't know if it's any worse in performance than the cloud models I've used.
There's a niceness, to me, knowing I don't have to rent it anymore. If you rent it, the terms can change regularly.
I use local models to explore, hosted models to refine. I somewhat envy those who can sustain local models (q8 120b+) running as a hobby.... for me, the practical path is a better SearXNG setup and knowing my routes forward.
I think it's important to be able to do both so you can stay in control of the price to value created relationship.
In last year, some people were publishing aider /ollama/open router [1] and now thankfully people are publishing all around about pi/qwen/llama.cpp/openrouter. It's widespread.
It’s a personal hobby project why should we care this is how someone chooses to spend their free time and money? Lots of hobbies are expensive and pointless if you think of commercially available offerings. That’s why it’s a hobby and not a small business
Any sane crypto miner undervolted and underclocked their GPUs for efficiency's sake; if anything, they went through less wear than, say, regular gaming.
Openrouter doesn't give you access to the models internals, i.e. complete control of logprobs, sampler stack, any PeFTs.
Openrouter fking sucks and I don't know why people here act like it's so great. Stop using it if you care about local AI and accept that the cost you'll pay for tokens is higher than you will when consumed via any cloud. That's the price for privacy, control, and better quality via inference time optimizations that otherwise aren't available.
> Openrouter doesn't give you access to the models internals, i.e. complete control of logprobs, sampler stack, any PeFTs.
Openrouter gives you access to whatever the inference provider gives. They're just the middleman. Many providers give logprobs if you ask, it's in their API. And yeah, no Peft or Lora, but that's an entirely different product. And some of the inference providers do that directly.
> Openrouter fking sucks and I don't know why people here act like it's so great. Stop using it if you care about local AI
But the whole point of openrouter is that you can run models by the token and you don't have to care about local AI? Sounds like you're more upset that people aren't making the same calculation on privacy and local control vs cost and ease of use.
That engineer went on to create Brave, a browser that pays you Monopoly money for watching ads, injected affiliate links, installed their commercial VPN without asking, and leaked DNS traffic when using Tor in its "privacy" mode. I'd say Mozilla dodged a bullet there.
I wouldn't say these are "basic bugs". The first is specific to using 'rrsync', and the second is when using the rsync daemon, and I can't remember when I last saw a system using that one (yes, I'm aware there are still use cases for using the rsync protocol, but I would consider it pretty obscure nowadays).
You could argue that he should've bumped the version more and should've done a longer beta test, but on the other hand, these were mostly security fixes, and I can understand he wanted to get them out there rather sooner than later (also "doing a beta test" is easier said than done - how do you get people to run a test version of rsync?).
Nice post and technically impressive work. I agree we need to understand the build pipeline and be able to do things locally. However, depending on your electricity cost, it might not make sense financially. These old servers are not energy efficient at all (I'm guessing that old Xeon server will easily pull 200W on load), and that model is currently at 0.1$/0.3$ per 1M tokens (with 76 tps and 262k context) in Openrouter (also, these servers are LOUD).
EDIT: I stand corrected, 200W is apparently way too high of an estimate. I used to run a bunch of old Xeon servers and they slurped watts like crazy, but I can't remember which ones exactly those were.
OK, then you're in luck. I had a bunch of old 1U rack servers and even in the next room it was too annoying to run them (they had a bunch of 40mm fans which always ran at full speed, because in a server room, no one can hear you scream).
Could it just be really bad cooling? Looking at 9800X3D, it seems like it's running in a similar range wrt TDP unless you really push the 9800X3D. I'm comparing with desktop cpu's because that's what my workload is. cpu governor is set to performance (no schedutil). No audible change in fan speed during heavy compilation or gaming (very silent humming), and i don't have any fans beside cheap intake, cpu and exhaust fans (1 each) + an excessive amount of dust.
These servers had no fan control whatsoever, they always ran full blast. That's not untypical for rack servers, because as written: they are designed for server rooms, and you're supposed to wear ear protection there anyway... Yes, I could've modified them, but I ditched them because running them simply made no sense (especially the high idle power consumption was ridiculous).
Yeah, 1u is gonna do that. Get something that can accommodate a big tower air cooler such as the Hyper 212 and your airflow will be quieter than the disks.
I don't run it anymore but my old server was a dual xeon (with two of those coolers crammed in) and I rarely heard a peep out of it.
Only when you remove it from the original server or enable low fan mode (if available). Most 1U/2U cases will happily blow at full speed well over 90db.
You likely need to replace the flow-through server chassis system with an active "normal" cooler to achieve a bit of silence.
85W might be about right. My old server CPU is in the same ballpark and compiling kernels it reached about 90w in power usage. If you want to keep it running: idle is not very low power unless you have one of the "low power" L versions, keep that in mind.
Get a 4U case, many options if you want to combine it with a NAS. Not hard to cool and keep somewhat quiet. If you can store it in a closet or something that helps too.
Well, you can use it for lots of other things as well.
Compared to the cloud you can probably save up to buy a new server every month. And don't underestimate the gains of having something to experiment on and play with.
These servers are loud if you're trying to fit them into a 1U or 2U, which requires high speed fans to generate the necessary static pressure to push air through the case. I run a similar setup in a 4U case with slow 120mm fans and it's fine.
Well, if Apple killed it, Lenovo killed it even more. I recently was looking for a laptop for a student. The Lenovo E14 Gen7 is 800 Euros here in Germany (where prices are always higher, the MacBook Neo is 700 Euros), it has 16GB of RAM, 1TB SSD, a 2.8k IPS display, a Intel Ultra5 12core CPU, and it has a repairability score of 9/10 from ifixit. Framework doesn't even come close to that package.
Same thought, as an owner of a similar Lenovo, that's top bang for the buck. Also, matte screen and hinge that opens 180 degrees is something the Neo and most Macs doesn't have.
Though I assume the Apple clientele is always different than those shopping for PCs, and doesn't care about specs, they just want MacOS and the Apple ecosystem, most likely they already have an iPhone or are planning to get one anyway so then a Macbook is the only thing on their radar. Those people aren't really shopping for PCs anyway unless they need some Windows/Linux exclusive apps like CAD/CAE.
But if you want to run linux and game then that Lenovo would be a good deal.
Similar to the Framework, it has its own niche clientele who values the company motto, tinkering and repairability aspects way more than the value proposition. Most likely they run Linux too.
It is funny how Mac OS is a draw for some, when it is the main reason I don't use a Mac. Their hardware is excellent, but when I've tried using a Mac as my main machine, my productivity suffered. The only part of the Apple ecosystem I wish I could get on Windows is iMessage, and maybe FaceTime.
> The only part of the Apple ecosystem I wish I could get on Windows is iMessage, and maybe FaceTime.
It annoys me that these are such a draw. There are a dozen other viable messaging and video call apps, but there's always someone who feels like spending two minutes to install and activate one is a major imposition.
I like Apple hardware. I like the Apple integration. I like the hardware quality. I LOVE the silence of the M series machines.
But for me you’re right. More than anything, I’m not giving up Mac OS. Despite Tahoe, which I do severely dislike, I’m still far happier using it daily than Windows or Linux.
Until that changes, or the hardware gets bad enough (it’s going in the other direction), I’m not leaving. I don’t even look at other options for my real computers.
“Toy” computers that I want to throw Linux or BSD or something on just to play with, yeah of course. But not what I want to use all day every day.
Framework is definitely premium-priced, but I don't think most people are cross-shopping the Framework 12 (a 12" convertible tablet) and the Thinkpad E14 (a 14" dedicated laptop).
No such model exists. The Framework 13 comes closest, but a 13" screen and a premium shell would compete more directly with the Thinkpad X13.
Direct price comparisons get tricky because different buyers care about different details. I really like the Thinkpad's Trackpoint, for example, but I also like the Framework's 3:2 aspect ratio. I'd have a hard time choosing.
Dammit. I got an IdeaPad of similar price in december 2024. It didn't have one of the fancier displays from the era but still a decent option, it has 16Gb and I thought I'd try a Ryzen mobile thing that time. Wish I'd gone for the Thinkpad E series had I known about it then : that lower-end IdeaPad feels like trash.
SSD IO is sluggish, fans always spin when plugged in, audio crackles if I so much as scroll a page while a youtube video is playing, the keyboard might be the worst I've touched in many, many years, the 3.5mm audio jack wore out into intermittent connectivity within a couple of months. At least the display still looks good.
Went through the windows optimization motions with it too. My x230 with an i5 still has lower and more stable DPC latency and has remained my DJ laptop.
Looking at all the unmerged pull requests in ripgrep, you can see what's going on. I will not link him here, but for instance, there's a "Senior Software Engineer at Microsoft", whose agent created 260 PRs in 211 repos with trivial typo fixes in code comments(!). Almost all of them are rejected (including those in ripgrep), but of course, a few get merged and he now boasts he "contributed" to sqlalchemy, Nim and others... What a time to be alive.
I used my human eyes to submit updates to Redis and Git, fixing typos in comments.
Sure it's low-hanging fruit, but if you're looking at the code it's good to have the comments be readable and not full of typos.
(That said this was a few years ago, and there were no LLMs at that point. I didn't go out of my way to make trivial contributions, but I figured since I saw the "problems" I should submit a patch to fix them.)
The project itself is MIT, meaning the scripts/docs I wrote. Everything else remains under its respective upstream license, GPL, vendor/proprietary blobs, debian packages, firmware, etc.
I did mention this in the license section, last line of the README.
Oh well, they really do their best to alienate people as well. They just completely overhauled their UX, and after that update, people at my company were so confused, they couldn't even open new issues anymore, because everything was somehow renamed to "work items". I kid you not, literally two decades of UX people were used to, just thrown out the window, it's absolutely mind-boggling. The feedback to this is devastating:
reply