That’s the reason they are trying to embed AI into everything.
Good.
Open models are still most cheaply hosted on a cloud, with batching and 24/7 use. API rates from developer lab are generally fair. Self hosting does have some significant tangible benefits though: Fine tuning for domain specific to organization, and not letting LLM provider train from your prompts/answers, followed by competing with your organization in the future as a result of “distilling your IP”.
We even got open weight models that’s 27B + 1-bit (and it still has good performance)
Bonsai? Or whatever it’s called? It’s a con, so far; it’s not better than smaller models quantized to 3-4 bits.
I love, love the idea of bitnet, but it only seems to work with models trained from scratch, which no one has done at scale yet.
The awkward part is that software has a habit of racing toward free once it becomes good enough. If an open weight model delivers 95 percent of the value without recurring API costs, plenty of companies will choose that and spend the savings on integration instead. History keeps rhyming, even if investors hate the tune.
The awkward part is that software has a habit of racing toward free once it becomes good enough
Has it? Jira, Teams, Slack, Google Meet, Cisco WebX… There’s obviously Jitsi but I don’t see companies racing towards it. When it comes to SaaS most companies prefer to pay and be done with it. Few are choosing to host open source solutions. Windows and Mac are also way more popular than Linux in office settings. With LLMs hosting it will be even more complicated because companies will have to invest heavily in GPUs.
If the bubble bursts companies will lose funds to work on new models but someone will still be able to offer existing models as a service. Companies will chose the one with better price and functionality. Being free or not will have little to do with it.
They don’t have to host it themselves. They could use a number of providers for the same model, and basically keep doing whatever they were doing with OpenAI/Anthropic via the exact same APIs.
When you pay for something that means passing responsibility (liability) for that function.
Much harder to sue over free software.
Unfortunately, based on many years of experience, I have to agree. However, I also see a light at the end of the tunnel, particularly in Europe, where there is a growing desire to break free from the US SaaS stranglehold, using open source software.
Regarding OWAI I see a fundamental difference: Slack isn’t just software, it’s a hosted service with identity, storage and network effects.
An open weight model is more like a compiler: once downloaded, nobody can revoke it. You may still pay for inference, but pricing power drops when anyone can host the same model.
I ran DeepSeek and Llama and Mistral at home on my consumer grade gaming PC.
With a little tweaking of the system prompts and configuring web search, I was running a local LLM that felt pretty darn close to the commercial LLMs.
With this technology out in the open internet where you can download the models in a few hours I don’t see how the commercial AI companies are going to last. If selling “Artificial Intelligence” subscriptions is all your company does for revenue, you’re screwed.
I downloaded and ran an LLM that I could have a conversation with and feed basic coding problems to for basically zero dollars and ran it on my puny gaming machine…puny compared to enterprise-class hardware. It would be trivial for a company with a very moderate budget to buy some servers and start running their own LLMs that they can use to feed all the PII and HIPPA data they want.
And Llama and Mistral are ancient history at this point.
The cutting edge of local is lightyears better now. It’s basically where ChatGPT/Anthropic were not that long ago, with a bit less world knowledge because of the size.
Thanks for the info. I don’t follow it closely but afaik wasn’t Mistral the only western open weight model around?
Mistral is still around, but for local LLMs… they’re kinda irrelevant, sadly. Their models have regressed. I think they’re being choked by ambiguous EU law.
If you’re looking for “western” weights, I’d look at the Laguna series first:
https://huggingface.co/poolside
And ik_llama.cpp quantizations like this: https://huggingface.co/sigargv/Laguna-M.1-GGUF/tree/main
There’s a couple of other interesting startups, but TBH its hard to keep track of where they’re from.
I wasn’t aware of that.
Thank you so much for the info.
What’s the cutting edge now? Skool me…I want to try it. Can I grab one using ollama?
Depends on your RAM (main + GPU), but assuming 32GB total: Qwen 3.6 35B A3B for coding support, Gemma 4 26B for general stuff. The LM Studio app curates a list of recommended models that will run well in it and makes it easy to run them.
Mind you, what I like most about local models is their limitations, because it turns out closed models have limitations of the same nature, just with quite a bit more runway; and becoming aware of those limitations is valuable.
https://sleepingrobots.com/dreams/stop-using-ollama/
And this is just the tip of the iceberg for ollama. They’re the same kind of scammy tech bros as OpenAI.
The best setup depends on your hardware. There is no “easy button” unfortunately, quantized LLMs are just too intense and finicky to run without making some informed choices.
It also depends on what you want to do with the LLM. For example, some are too slow or bad at long context for agenic use, some quantizations are great at scripts but terrible outside that, or vice versa.
But LM Studio and Qwen 3.5 35B Q4 is probably the “easiest” flat recommendation I can make.
Or… honestly, just pay $40 for basically unlimited usage for a year from an API, then roll your own frontend.
Or… honestly, just pay $40 for basically unlimited usage for a year from an API, then roll your own frontend.
Can you clarify, what you mean by this? Rent a VPS? Or is there a legitimately good place that’ll offer “unlimited” llm api access for $40 / year (and would you have any sort of privacy with this)?
I was thinking of Xiaomi’s Lite coding plan:
https://platform.xiaomimimo.com/token-plan
My mistake, it’s $60. But that’s still dirt cheap.
You can read their terms of service; coding APIs are generally less invasive than something like the ChatGPT app, but don’t bet on any privacy either.
For my personal setup, I actually run most queries locally, via MiMo 2.5, but send “bulk” privacy-insensetive prompts to the same model on the API, since its so much faster. I kinda use them in parallel.
Thanks for sharing the link. I’ll take a peek at them and their ToS.
I also forgot to emphasize this, but Xiaomi’s plan, in my opinion, is undiscovered fruit.
GLM had a similar coding plan, but once it got in the news and popular, it got WAY more expensive and limited. I’m grandfathered into 6 more months of a GLM plan you literally cannot buy now.
And I think Xiaomi is in the same situation GLM was 6+ months ago. It’s a fantastic model series, but unlike Kimi/GLM no one knows about it yet, which is how it’s still $60 for a year.
Why are quantised LLMs harder to run than non quantised ones?
I just meant that you have to be cognizant of what went into the quantization.
As an example, a “Q4_K_M” could be too much quantization to be usable on one model, and an inefficient waste of space on the other. Two Q4_K_Ms of the exact same model could be completely different, one totally borked. Or one particular Q4_K_M could excel in one task, but be totally useless for another, even with the exact same settings, when a slightly different sized or type of quantization would excel.
It’s a deep rabbit hole. It’s not random either; there are distinct technical reasons behind every case mentioned above.
And that’s not even at the cutting edge quantization anymore, though what’s “cutting edge” completely depends on your particular hardware and use case.
I’m trying to make this sound daunting on purpose.
Many people have really horrible experience with a default “ollama run” for this exact reason, because the defaults are terrible and the customization is critical to getting coherent, performant output.
Unquantized LLMs, on the other hand, are basically always run the same way: vllm docker image on a big server, official weights. There’s less to “go wrong” trying to squeeze it on hardware with unofficial runtimes and compressors.
Well that sucks. I was really impressed as a novice to open weight LLMs with the ease of use for Ollama on Bazzite.
I’ve been running LM Studio on Bazzite and I had to do nothing to get it working. Just go to the LM Studio website and download the
.appimagefor Linux. If you open it with Gear Lever it will install like an app from the app store and show up in your launcher with an icon.From there I have just been able to download models and use them from in the app. In fact I setup a local server to connect to my IDE and have been trying out local models for coding. It’s pretty cool
Awesome! I’ll give it a try.
I can also vouch for lmstudio. If you can get Hermes running on Linux I would suggest trying that as well. It connects to lm studio and you use Hermes to communicate with the model. Iook into it as there’s a lot to it, I’ve really been enjoying using it so far it even learns how I like to create tasks and I’ve stopped having to ask it to delegate certain tasks, it just knows to do it and to break down the tasks so my fairly context starved local model can handle it.
As for a model, the Qwen 3.6 family of models do really well. I’d suggest the Qwen 3.6 35B a3b probably Q4 depending on your hardware. It’s large, but because it’s a mixture of experts model only 3b of experts are kept on vram at any one time so it stays fast. Qwen 3.6 27b is the smarter “dense” model, but trying to stay with Q4 for quality it becomes too large for 16GB vram and for me runs at like 2 tokens per second lol
Since you mention using Ollama, you probably aren’t running actual deepseek on your pc. Ollama took a Qwen model that was finetuned using deepseek output and named it deepseek.
Those are pretty out of date models at this point. Right now, the model most people would recommend for consumer gaming hardware is Qwen 3.6 27b.
I actually tried Qwen 3.6 27B but it wouldn’t quite fit in my 6900XT so I had to go down to the 14B. I don’t have the tools or the skillset to really test the capabilities of an LLM but with some very rudimentary system prompts it felt quite natural to me. Shockingly natural considering that talking to a real LLM running on my own PC felt like it was smarter than the Majel Barrett computer on Star Trek:TNG…
Oh yeah, 16 GB of VRAM is a strange spot to be in. Most of the focus goes onto the models that fit in 24 GB cards.
Qwen 3.6 35b-a3b is pretty solid when you don’t have enough VRAM for a dense model. Most of the weights can be left in RAM and it still runs really quickly.
Relevant to the intentional murder of the consumer electronics industry
The Western model was doomed to failure from the start. The only barrier to entry was being able to download thousands of TBs of internet archives/books and to have a lot of compute.
The math for these models isn’t proprietary and most CS students are exposed to machine learning and neural networks while in school.
The only advantage western companies had was the ability to buy up the entire hardware market, pricing out domestic competition, and to use their politicians to manipulate trade policy in order to restrict sales of critical hardware to China.
Every US tech company has dumped billions investing in an unsustainable business model with the hope of buying a global monopoly by strangling competition.
China can destroy all of that by making their models open weight. The real money is in finding and implementing custom AI solutions… not in charging for access to the models. By having freely available models, they’re making the barrier of entry as low as possible.
Not to mention that the insane amount of money being poured into hardware by US tech companies has created an environment where building fabs has a much shorter ROI, which also helps China’s development in that sector.
US companies are playing Monopoly while China is playing Civilization.
Buy up the entire hardware market, that’s an interesting take I’ve never come across before. I am unsure of the cost benefit of such a big move as I got little to no idea or data on this as the whole industry would need to operate as a cartel as the assumption.
I still think this type of AI can lead to a worse state of living for most of us, but it’s better than the bullshit in the US.
Eat my ass Musk, Zuck, Altman and all you other fuck face tech bros
The article wastes no time getting to the underlying point in the very first paragraph:
Top executives at leading Western AI companies are increasingly warning about the safety and national security risks posed by Chinese open-weight frontier models. What they tend not to mention is that these models are improving rapidly and, because they are freely available, pose a serious threat to Western labs’ business models.
I found the following two paragraphs interesting:
By mid-2026, however, open-weight frontier models from Chinese labs such as Alibaba, DeepSeek, and Moonshot AI had nearly matched the leading Western models in intelligence and performance. Many companies have already begun building their AI systems on top of these free models, avoiding the high cost of closed-model APIs.
Because businesses can host open-weight models in their own private clouds, they can also avoid sending proprietary data to systems controlled by outside providers. Developers can fine-tune the models for specific needs, build applications and tools on top of them, and optimize them for their preferred infrastructure.
Back to local computing is the way to go
It always has been. And yet consistently for decades they continually turn to Big Tech anyway…
they can also avoid sending proprietary data to systems controlled by outside providers.
Not only proprietary but also personal or other kinds of sensitive data.
If you are doing health research on databases of personal health data, you should be able to guarantee the safety of that data.
That means you can’t use the current American systems, because they’ve been shown to be insecure.
This would be a major issue in EU, where such data is legally protected.This would be a major issue in EU, where such data is legally protected.
Nothing a EU–US Data Privacy Framework can’t handle.
I absolutely agree that that agreement is complete and utter bullshit.
Hopefully the shift there has been to achieve IT independence from USA will mean EU doesn’t give in so easy next time.
These people are too rich to be punished by laws. They’re the people laws protect not the ones they bind.
Not in EU, Eu has given fines to those big tech companies before and can do it again.
Fines are just the cost of doing business.
Wake me up when corporate charters are dissolved and executives are put in prison.
More than their profits? Their profits over course of the shittiness?
Being less bald face and trying to hide your corruption better than America doesn’t make you the good guy.
The localllama crowd has know this for years.
It happened faster than I expected, though; OpenAI/Anthropic hardly even got the chance to tighten the screws.
I just wish I could buy enough memory to run one of these models locally. Specially Kimi K3
Same, getting ~3 trillion parameters in consumer hardware is rough.
If Nvidia has any foresight they’ll see the writing on the wall and start getting higher memory Spark style SMB inference machines, few people in the long run are going to pay retail API token costs,
Especially giving the lack of trust organizations rightfully have in big AI companies
Exactly, having the mustard toddler pull fable because of, who knows? That burned trust in the west’s AI companies.
And there is also the question of what they’re doing with the data you send into the models.
How much RAM do you have?
I can run MiMo 2.5 at about 9 tokens/sec, on 128GB RAM, a 7800 and a 3090 in an SFF rig. That’s a fantastic 310B model. I’m requantizing it right now, to see if I can speed it up with Dflash.
Still fantastic models can be run on 64GB or 32GB CPU RAM, as long as you have some GPU. We’re awash in sparse models these days.
I’ve got 128GB RAM + 24GB VRAM on a 4090. I’ve managed to get a 400B parameter model running on a single board computer with 64GB RAM by using MMAP. But I want to run Kimi K3 locally so I would need a lot more RAM / bandwidth
Oh that’s perfect!
You can’t realistically run Kimi (as it’s a 1T+ model), but you’re set. There’s a glut of excellent 120B-300B models for you to choose from.
I’m quantizing MiMo 2.5 specifically to cram into 24GB/128GB as I type this. It’s tight, and will use up most of your memory, but it’s fantastically smart and plenty fast. The quantization won’t finish cooking until tomorrow, but I’ll upload it to huggingface then.
But there are others existing quants would fit, like Deepseek Flash IQ3_S: https://huggingface.co/unsloth/DeepSeek-V4-Flash-GGUF/tree/main/UD-IQ3_S
Or… well, any of these! https://huggingface.co/models?num_parameters=min%3A128B%2Cmax%3A256B&library=gguf&sort=modified
The key for you is to run the ik_llama.cpp fork: https://github.com/ikawrakow/ik_llama.cpp/
It’s specifically optimized for hybrid (CPU + GPU) inference on Nvidia desktops; you’ll get MUCH faster speeds than mainline llama.cpp or anything based on it. It also supports some more exotic quantization type; as an example, I’m quantizing MiMo 2.5 as a hybrid quant, with the dense layers at IQ6K/Q8_0 and the sparse experts as an IQ3_KT “trellis” quant type. This should yield a higher fidelity quantization than a typical Q3 GGUF while taking less RAM, at the cost of taking forever to quantize and a slight speed hit.
Some other quantization types (like the KS or R4 types) are specifically configured to be fast on CPU.
There are specialized “quant cookers” that make GGUFs specifically for ik_llama.cpp, like:
https://huggingface.co/ubergarm
https://huggingface.co/AesSedai
https://huggingface.co/sigargv/Laguna-M.1-GGUF
https://huggingface.co/models?other=ik_llama.cpp&sort=modified
Mine will be here once I upload it:
https://huggingface.co/Downtown-Case
For anyone else reading this: none of this is applicable to you if you have an AMD/Intel GPU, or an older Nvidia GPU, or less than a certain amount of RAM, or a non AVX2 CPU or… well, there’s all sorts of caveats.
The optimal runtime is different for everyone. As an example, exllamav3 is WAY better than llama.cpp on modern Nvidia GPUs until you get above a certain amount of CPU RAM; then using all that RAM for hybrid inference makes more sense. On AMD, different backends work better on different GPUs, and… well, you get the point. Basically all LLM running advice is irrelevant without specifics of your hardware, even this post will be obsolete in a month.
The other problem is bandwidth. Even if you can run Kimi K3 locally, AI usage patterns tend to be very bursty. Having the scalability of cloud compute to run multiple agents and sessions at once makes it much more usable.
My current workflow is passing a human written spec to an agent to implement with strict coding guidelines, architectural decisions, etc. The agent isn’t making any decisions about the abstractions to use, it’s just creating the objects and test suites. So I don’t mind the slower bandwidth because I’m running the heavy agentic lifting over night with no need for human supervision.
But I fully appreciate that my workflow isn’t the norm. In fact my workflow it’s the exact opposite the AI grifters like Sam Altman are selling because it still involves a human with knowledge of the systems making different decisions.
If you’re running stuff overnight on a homelab, you can get away with using very large models as agents in CPU RAM.
Another option is a hybrid approach: farm “dumb” privacy-insensitive calls to an API you can subscribe to for like $40 a year, and let your local LLM do the orchestrating. This is what I do: I run MiMo locally, but also farm calls out to the exact same LLM in the cloud when appropriate… And I don’t mind one bit. I’m happy to tip them for open sourcing it in the first place.
is their a meaningful difference between open weight and open source?
Pretty big difference. An open weight model is a model that you can run on your own machine. You just download and it’s yours to host and use. You don’t need to have anyone host it on their own backend for you, the entire model is available to you to do that on your own. What you don’t have is any control over or access to anything related to how the model was trained. You don’t know what kind of data they used to train it, and how exactly they used that dataset. If you did, that’d be an open source model.
Very important point, i think many people are mislead by the word “open”. Is there any “open source model” around?
There might be, but very few. Most (like gpt-oss, or Deepseek R1) are open weight, because no one wants to make their model training data public.
Open sourcing a model would likely give away they are using pirated content or scrap internet websites without permission. That’s something Anthropic was fined for last year, they were caught using pirated books for model training.
Moreover, training data is one of most lucrative resources in AI race next to GPUs, and AI companies are going all in on finding it, desperately fighting for any possible data. Heck recently they even started buying antique books, digitizing, and burning them.
They invest a whole lot of resources to get quality data, and they don’t want to share it with competitors for free.
My personal opinion though is FOSS philosophy doesn’t apply that cleanly to LLMs anyway. If compiling Chromium or Linux kernel from source is tedious enough, “compiling” an LLM from source on consumer hardware is straight impossible. It’s something only your competitors could do.
Thus you can’t verify the LLM is the same as it’s source code anyway, so open source label becomes meaningless.
And open weight model is still an improvement over using a cloud model, as it’s local and doesn’t call home with your chats data.
Open weight is analogous to a compiled binary. Similar to how Windows is closed source, but still runs in your own hardware, whereas Linux is truly open source.
If all AI really is classified as a “weapon” for export-control purposes, well… in the US possession of weapons is a protected individual right. The US gov should be training open weight models and encouraging self-hosting.
Okay but what about the fact all of this is useless bullshit that will only make the world worse?
It’s far from useless. Overblown, for sure.
They made an awesome hammer that can solve many hammer-related tasks, but they’re selling it like it can also cook, drive and keep the house clean.
Blame the companies, not the math.
100%. These LLMs are fantastic tools for when you could use an LLM to solve a problem or make something better.
Using the right tool for the right job is really not a complicated concept.
The problem, as always, is marketing and hype noise caused by the people who see dollar signs.
Every time I hear about these hammer related tasks I find out they kind of suck or drained lake Tahoe to translate a book that already has a hundred translations.
Or the hammer related task is that one scene from ‘oldboy’.
