Borrowed Silicon The model does not need to know anything

Three machines on my desk, all running an AI model at the same time.

A Raspberry Pi 4 with 1 GB of RAM. A Mac Studio with exactly 512 times as much fast memory. And a graphics card with 16 GB, sitting in a box that breathes hot air into a room where the air conditioning was already losing. You hear all three of them before you see them.

I have been doing this obsessively for weeks. Months, if I count honestly from the point where tinkering became a problem.

How good are local models next to the cloud ones, really? How would you even measure that? Which model fits which hardware? What does it cost to run something that competes with Claude or Codex, and does that hardware exist at a price a normal person can pay? And what happens when you ask a Chinese model about Tiananmen Square?

I have answers to all of those now. One of them surprised me enough that I rebuilt my whole mental model of where this is going.

Why anyone bothers

The case for running your own is not really about the models. It is about everything around them.

Absurd amounts of money are going into AI data centers, mostly in the United States, and the two visible consequences are protests in the neighborhoods that get one, and hardware prices going up everywhere else because those buildings buy components by the pallet. Then there is the trillion-dollar question nobody in the industry wants asked out loud: what happens if subscriptions and token sales never add up to what was spent. And plenty of people are simply not comfortable handing their data to an American company and hoping it stays where they put it.

So: your own hardware, or hardware you rent for exactly this and nothing else.

The catch is quality. Models you can run on a normal PC are still generally worse than what Anthropic and friends are serving, which should surprise nobody – the big ones live on machines that cost millions of dollars. Set expectations accordingly and you will be much happier.

But the gap has gotten strange. Qwen3.8-27B runs fine in 24 GB and does an amount of work that would have been ridiculous to claim a year ago.

The number that started it

There is an observation I kept running into: with a lag of six or seven months, a home PC can run something roughly as capable as what was in a data center back then.

I thought that was optimistic marketing. So I checked it against Qwen3.8-27B, released on 14 August 2026, which needs a 24 GB card. A used RTX 3090 – a card that is almost six years old now – clears that bar.

On Artificial Analysis it scores 52 on the Intelligence Index.

Now line that up against what was state of the art when this was strictly datacenter territory. Claude Opus 4.5 and Gemini 3 Pro both landed in November 2025. GPT-5.2 followed on 11 December. Re-scored on the current index, all three of them sit in the low forties. Below the thing on the graphics card.

So the observation holds. The lag is just closer to nine months than to six, and there is an asterisk worth saying out loud: Artificial Analysis overhauled the index in the middle of that window, so this compares current re-scores, not the numbers those models launched with. I would rather hand you the asterisk than a cleaner story.

Nine months. That is one pregnancy between a rack of accelerators and a used gaming card.

And then I asked it something

I ran a pile of random knowledge questions at Qwen3.8-27B, the kind of thing you would ask to get a feel for a model.

Almost every answer had something wrong in it. Not subtly wrong – invented names, invented dates, confident nonsense delivered in the tone of a man who has read the file.

Which raises the obvious question: how does a model score 52 on an intelligence index while lying to me about basic facts?

The answer is embarrassingly simple, and I should have looked it up before I got annoyed. Almost none of those benchmarks test factual knowledge. They test coding, driving a terminal, using tools like web search, and pulling a specific detail out of an enormous context without losing it.

Exactly one of the nine evaluations in the index is about knowing things: AA-Omniscience. It contributes 12%, and even that splits in two – 8% for getting facts right, 4% for not making them up. So hallucination is measured. It is measured through a keyhole.

My first reaction was that this is a broken way to rank intelligence.

My second reaction, about a day later, was that it is probably correct.

Because in 2026, whether a model has a fact memorized is not a very interesting property. It can search. It can read a file. It can call a tool and check. Testing recall is like grading a mechanic on whether they can name every torque spec from memory, in a garage that has the manual on the shelf.

Maybe the enormous models were never the point. Maybe the point is a small model that knows how to go find out.

The borrowed Mac that still was not enough

Best open-weight model you can run yourself right now is Kimi K3, tied at 60 with GLM-5.3, two points behind Claude Fable 5. So naturally I wanted it running on something of mine.

I borrowed a Mac Studio with an M3 Ultra for this (thx, Trương). The 512 GB one. One of the configurations where you look at the price and briefly reconsider your life.

It is not enough. It is not remotely enough. Kimi K3 is 2,800,000,000,000 [2.8 trillion] parameters, the weights alone are a ≈1.56 TB download, and a real deployment wants something like ≈1.6 TB of GPU memory spread across multiple nodes. I had borrowed one of the biggest Mac Apple sells and I was off by a factor of three.

Well. That is one way to learn a lesson.

(Also worth knowing before you get excited about open weights: the K3 license restricts commercial use above a revenue threshold. Downloadable is not the same as free).

Fine. GLM-5.2 fits – that one wants ≈450 GB, so it lives comfortably on the borrowed machine, and I genuinely like working with it.

It scores 53.

One point above the model that runs on a 24 GB gaming card. One fucking point, for roughly 19 times the memory. I sat with that number a while and then laughed, because there is no other honest response available.

Bigger, faster and more expensive does not automatically solve anything here. That is the whole lesson of the borrowed Mac and I paid for it in shipping.

Tiananmen Square, and who will talk about it

Ask Qwen3.8-27B what happened in Tiananmen Square in 1989 and it tells you it cannot provide information on that subject. Watch its reasoning trace and it is even clearer: the topic is flagged, internally, as too sensitive to touch.

You cannot generalize that to Chinese models as a category, though, and I want to be careful here because the lazy version of this take is everywhere. GLM-5.2 is also from China. It answers the question in detail. It then offers – unprompted – to talk about the decades the Chinese government has spent suppressing discussion of the massacre.

Same country, same regulatory environment, completely different decision at the alignment layer. Whatever you think is happening, it is not one thing.

The uncensored ones

Because the weights are open, people strip the refusals out and republish. These are called abliterated models, and they are not fine-tunes – abliteration finds the direction in the model’s activation space that corresponds to saying no, and deletes it.

What comes out answers everything. Bombs, malware, whatever you hand it, no hesitation, no lecture.

It also hallucinates like it gets paid per assertion. Which makes perfect sense once you notice what was removed: the model’s capacity to refuse is not a separate module bolted onto the front. It is entangled with the model’s capacity to decline in general – including declining to be sure.

There is research on this. Strip the refusal direction and MMLU, HellaSwag and IFEval all move by less than a point. The only real regression is TruthfulQA, down 7.1 points.

Now go read a model card for one of these things. MMLU, MMLU-Pro, GSM8K, CMMLU, perplexity. No truthfulness benchmark anywhere on it.

The one thing the literature says breaks is the one thing nobody puts on the card: I wrote a short piece about that when it first annoyed me and I have not gotten less annoyed since.

It gets worse when you ask what “uncensored” even means to the people selling it. Two teams abliterated the same base model and measured the result two different ways – one with a classifier checking whether the answer opens with I cannot, one with a language model judging the actual content. The first reports near-zero refusals. The second still flags 36 of 100 harmful prompts as refused. Same operation, same weights, answers a factor of 15 apart.

Nobody in this market knows what they are selling. They just know it sells.

Rounding the weights until something breaks

Quantization is the thing that makes any of this possible on hardware you own. You reduce the precision of the stored weights – round them off, essentially – and the model gets smaller.

Piotr Migdał ran the full sweep on Qwen3.6-27B and published the results, from BF16 down to two-bit.

At BF16 the weights need 54.7 GB. The smallest two-bit build needs 9.6 GB.

That is a brutal difference in size for a surprisingly small difference in output. Pelican drawings came out unaffected at four bits and above, and even the three- and two-bit versions only degraded in ways you notice when you average over many samples rather than look at one.

The thing that actually breaks is knowledge, and it breaks below four bits, not above. Capability compresses. Facts do not. You cannot fold the internet into a few gigabytes no matter how carefully you round.

Which is why everyone lands on Q4 or Q5 and stops. Going higher buys almost nothing, and nobody has memory to burn – especially not in a year when new phones and laptops are shipping with less RAM than their predecessors at the same price. Read that sentence again. In 2026. LESS.

16 GB is enough for a Mixture of Experts model like Qwen3.6-35B-A3B, which has more total parameters than the dense 27B but only activates a slice of them per token. The experts in use sit on the card, the rest wait in system RAM. That is how a model with a bigger number in its name runs on a smaller card, and the first time it worked I refreshed the memory readout twice.

Electricity, and what you plug into it

Small models only get interesting when they can use tools, and that needs a harness: the software wrapped around the model that decides what it is allowed to do and hands it a terminal, a browser, a Python interpreter.

I like keeping the model and the harness separate, and the way I think about it is electricity.

The model is current coming out of a socket. The harness is whatever you plug in – a TV, a rice cooker, a fan you bought on Bùi Viện that has never once been balanced. In AI vocabulary the socket is the endpoint, and the thing providing it is the runtime.

Runtimes are not interchangeable, which took me embarrassingly long to internalize. On a single server with eight RTX Pro 6000 Blackwell cards (rent by the hour via Hostinger), Ollama gave me around 21 tokens per second on Qwen3.8-27B. vLLM on identical hardware reached roughly four times that. (Most of that gap is batching, so if you are running a single conversation the difference shrinks – but for agent work, where the machine talks to itself constantly, batching is the whole game).

For continuous agentic setups there is OpenClaw and Nous Research’s Hermes Agent. For on-demand work there is OpenCode, which I use daily from the command line: organize the images in this folder, read the logs and tell me why this service keeps dying. It works far better than it has any business working. If you want a window instead of a terminal, LM Studio Bionic does tool calls and web search, though the search toggle stays greyed out until you sign in.

I have never manually installed vLLM. Not once. I pointed a GLM-5.2 endpoint at the machine, told it to serve Qwen3.8-27B and figure out the best configuration, and it tried quantizations, tried runtimes, downloaded what it needed, benchmarked, and landed on something that worked. I have done this on several machines now and it keeps working.

Most people will not have a spare GLM endpoint lying around, obviously. That one was a coincidence. A Claude subscription at $17-20 a month does the same job, and Claude Code installs in about a minute. One warning from repeated attempts: letting it pick the model for the machine is hit-or-miss. Name the model yourself.

Then wire Open WebUI and OpenCode to the same endpoint, uninstall Claude Code, and the thing that set your local stack up has made itself redundant. There is a joke in there and I am too tired to find it.

One thing you should not do

If the machine serving the endpoint is not the machine you are sitting at – especially if it lives outside your local network – do not put that endpoint on the open internet.

Put it behind WireGuard. Takes 20 minutes, costs nothing, and saves you the specific flavor of dread that arrives when your GPU fans spin up while you are asleep. An unauthenticated LLM endpoint reachable from anywhere is somebody else’s free compute and eventually somebody else’s problem to explain to you.

Where this actually goes

There is much more to say, particularly about Apple hardware, and it will keep.

But the shape is clear. A 27-billion-parameter model is now enough for most of what I want a machine to do, and super-high-end hardware is not the requirement I assumed it was. These models hold less knowledge than the giants. They make things up. Mine lies to me weekly.

And it does not matter nearly as much as it should, because most people are still using local models the wrong way – asking them questions, copying code out of them, treating them like a worse search engine that runs on your own electricity. Give one a harness and it stops being a thing you interrogate and starts being a thing that goes and does the work.

Small model. Right hardware. Tools it knows how to use. That combination beats the benchmark numbers by a margin the benchmarks were never built to show.

The model does not need to know anything. It needs to know where to look.