All is forgiven, Strix Halo - running DeepSeek v4 locally

Prompt: "I want an image of a Strix Halo chip with the DeepSeek logo on it, and it must have flames that make it look like it's on fire, but in a good way"

I was a bit disparaging about my GMKTec Evo X2, a 128GB beast that can run huge models in theory, but felt just too slow for everyday local AI. I take that all back. I was wrong.

Look, it's still slow. It's still not my everyday driver. My RTX 3090 still kills it for speed, and my Mac Studio is still where I turn for bigger models. But now there is DwarfStar, made by Salvatore Sanfilippo, aka "antirez". It changes everything.

Antirez was already awesome in my book: first, he's my age. Second, he wrote Redis. Fucking Redis! If you don't know what Redis is, it's the little piece of software that makes the internet work, despite shitty disk-based databases.

Here's an interactive professional architectural diagram of the Internet. Try pulling Redis out...

DwarfStar is some black-magic-next-level-shit. It does a bunch of tricks, including using very cleverly quantised models, to juuuuust squeeze DeepSeek v4 onto 128GB of Ram. Yes, I said DeepSeek v4. With very little quality loss.

Now for my wildly inaccurate technical description of how this is possible:

To do the quantizations without losing quality, some very clever people essentially attached an MRI to DeepSeek, asked it questions, and saw what parts of its brain lit up. The parts that didn't light up weren't very important, so they could get quantized more. The important bits, less.

The key/value cache chews a lot of memory when running LLMs. But do we really need this all on your GPU? DwarfStar moves this out to your SSDs - slower, but suddenly doable without multiple RTX 6000s.

It also has very impressive-looking parallel abilities, running the model across multiple Macs, for example. I'm not in that league, but should I ever get multiple Mac Studios, I'll definitely try it.

DeepSeek just fits, providing I shut down almost everything else I have running If you squint, you'll see I'm sitting at around 117GB of my 121GB available. A single Docker container can send me over the edge, and cause enough of a crash to force a full system reboot.

I'm also restricted to 100k context size, far from the theoretical 1m limit. And I'm running the Flash version of DeepSeek v4 - the Pro still needs 512GB.

I get an average of 11 tokens per second. It's not fast, but I've given it big tasks, gone away for a couple hours, and returned to see impressive results.

I know it's pretty geeky, but knowing that I can run a frontier model on my desk is absolutely thrilling. My humblest apologies, Strix Halo, you're not completely crap at AI.