Backbone

The best open models keep getting bigger, and the standard answer is the same: rent a cluster, or buy one very expensive machine. Meanwhile most technical households and small shops are sitting on a pile of idle silicon — a Mac or two, a gaming PC, a retired workstation, an old server, phones.

Added up, that pile often has more total memory and compute than the single machine people wish they could afford. Can it be made to act like one?

Distributed inference is a solved problem inside a datacenter, where every node is identical and connected by exotic interconnects. It is decidedly unsolved on a desk: heterogeneous architectures (Apple Silicon next to NVIDIA next to whatever), wildly different memory sizes, consumer Ethernet and Wi-Fi instead of InfiniBand, and machines that go to sleep, reboot, or get borrowed to play games.

Model partitioning that assumes uniform nodes falls over immediately when one node is a laptop and another is a GPU tower.

Treat heterogeneity as the design center instead of the failure case. Probe every machine for what it actually has — memory, compute, interconnect bandwidth — and split models proportionally, so each node carries the share of layers it can genuinely hold and serve. Slow links get respected rather than wished away: partitioning minimizes cross-machine traffic, because the network is the scarcest resource on a desk.

The system has to assemble itself. Nobody is going to hand-configure a cluster out of their own house.

• Open model weights. Frontier-adjacent models are downloadable. The scarce input is no longer the model — it’s the memory to hold it.
• Open inference runtimes. Mature open-source engines for both Apple Silicon and CUDA mean the per-architecture work is integration, not invention.
• Quantization. Aggressive, quality-aware compression shrinks the memory bill enough that consumer machines are back in the game.
• Commodity hardware depreciation. Yesterday’s expensive machines are today’s cheap shards. The used market is an AI supercomputer sold by the pound.

A working pipeline that discovers machines on the local network, probes their real capabilities, partitions a model across them, launches the right runtime per architecture, and exposes the whole arrangement as a single local inference endpoint. It runs daily inside the lab, serving models that no single machine we own could hold — including mixed Mac-and-GPU topologies, and keepalive supervision so the cluster survives sleep, reboots, and human interference.

• Wi-Fi is not an interconnect. Early attempts to shard across wireless links produced token rates best described as geological. Wired networking, or partitionings that keep chatty layers co-located, are non-negotiable.
• Equal splits across unequal machines waste the big nodes and drown the small ones. Proportional sharding wasn’t an optimization — it was the difference between working and not.
• Consumer machines are ill-behaved cluster citizens. Sleep, thermal throttling, and automatic updates all took the system down until supervision and self-healing were built in. The reliability layer ended up being as much work as the inference layer.

The lab runs large open models on hardware it already owned. The system is genuinely useful today for large-memory, throughput-tolerant work — big models at usable speeds — and honest about its physics: interconnect bandwidth, not compute, is the ceiling, and interactive latency on chatty topologies is still an open front.

Serious AI capability stops being gated on capital expenditure or cloud dependence. If aggregate household silicon can be pooled, the unit of AI infrastructure shifts from “datacenter” to “whatever is plugged in nearby” — which changes who gets to run frontier-class systems privately, offline, and on their own terms.

NotebookEvery few months the models get smaller and the used hardware gets cheaper. This experiment gets easier while we sleep.