Language models, disassembled
and rebuilt across a mesh of machines.
A distributed inference engine that splits a transformer's layers across independent nodes and runs a single coherent forward pass across all of them — no one machine ever holds the whole model. Custom protocol. Custom runtime. No inference framework underneath it.
Large models need large, single machines. Most compute isn't that.
Running a capable model has meant renting a machine big enough to hold every one of its layers in memory at once — even though most available hardware, across most homes, labs, and desks, only ever holds a slice of that.
The gap isn't compute in the abstract — it's compute in one place. A cluster of ordinary machines, none individually capable of the full model, can still run it together if the model itself can be taken apart along its layers and reassembled across a network.
That's the shape of the problem this project takes on: a transformer split cleanly enough that any machine can own a slice of it, a protocol precise enough to move state between machines without losing correctness, and a mesh that stays coherent as machines join, leave, or swap.
Four components, one coherent chain
A registry, a mesh of nodes, a coordinator, and a hot-reload path that keeps every machine in sync when the model changes.
The registry — the network's phonebook
Nodes announce on startup, either with a fixed layer range or as an unassigned node waiting to be given one. The coordinator resolves the current live chain before every request, so the mesh can reshape itself at any time.
Each node computes only its slice
A node reads just its assigned layers off disk rather than the full checkpoint, receives a tensor from the previous node, runs its layers, and forwards the result on.
The coordinator drives the request
It tokenizes the prompt, sends it into the chain, and streams logits back as they arrive. The prompt crosses the wire once — after that, only the newest token's hidden state travels, since every node holds its own local cache.
Hot reload reaches the whole mesh
Every node polls the registry for the active model and its own assignment, so a model switch reaches every machine within seconds — not just the one that initiated it.
Engineered for hardware that shouldn't be able to run this
Three subsystems that exist specifically because "spread a large model across weak devices" falls apart without them.
Lazy layer loading
Nodes read only their own layers directly off disk instead of loading a full checkpoint and discarding the rest. Verified tensor-by-tensor, not by inspection.
Distributed KV cache
Every node holds a local cache keyed to the request. Only the newest token's hidden state crosses the wire after prefill — caches themselves never leave the process.
Quantization
Catalogue models load in int8 or int4 via bitsandbytes in place of full precision. Requires a CUDA node on that layer range; others fall back automatically.
What a node is, what the mesh does
Node
- Any machine on the network
- Owns a contiguous layer range
- Reads only its own layers from disk
- Serves over a length-prefixed TCP protocol
- int8 / int4 on CUDA, fp fallback elsewhere
Mesh
- Central registry + heartbeat, 5s interval
- Auto layer-split across confirmed nodes
- Hot model switch, network-wide
- LAN-native; Tailscale for cross-network
- Live pipeline view via web dashboard
Model support
- GPT-2 family, OPT, TinyLlama
- Phi-3 Mini, Mistral 7B Instruct
- Sharded, per-node shard fetch
- Safetensors required for lazy load
- Recommended entry: TinyLlama 1.1B
A live view into the mesh
The pipeline as tokens move through it, and model control without a terminal. Reads and writes only what each node already exposes on its management port.
Layout mockup, built to match the running dashboard's CSS exactly — swapped for a live capture once deployed on your mesh.
What this doesn't do yet
Disclosed plainly, the way any engineering brief should read.
Centralized registry
A single point of coordination. If it drops, new requests can't be routed and switches can't propagate — in-flight requests still complete. A peer-to-peer topology would remove this but is a materially larger build.
No load balancing
Requests run through one linear chain. Concurrent requests queue rather than route across spare capacity.
Node failure isn't mesh failure — but it isn't retried either
A dropped node is detected in seconds and the request returns what it generated so far. It's not auto-retried: with KV caching in play, a silent resend risks double-applying a token.
Lazy loading and quantization are not yet composable
Quantized loads use the full-load-then-slice path — packing weights safely from hand-picked tensors is separate work.
Tensor transport uses pickle
Appropriate for a trusted network. Node ports are not designed to face the open internet.
Open threads
Bring up a mesh in three commands
A terminal launcher shows live status — registry health, connected nodes, chain integrity — and walks through the rest. Startup order: registry, then nodes, then client, each in its own terminal.
| run.py | Terminal launcher, live status view |
| registry/server.py | Node phonebook |
| node/server.py | Loads a model slice, serves inference |
| coordinator/client.py | Sends prompts, streams output |
| dashboard/server.py | Web dashboard, port 7000 |
Get in touch
Building something similar, evaluating this for a role, or want the full source and technical documentation — leave a note and a way to reach you.