Distributed serving across two Macs

Two ways to split, two ways to talk

How a model splits depends on its architecture. In a tensor split both Macs work on every layer and exchange results at each step; Elzi uses it for models on the Qwen3.5 architecture, Qwen3.8-27B among them. In a pipeline split each Mac holds a range of layers and passes the work along; Elzi uses it for Qwen3.8-Flash-Next. A model type with no split runner is refused by name.

The two Macs talk over the cable in one of two ways. JACCL is MLX's RDMA backend: data moves memory to memory over Thunderbolt without the network stack. Ring is MLX's TCP backend over the same cable. Setup picks JACCL when both Macs have RDMA turned on and tells you why it chose what it did.

For a pipeline the backend barely matters: only about 20 KB cross the cable per token, so the speed is set by the Macs' compute, not the link. A tensor split exchanges far more at every step, and there JACCL is clearly faster, as the numbers below show.

What you need

• Two Apple silicon Macs with Thunderbolt 5 (the setup we test), joined by a direct Thunderbolt 5 cable. Turn off the Thunderbolt Bridge, and give the Thunderbolt link its own IPv4 address (the app's preflight checks this and can set it again).
• For JACCL: macOS 26.2 or later, and RDMA enabled once on each Mac. Start the Mac in Recovery, open Terminal, run rdma_ctl enable and restart. It can't be done remotely.
• Elzi on both Macs, signed in to LeanZero Link, with "Run part of a split model" allowed on the Mac that serves the second half, and macOS Local Network permission allowed for the app.
• The model on both Macs. The app can copy it over Thunderbolt and installs the same Python stack on each Mac when you save the setup.
• Enough memory on each Mac for its slice. The app's preflight checks every Mac before anything loads and refuses with numbers if a slice won't fit.

Only one engine owns a Mac at a time. Starting a split while a single model is loaded is refused by name, and the app offers to unload it and continue.

When it helps

• Fitting a model. This is how Elzi runs Qwen3.8-Flash-Next, about 98 GB at 4-bit, on Macs that can't hold it alone: a pipeline split started from the app, with streaming, tool calls and cancellation.
• Reading prompts. A tensor split of Qwen3-32B (a community 4-bit build) over JACCL read a 2k prompt at 342 tokens per second, against 269 on the M3 Ultra Mac Studio alone (23 September 2026, a raw MLX test outside the app).

When it doesn't

A model that fits one Mac writes faster there. In the same Qwen3-32B test, the M3 Ultra alone decoded at 31.9 tokens per second, the JACCL split at 15.7 and the ring split at 9.6. So we use a split to fit a model or to read long prompts faster, never to speed up a model that already fits.

A split also costs time to start: loading MiMo-V2.6-Flash, a model we no longer serve, across our two Mac Studios took 229 to 277 seconds in October 2026. Some architectures can't split at all, and a big job on either Mac can leave too little room for its half. Prompt lookup and the memory lock run on one Mac only for now.

How the app keeps it running

The app starts its own rank program on each Mac. The first Mac serves the API and the second follows. Both Macs show the split in the menu bar with their own rank. In a 62-minute soak test with Qwen3.8-27B over JACCL (September 2026), all 275 requests succeeded, a frozen rank was found and restarted, and a killed rank was reported within 2 seconds. Under memory pressure the split holds new requests until there is room, rather than failing them.