Swarms (MoTA)

A swarm (MoTA, Mixture of Tuned Adapters) is many tiny specialists answering at the same time on one small model. Each specialist is a LoRA adapter, a small add-on file tuned for one job, and Gerbil runs them all together in a single batched pass over one shared base model. One model in GPU memory, a few megabytes per specialist, and every specialist gives the same answers it would give running alone (verified token-exact).

Requirements & flags

Swarm decode builds on the batched-decode path, which currently runs on the Node (Dawn) backend and is enabled by environment flags:

# Batched decode with 4 lanes + per-lane LoRA selection
GERBIL_BATCH=4 GERBIL_BATCH_LORA=1 node swarm.mjs

• ·GERBIL_BATCH=N (N ≥ 2) turns on lockstep batched decode with a fixed batch width of N lanes.
• ·GERBIL_BATCH_LORA=1 additionally enables the per-lane adapter machinery: registerAdapter and the adapters option on generateBatch.

Batched decode and multi-adapter selection are on track to become the engine's default execution path, at which point these flags will no longer be needed.

registerAdapter(name, source)

Registers a named LoRA adapter for per-lane batched decode. The adapter's low-rank factors are fetched and packed into the executor's shared GPU factor buffers once; after that, any number of lanes can decode with it concurrently by name. Registering the same name twice is a no-op.

01import { WebGPUEngine } from "@tryhamster/gerbil/gpu";
02
03const engine = await WebGPUEngine.create({
04  repo: "mlx-community/Qwen3.5-0.8B-4bit", // the shared base
05});
06
07// Fetch + pack factors into shared GPU buffers. Once per adapter.
08await engine.registerAdapter("support-router", "hf:acme/support-router-lora");
09await engine.registerAdapter("pii-redactor", "hf:acme/pii-redactor-lora");
10
11engine.getRegisteredAdapters(); // ["support-router", "pii-redactor"]

• ·Sources are the same as the rest of the adapter system: a Hugging Face repo (hf:owner/repo), a local path, or a Gerbil Tune output. Private repos honor the engine's hfToken.
• ·Throws if the batch-LoRA path isn't enabled (needs GERBIL_BATCH=N and GERBIL_BATCH_LORA=1) or if no adapter exists at the source.
• ·Distinct from the runtime overlay. loadAdapter hot-swaps one adapter onto the warm base for single-sequence generation (one specialist at a time, serialized). registerAdapter is the concurrent path: many specialists resident at once, selected per lane.

Per-lane adapters on generateBatch

generateBatch decodes N prompts in lockstep: one batched dispatch stream per step, one token per lane. The adapters option assigns each lane a registered adapter by name; null lanes decode the bare base. Mixed lanes, with different adapters and no-adapter lanes in the same batch, are the intended shape.

01// GERBIL_BATCH=4 GERBIL_BATCH_LORA=1
02const results = await engine.generateBatch(
03  [
04    "Route this ticket: my invoice is wrong and I was double charged",
05    "Draft a reply for: where is my order?",
06    "Redact PII: John Smith, john@acme.com, +1 555 0100, ...",
07    "Summarize: ...",
08  ],
09  {
10    // adapters[i] selects lane i's specialist; null = bare base.
11    adapters: ["support-router", null, "pii-redactor", null],
12    maxTokens: 256,
13    sampling: { temperature: 0 }, // batched decode is greedy-only today
14  },
15);
16
17// results[i] is a normal GenerateResult for lane i.

The correctness contract is the load-bearing part: a lane running adapter A is token-exact against single-sequence generation with adapter A applied. Lanes never interact; the base GEMM stays shared across the batch and each lane adds only its own gathered low-rank correction. Concurrency changes throughput, never output.

Current limits

• ·Node (Dawn) only. Batched decode ships on the Node backend today. The kernels are the same WGSL the browser runs, but browser batch widths (especially phone-class WebKit) are unmeasured. Browser support is coming.
• ·Fixed batch width. generateBatch requires exactly N prompts for GERBIL_BATCH=N; use maxTokensPerRow for per-lane budgets. A planned scheduler lifts the fixed-width restriction with dynamic admission.
• ·Greedy-only. Pass sampling: { temperature: 0 }. Sampled batched decode comes later.
• ·Text models only. Multimodal models, PLE models, and engines with a runtime-overlay adapter (loadAdapter) are rejected on the batch path for now.
• ·Specialists are pre-tuned. There is no per-request training, no gradients in the serving path, and no “learns as it goes”. At request time the engine only selects among resident adapters.

What's coming

What is planned next, in order (names may change):

• ·Adapter on the scheduler. scheduler.submit(prompt, { adapter: "billing" }) adds per-request adapter selection with dynamic admission instead of fixed lockstep batches. Serving many product features through one engine becomes the default shape.
• ·Request groups. submitGroup / cancelGroup group many lanes into one logical task: joint cancellation, per-task stats, and (later) shared-prefix KV reuse across a fan-out.
• ·gerbil.swarm(). The SDK primitive: declare members, a plan (input → sub-tasks routed to members), and a reduce (sub-results → answer); the fan-out runs as one batched pass. A swarm declared like a model, run like a function.

01// Sketch of the planned SDK primitive (not shipped yet)
02const swarm = await g.swarm({
03  members: {
04    router:    "hf:acme/intent-router-lora",
05    billing:   "hf:acme/billing-expert-lora",
06    extractor: "hf:acme/order-extractor-lora",
07  },
08  plan: (input) => [
09    { member: "extractor", prompt: extractPrompt(input) },
10    { member: "billing",   prompt: billingPrompt(input) },
11  ],
12  reduce: (results) => mergeStructured(results),
13});
14
15const answer = await swarm.run(ticket); // fan-out = ONE batched pass

When to reach for it

Swarms fit decomposable tasks with production shape: classification cascades and intent routing, multi-aspect extraction, map-reduce over documents, ensemble voting, and agentic fan-out. They are not a substitute for a frontier model on open-ended novel reasoning: a 0.8B base with rank-16 adapters doesn't become GPT-class because thirty-two of it run at once. For tasks that decompose, though, tuned specialists on one device are faster, cheaper, private, and offline.

To build your first specialist, start with Gerbil Tune. Every Tune job outputs an adapter on a shared base, which is exactly the artifact registerAdapter takes. For single-adapter usage without batching, see Adapters; for engine sharing and memory budgeting, see Concurrency & Memory.