LLMInference forSiliconArchitecture

First full Rust implementation, with no MLX or any tensor framework. Everything built from scratch on Metal.

<10 MB binary

$brew install agnosticeng/lisa/lisa

From scratch, without the bloat

The first full Rust inference engine, with zero ML frameworks and zero bindings. A small set of crates you can build on top of.

The first full Rust implementation

No MLX, no bindings, no tensor framework. Rust talks straight to the Metal C API through objc2 and compiles 85 .metal shaders at startup. Everything is built from scratch: one binary, no runtime to install.

crates/
  lisa-mlx/      tensor runtime + Metal backend
  lisa-engine/   the inference library
  lisa-serve/    OpenAI-compatible server
  lisa-cli/      the `lisa` binary

Lean crates to build on

Four sharp crates, minimal deps: objc2, block2 and half for the backend, tokenizers + serde elsewhere. Import lisa-engine as a library and ship without the bloat.

[dependencies]
lisa-engine = { path = "crates/lisa-engine" }
lisa-serve  = { path = "crates/lisa-serve" }

# backend pulls only:
#   objc2 · block2 · half
# engine pulls only:
#   tokenizers · serde

Continuous batching

Cohort, ragged, and continuous batching with a scheduler that keeps the GPU saturated, with multi-turn sessions and chunked prefill for long contexts.

# serve any HF repo id or local dir
lisa serve --model qwen3.8-flash-next \
  --addr 127.0.0.1:8000

# new requests join mid-flight, no restart

OpenAI-compatible API

Drop-in POST /v1/chat/completions with any OpenAI client, plus POST /v1/decisions for typed decision models like laya.

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [{"role": "user",
      "content": "Explain MTP speculation in one sentence."}],
    "max_tokens": 64
  }'

125B

params running on your Mac

~2200

tok/s prefill on M5 Max

~100

tok/s MTP depth-5 decode

ML frameworks or bindings

Run your first model

We've optimized the engine and kernels for these checkpoints, end to end.

M5+qwen4

qwen3.8 flash next

Qwen 3.8 Flash-Next text tower (125B total / A6B active). Our reference model: hybrid GDN + attention, 512-expert MoE, MTP speculative decode.

M3+qwen3.5

qwen3.8 27b

Dense 27B checkpoint with MTP head. The sweet spot for 16–32 GB Macs: full speed on M3 and above, 4-bit on disk.

M1+ModernBERT-large

laya

Typed decision model from Conv AI. Non-generative, served via POST /v1/decisions.

Incoming work ↗

Benchmarks

qwen3.8 flash next, 4-bit MTP checkpoint, measured on an M5 Max (128 GB). Higher is better.

Mode	Decode	Prefill
MTP decode (depth 5)	98–107 tok/s	~2200 tok/s
Serial decode	68–72 tok/s	~2200 tok/s

Ship your first endpoint in minutes

Install with Homebrew and serve an OpenAI-compatible API on your Mac, with no ML runtime required.

Quickstart Star on GitHub