🐶 Labomaru’s Quick Take & Specs

“By discarding general-purpose runtime abstractions, Perplexity’s Lily engine turns Apple Silicon into a zero-overhead local inference engine. It trades flexibility for raw, metal-level speed on Qwen3.6-35B! 🐶⚡”

• 🏢 Developer / Lab: Perplexity AI
• 🚀 Engine Category: Specialized Local MoE / Hybrid Inference Engine
• ⚡ Core Performance Delta: +77.4% Prefill Speed (512 tokens), +89% GPU-Internal Routing Speed
• 🛠️ Runtime Environment: macOS 15+ (Apple Silicon, Metal 3, Rust Toolchain)
• 📜 License: Apache License 2.0 (100% Open Source)
• 💰 Memory Footprint: 19.4 GB Unified Memory (4-bit Groupwise Quantization)

Executive Summary & Production Impact (TL;DR)

In September 2026, Perplexity AI open-sourced Lily, a bespoke Rust and Metal inference engine tailored explicitly for the Qwen3.6-35B-A3B model on Apple Silicon. Rather than attempting to serve as a universal runtime like vLLM, llama.cpp, or Ollama, Lily takes a radically opinionated architecture path: complete specialization for a single model structure and hardware target.

Qwen3.6-35B-A3B utilizes a complex hybrid architecture featuring 35B total parameters with approximately 3B active parameters per token, structured across 256 fine-grained experts (8 selected plus 1 shared expert), 10 Grouped-Query Attention (GQA) layers, and 30 Gated DeltaNet state-space layers. Running such a model across generalized local framework stacks often results in high scheduling overhead and inefficient memory transfers.

Lily bypasses PyTorch, MLX, and general C++ graph abstractions. By pairing a zero-dependency Rust executor with hand-rolled Metal Shader Language (MSL) kernels, Lily compresses the model’s 70 GB bfloat16 weights into a 19.4 GB 4-bit groupwise affine format. It executes prefill and decoding phases with near-zero host CPU overhead, delivering up to +77.4% faster prefill speeds and +89% faster expert routing directly inside Apple Silicon Unified Memory.

+-------------------------------------------------------------------------+
|                        Lily Engine Architecture                         |
|                                                                         |
|  +--------------------+      Zero Sync Pipeline     +----------------+  |
|  |   Rust Runtime     |---------------------------->|   Metal GPU    |  |
|  | (Axum HTTP Server) |  Greedy Token Feedback      | (Unified RAM)  |  |
|  +--------------------+  [Next Input Slot Write]    +----------------+  |
|                                                             |           |
|  +----------------------------------------------------------+           |
|  | MSL Kernel Execution                                                 |
|  |  * Fused Grouped GEMM (SRAM 4-bit Unpack -> FP32 Accumulate)         |
|  |  * GPU-Resident MoE Top-K Routing (No CPU Roundtrip)                 |
|  |  * Register-Resident Gated DeltaNet State Scan                       |
|  +----------------------------------------------------------------------+  |
+-------------------------------------------------------------------------+

---

The Catch & Reality Check (Constraints, Mode Gaps & Benchmarks)

While Lily achieves breakthrough throughput metrics on M-series chips, engineers evaluating Lily for production workloads must understand its severe constraints:

1. Zero Generalization: Lily cannot load Llama 3, Mistral, or standard dense Qwen models. It is hardcoded strictly for the 4-bit layout of Qwen3.6-35B-A3B.
2. Greedy Sampling Only: The runtime lacks support for temperature, top-p, top-k, or speculative decoding. It operates solely in greedy deterministic generation mode.
3. Minimalist API Surface: The embedded HTTP engine provides an OpenAI-compatible endpoint, but lacks advanced batching, tool-calling schema parsing, or multi-tenant queue orchestration.
4. Host OS & Memory Floor: Requires macOS 15 or newer with a minimum of 24 GB Unified Memory (32 GB recommended to preserve overhead for system buffers).

+-----------------------+-----------------------+-----------------------+
| Feature / Dimension   | Standard Frameworks   | Lily Engine           |
|                       | (MLX / llama.cpp)     |                       |
+-----------------------+-----------------------+-----------------------+
| Model Support         | Multi-Architecture    | Qwen3.6-35B-A3B Only  |
| Sampling Control      | Full (Temp, Top-P/K)  | Deterministic Greedy  |
| Prefill Speed (512 t) | Baseline              | +77.4% Faster         |
| Expert Routing Latency| Host CPU Synchronized | 100% On-GPU Routing   |
| Dependency Overhead   | Python / Heavy C++    | Standalone Rust + MSL |
+-----------------------+-----------------------+-----------------------+

---

Behavior & Interaction Design (Agent Safety & Workflow Shift)

From an agentic workflow and systems perspective, Lily introduces a zero-synchronization execution pipeline. Standard LLM serving engines synchronize CPU and GPU states at every generated token to extract the output token ID, run logit processing, and re-inject the payload into the next forward pass.

Lily breaks this pattern by implementing GPU-resident feedback loops:

• Token Writing: Generated token IDs are directly written into the next input slot within Apple Silicon Unified Memory without CPU roundtrips.
• Stream Emission: The Rust process monitors GPU completion flags asynchronously to stream text back over HTTP SSE while the Metal GPU pipelines the subsequent step uninterrupted.
• Deterministic Execution: The forced greedy sampling guarantee eliminates non-deterministic variance in multi-step reasoning agents, ensuring predictable operational behavior.

---

Implementation & Minimal Reproducible Code

Deploying Lily requires Rust (cargo) and Apple XCode command-line tools installed on macOS 15+.

Installation & Compilation

# Clone Perplexity Garden repository
git clone https://github.com/perplexityai/pplx-garden.git
cd pplx-garden/lily

# Build the optimized release binary
cargo build --release --features metal

Running the Engine & Executing Inference

# Start the server with quantized weights (automatically downloads if missing)
./target/release/lily-server \
  --model-path ./weights/qwen3.6-35b-a3b-q4.bin \
  --port 8000 \
  --ctx-len 4096

Interacting via OpenAI-Compatible Endpoint

import requests
import json

url = "http://localhost:8000/v1/chat/completions"
headers = {"Content-Type": "application/json"}

payload = {
    "model": "qwen3.6-35b-a3b",
    "messages": [
        {"role": "system", "content": "You are a systems performance engineer."},
        {"role": "user", "content": "Explain how 4-bit groupwise quantization reduces memory bus pressure on Apple Silicon."}
    ],
    "stream": True
}

response = requests.post(url, headers=headers, data=json.dumps(payload), stream=True)

for line in response.iter_lines():
    if line:
        decoded = line.decode('utf-8')
        if decoded.startswith("data: ") and decoded != "data: [DONE]":
            content = json.loads(decoded[6:])["choices"][0]["delta"].get("content", "")
            print(content, end="", flush=True)
print()

---

Cost-Benefit Matrix & Benchmarks (As of September 05, 2026)

Lily transforms local execution unit economics by replacing $1.50–$3.00/hour cloud GPU API costs with zero operational expenses on local Mac hardware.

+------------------------------------+-----------------------+-----------------------+
| Metric / Subsystem                 | Standard MLX Setup    | Lily Engine (Metal)   |
+------------------------------------+-----------------------+-----------------------+
| Model Weights Size                 | ~22.1 GB (Q4_K_M)     | 19.4 GB (Affine Q4)   |
| Prefill Throughput (512 tokens)    | 142 tok/s             | 252 tok/s (+77.4%)    |
| MoE Routing Latency (Top-K Expert) | 1.82 ms / layer       | 0.20 ms / layer       |
| 2K Token Prefill Tiling Gain       | Baseline Tile         | +13.2% (32-Row Tile)  |
| Gated DeltaNet Scan Overhead       | Global DRAM Access    | Register Resident     |
| Operational Hosting Cost           | Cloud API ~$0.002/1k  | $0.00 (Local Hardware)|
+------------------------------------+-----------------------+-----------------------+

Deep Tech Hack: Threadgroup Memory Dequantization

During Grouped GEMM execution across the 256 fine-grained MoE experts, Lily performs weight unpacking solely inside Metal threadgroup (SRAM) memory. The 4-bit quantized weights are expanded to FP32 accumulators directly within registers, preventing unpacked weight matrices from ever spilling back out to Unified Main Memory.

---

Community Insights & Field-Tested Optimizations

Feedback from engineering teams testing Lily in early September 2026 highlights several practical takeaways:

1. Unified Memory Allocation Hints: On 32 GB Macs, setting sysctl iogpu.wired_mem_limit ensures the macOS window server does not cause context switches during heavy GEMM prefill spikes.
2. Thermals & Sustained Throughput: Running prolonged 16K context completions on MacBook Pro laptops triggers thermal throttling after ~4 minutes. Mac Studio units maintain flat 100% Metal core utilization indefinitely.
3. Agent Integration: Developers building local codegen agents pair Lily’s deterministic greedy output with fast local testing harnesses, bypassing volatile API response variance.

---

Adoption Checklist: When to Adopt vs. Pass

Adopt If:

• You run local workflows explicitly backed by Qwen3.6-35B-A3B.
• You require local, high-speed prefill latency for long context or code analysis.
• You want zero external cloud dependencies or API billing meters.
• You are deploying dedicated single-purpose background indexing or streaming services on Apple Silicon hardware.

Pass If:

• You need to run multiple model architectures (e.g., Llama 3.1, Claude/GPT API fallbacks).
• Your workflow relies heavily on non-zero temperature sampling or dynamic top-p exploration.
• Your Mac has less than 24 GB of Unified Memory.
• You require complex enterprise API middleware (JSON schema enforcement, multi-tenant token buckets).

---

Frequently Asked Questions (FAQ)