Colibri Complete Guide — Run a 744B Parameter Model on 25GB RAM

TL;DR: Colibri is a revolutionary MoE (Mixture of Experts) inference engine written in pure C. It runs entirely without a GPU, needs only 25GB of RAM, and can smoothly run ultra-large models like GLM-5.2 (744B parameters) on consumer hardware. It treats VRAM, RAM, and disk as a single unified memory hierarchy through its innovative "expert streaming" technique, striking a perfect balance between performance and resource usage.

What Is Colibri?

Colibri (named after the hummingbird) is an open-source project by JustVugg. It tackles the ultimate challenge of deploying large language models: how do you run ultra-massive models on severely limited hardware?

Traditionally, running a 744B-parameter model would require terabytes of VRAM and an expensive cluster of A100s or H100s. Colibri flips that assumption entirely:

  • Tiny Engine, Immense Model: The entire engine is a single C file (c/glm.c, roughly 2,400 lines) with zero external dependencies (no BLAS, no Python at runtime).
  • Expert Streaming: The model's 21,504 routing experts (~19MB each) are never all loaded into memory at once. Instead, they are streamed from disk on demand, optimized with an LRU cache and the OS page cache.
  • Memory Hierarchy: VRAM (if available), RAM, and SSD are treated as one unified, manageable memory pool. The model gracefully degrades when resources are tight, but it never sacrifices accuracy or correctness.

Core Technical Highlights

Technology Description Advantage
MLA Attention Uses GLM-5.2's native MLA (Multi-Layer Attention) architecture with a compressed KV-Cache (576 floats/token vs. 32,768). KV-Cache reduced by 57×, massive memory savings.
DeepSeek-V3-Style Router Adopts the same sigmoid router as DeepSeek-V3, supporting shared experts and dense layers in the first 3 layers. More precise expert routing, better model output.
MTP Speculative Decoding Leverages GLM-5.2's built-in Multi-Token Prediction (MTP) heads for speculative decoding. Achieves 39–59% acceptance rates, averaging 2.2–2.8 tokens per forward pass. Significantly boosts generation speed.
Grammar-Forced Speculation Supports GBNF grammar enforcement. In structured-output scenarios like JSON or function calling, acceptance rates approach 100%. Extreme efficiency on targeted tasks.
Integer Dot-Product Kernels Implements int8 and packed int4 matrix-multiplication kernels using AVX2 maddubs, running 1.4–2.5× faster than floating-point operations. Fully exploits CPU compute power for faster inference.
DSA Sparse Attention Fully implements GLM-5.2's DSA (Dynamic Sparse Attention) indexer, selecting only the top 2,048 causal keys per layer. Drastically reduces compute while preserving model quality.

Quick Start: Three Steps to Deploy

1. Environment Setup

Colibri has minimal requirements — just a modern Linux/macOS system and a GCC compiler.

# Ubuntu/Debian
sudo apt update && sudo apt install -y build-essential curl git

# macOS (Homebrew)
brew install gcc git

2. Clone and Build

# Clone the repository
git clone https://github.com/JustVugg/colibri
cd colibri

# Build (CPU-only by default)
make

# Or, if you have an NVIDIA GPU and want CUDA acceleration (optional)
# make COLI_CUDA=1

3. Run the Model

# Start interactive chat
./coli chat

# Or run batch inference
./coli batch --prompt "Write a poem about spring in Chinese."

💡 Tip: The first run requires downloading model weights (~370GB), so prepare in advance. Subsequent runs are blazingly fast.

4. Download Model Weights

Colibri uses pre-converted int4-quantized models, available directly from Hugging Face:

# Recommended version (int8 MTP heads, supports speculative decoding)
# https://huggingface.co/mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp

# Download with huggingface-cli
pip install huggingface_hub
huggingface-cli download mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp \
  --local-dir ./glm52-int4

⚠️ Critical Warning: MTP heads must be the int8 version!

The most common community question — "Why is my MTP acceptance rate 0%?" — almost always comes down to downloading the wrong model version. The original version (jlnsrk/GLM-5.2-colibri-int4) ships with int4-quantized MTP heads, which completely breaks speculative decoding (0% acceptance rate) and costs you roughly 2× the performance.

How to verify: check the out-mtp-* file sizes: - int8 (correct): 3527131672 / 5366238584 / 1065950496 - int4 (wrong): 1765523544 / 2686077736 / 536747200

Deep Dive: How Does Colibri Work?

Colibri's power lies in its elegant memory management and algorithmic design. Let's break down the core workflow:

  1. Startup & Initialization: On boot, the engine automatically calculates the expert cache size based on the system's MemAvailable, ensuring the OOM Killer is never triggered.
  2. Expert Loading: When the model needs a specific expert, the engine reads its weights from disk. To reduce I/O wait, it uses the WILLNEED system call for asynchronous expert readahead.
  3. Compute Execution: Loaded expert weights feed into highly optimized integer dot-product kernels. Single-token decode uses f32 computation; batched prefill uses the faster int4 kernel.
  4. KV-Cache Persistence: The conversation's KV-Cache is compressed and persisted to a .coli_kv file. This means your conversation context stays "warm" even after closing and reopening the program — no need to re-prefill history.

This design ensures that when resources are constrained, Colibri's performance degrades gracefully rather than crashing or producing incorrect results.

KV-Cache Persistence: No Lost Conversations

One of Colibri's killer features is KV-Cache persistence. After each conversation, the compressed MLA KV-Cache is appended to a .coli_kv file (~182 KB/token, crash-safe). It's automatically restored on the next startup, skipping the expensive prefill for historical context.

# KV-Cache persistence is on by default
./coli chat

# To disable it
KVSAVE=0 ./coli chat

Router-Lookahead Prefetching (Experimental)

Colibri implements a clever optimization: the expert routing for the next layer is 71.6% predictable based on the current layer's post-attention state. With PILOT=1, a dedicated I/O thread prefetches the next layer's required experts while the current layer is still computing.

# Enable router-lookahead prefetching
PILOT=1 ./coli chat

Configuration Parameters at a Glance

Environment Variable Default Description
DRAFT 1 MTP speculative decoding toggle (0 = disabled)
DSA 1 DSA sparse attention toggle (0 = disabled, falls back to dense attention)
DSA_TOPK 2048 Number of top-K causal keys selected per layer in DSA
PILOT 0 Router-lookahead prefetching toggle
KVSAVE 1 KV-Cache persistence toggle
IDOT 1 Integer dot-product kernel toggle
COLI_CUDA 0 CUDA acceleration toggle (requires compile-time enablement)
GRAMMAR - Path to GBNF grammar file (for structured output)
GRAMMAR_DRAFT 24 Maximum grammar enforcement span per forward pass

Performance Benchmarks: Real-World Numbers

Official tests on WSL2 (12 cores, 25GB RAM, NVMe):

  • Cold-start time: ~32 seconds (model loading + cache initialization).
  • Resident memory: ~9.9 GB (int4 dense portion).
  • Peak disk usage: ~370 GB (all expert weights).
  • Generation speed: With MTP speculative decoding enabled and cache warmed up, 2.2–2.8 tokens per forward pass.

📊 Comparison: That's on par with running a 7B-parameter model on a high-end GPU — except Colibri is running a model 100× larger at 744B parameters!

GPU Acceleration Benchmarks: 6× RTX 5090

According to an official experiment report dated 2026-07-12, with all experts resident across 6× RTX 5090 (VRAM + RAM), single-request decode speed reaches 6.84 tok/s. This proves Colibri's elastic architecture — the same codebase scales seamlessly from pure CPU to multi-GPU setups.

Cold Cache vs. Warm Cache

Scenario Disk Read per Token Notes
Cold cache ~11 GB (75 layers × 8 experts) First inference — all experts loaded from disk
Warm cache Significantly reduced Frequently used experts already in RAM
Full GPU resident ~0 All experts in VRAM, zero disk I/O

💡 SSD Notes: Colibri's streaming is read-only and won't meaningfully wear your SSD. What to actually watch out for: (1) swap traffic from insufficient system RAM (writes wear SSDs); (2) elevated SSD temperatures from sustained heavy reads. Colibri's automatic memory-budgeting mechanism avoids swap by design.

How Colibri Compares to Other Inference Engines

Feature Colibri llama.cpp Ollama
Language Pure C (~2,400 lines) C/C++ Go + llama.cpp
Target Model GLM-5.2 (744B MoE) General (LLaMA series primarily) General
GPU Required No (CUDA optional) Recommended Recommended
Memory Needed 25GB RAM Depends on model size Depends on model size
Expert Streaming ✅ Core feature ❌ ❌
KV-Cache Persistence ✅ ❌ ❌
MTP Speculative Decoding ✅ Native support Partial models Partial models
DSA Sparse Attention ✅ ❌ ❌
External Dependencies Zero BLAS, etc. More

🔍 Positioning: Colibri is not a llama.cpp replacement — it's a specialized tool for a specific scenario. If you need to run ultra-large MoE models on consumer hardware, Colibri is currently the only viable option.

Practical Use Cases

Scenario 1: Local AI Assistant (CPU-Only)

Perfect for developers without a GPU who want a powerful local AI assistant on a laptop or desktop:

# Start an interactive conversation
./coli chat

# JSON output with grammar constraints
./coli chat --grammar schemas/response.gbnf

Scenario 2: Structured Data Extraction

Leverage Grammar-Forced Speculation for peak performance in JSON and function-calling scenarios:

# Define a GBNF grammar file
cat > schema.gbnf << 'EOF'
root ::= "{" ws "\"name\"" ws ":" ws string "," ws "\"age\"" ws ":" ws number ws "}"
string ::= "\"" [^"]* "\""
number ::= [0-9]+
ws ::= [ \t\n]*
EOF

# Run grammar-constrained inference
GRAMMAR=schema.gbnf ./coli batch \
  --prompt "Extract info from the following: John Doe, 28, software engineer"

Scenario 3: Multi-GPU Cluster Inference

For users with GPU resources, Colibri supports hybrid deployment:

# Build the CUDA version
make COLI_CUDA=1

# Run (automatically pins hot experts to GPU VRAM)
COLI_CUDA=1 ./coli chat

FAQ

Q: My machine only has 16GB of RAM. Can I run it? A: Yes, but the experience will be limited. Colibri's minimum requirement is fitting the dense portion (~9.9GB int4), plus the KV-Cache and working buffers. With 16GB you can run it, but the expert cache will be smaller and cold starts will happen more frequently.

Q: How large does my SSD need to be? A: Model weights are ~370GB (int4 quantized). Add in the KV-Cache and temporary files, and you'll want at least 500GB of free space. NVMe SSDs are strongly recommended — SATA SSDs will work but will be slower.

Q: Does it support Windows? A: Official support is available through WSL2 (Windows Subsystem for Linux). Native Windows compilation isn't provided yet, but the C code should theoretically compile under MSVC or MinGW.

Q: Is it compatible with llama.cpp's GGUF format? A: No. Colibri uses its own int4 container format, specifically optimized for MoE expert streaming. You'll need the official FP8→int4 conversion tool.

Conclusion: A Milestone for AI Democratization

Colibri is more than a technical curiosity. It represents an important direction for AI development: decentralization and democratization. It proves that cutting-edge AI capability is no longer monopolized by a handful of tech giants with bottomless hardware budgets. Any ordinary developer with a laptop can now explore and apply the most advanced large-model technology.

Colibri's success is the perfect marriage of engineering elegance and algorithmic ingenuity. It didn't chase "bigger" — it chased "smarter." It reminds us that real innovation is often born from respecting resource constraints and pursuing efficiency to its extreme.

References

This article is based on Colibri v1.0 (2026-07-01), under Apache License 2.0.