What Is AirLLM?

AirLLM is an open-source Python library that tackles a seemingly impossible challenge: running a 70B-parameter large language model on a GPU with just 4GB of VRAM.

A 70B model at FP16 precision typically needs about 140GB of VRAM, requiring at least two A100 GPUs (80GB each) just to load it. AirLLM uses Layer-wise Inference — loading only one model layer onto the GPU at a time, computing its output, then unloading it and moving on to the next. This brings VRAM usage down from 140GB to under 4GB.

As of July 2026, AirLLM has earned 23,000+ stars on GitHub, with PyPI downloads climbing steadily. It's one of the hottest local LLM inference tools out there right now.

Key Advantages

Feature Traditional Approach AirLLM
70B model VRAM requirement ~140GB (2× A100 80GB) ~4GB
671B (DeepSeek-V3) ~1.3TB (multi-GPU cluster) ~12GB
Quantization/distillation Required, loses precision Not needed (optional)
Hardware barrier Data-center GPUs Consumer-grade GPUs work fine
Setup complexity Complex (CUDA, multi-GPU config) One command: pip install airllm

How It Works: Layer-wise Inference

Traditional inference:  [Layer 1] → [Layer 2] → ... → [Layer 80]   ← all loaded into GPU
                         Needs ~140GB VRAM

AirLLM:                  [Layer 1] ← load into GPU → compute → save to CPU → free GPU
                         [Layer 2] ← load into GPU → compute → save to CPU → free GPU
                         ...
                         [Layer 80] ← load into GPU → compute → output result
                         Only ~4GB VRAM needed (single layer size)

The core idea: LLM inference is sequential — layer N's output is layer N+1's input. AirLLM exploits this by keeping only one layer on the GPU at a time, saving intermediate results back to CPU RAM, then freeing the GPU for the next layer.

💡 What's the trade-off? Speed. Since data shuttles back and forth between GPU and CPU for every layer, inference is slower than full GPU loading. But for individual developers, being able to run it at all matters more than running it fast.

Installation & Setup

Basic Install

# Create a virtual environment (recommended)
python -m venv airllm-env
source airllm-env/bin/activate

# Install airllm
pip install airllm

# If you want quantization acceleration, install bitsandbytes
pip install -U bitsandbytes

System Requirements

  • GPU inference: NVIDIA GPU, 4GB+ VRAM, CUDA support
  • CPU inference: Any x86_64 CPU (supported since August 2024)
  • macOS: Apple Silicon (M1/M2/M3), requires the mlx framework
  • Disk space: ~130GB for a 70B model (downloaded on first run)
  • RAM: 16GB+ system memory recommended (for caching intermediate results)

Verify Installation

from airllm import AutoModel
print("AirLLM installed successfully!")

Quick Start: Up and Running in 5 Minutes

Your First Hello World

from airllm import AutoModel

MAX_LENGTH = 128

# One line to load the model (auto-download + layer-wise processing)
model = AutoModel.from_pretrained("Qwen/Qwen3-32B")

# Prepare input
input_text = ["What is the capital of France?"]
input_tokens = model.tokenizer(
    input_text,
    return_tensors="pt",
    return_attention_mask=False,
    truncation=True,
    max_length=MAX_LENGTH,
    padding=False
)

# Generate
generation_output = model.generate(
    input_tokens['input_ids'].cuda(),
    max_new_tokens=50,
    use_cache=True,
    return_dict_in_generate=True
)

# Decode output
output = model.tokenizer.decode(generation_output.sequences[0])
print(output)

Example output:

What is the capital of France? The capital of France is Paris.

That's it. No manual multi-GPU config, no VRAM optimization code, no quantization — AirLLM's AutoModel handles everything automatically.

Supported Models

AirLLM's AutoModel auto-detects the model type and supports nearly all popular open-source LLMs:

Model Family Example Min VRAM
Qwen3 Qwen3-32B / Qwen3-235B ~3GB
DeepSeek DeepSeek-V3 (671B) ~12GB
Llama Llama 3.1 405B / Llama 3 70B ~8GB / ~4GB
Mistral Mixtral 8x7B ~4GB
ChatGLM ChatGLM3-6B ~4GB
Phi Phi-4 ~4GB
Gemma Gemma 2B/7B ~4GB
Baichuan Baichuan2-7B ~4GB

Advanced Features

1. 4-bit / 8-bit Quantization for Speed

AirLLM includes block-wise quantization, delivering up to 3× faster inference with minimal accuracy loss.

from airllm import AutoModel

# 4-bit quantization (fastest, smallest VRAM)
model = AutoModel.from_pretrained(
    "Qwen/Qwen3-32B",
    compression='4bit'  # or '8bit'
)

Why does AirLLM's quantization lose so little precision? Traditional quantization needs to compress both weights and activations to speed up computation. But AirLLM's bottleneck is disk I/O (loading layers from disk), so only weights need quantizing — which is far easier to do without accuracy loss.

2. Prefetching

Prefetching loads the next layer into memory while the GPU computes the current one, overlapping computation with I/O for roughly 10% speed improvement.

from airllm import AutoModel

# Prefetching is on by default (supported by AirLLMLlama2)
model = AutoModel.from_pretrained(
    "meta-llama/Llama-2-70b-hf",
    hf_token="your_huggingface_token",  # required for gated models
)

3. Configuration Reference

model = AutoModel.from_pretrained(
    "model_name_or_path",

    # Quantization: '4bit' / '8bit' / None
    compression='4bit',

    # Profiling: prints timing for each stage
    profiling_mode=True,

    # Layer shards save path (optional)
    layer_shards_saving_path='/data/airllm_cache/',

    # HuggingFace token (required for gated models)
    hf_token="hf_xxxx",

    # Prefetching (enabled by default)
    prefetching=True,

    # Delete original model files after conversion (saves disk space)
    delete_original=True,
)

4. Multi-turn Conversation Example

from airllm import AutoModel

model = AutoModel.from_pretrained("Qwen/Qwen3-32B")

def chat_with_model(prompt, history=None):
    """Simple single-turn conversation"""
    if history:
        full_prompt = history + "\nUser: " + prompt + "\nAssistant: "
    else:
        full_prompt = f"User: {prompt}\nAssistant: "

    input_tokens = model.tokenizer(
        [full_prompt],
        return_tensors="pt",
        return_attention_mask=False,
        truncation=True,
        max_length=512,
        padding=False
    )

    generation_output = model.generate(
        input_tokens['input_ids'].cuda(),
        max_new_tokens=256,
        use_cache=True,
        return_dict_in_generate=True,
        temperature=0.7,
        top_p=0.9,
    )

    response = model.tokenizer.decode(generation_output.sequences[0])
    # Extract Assistant's response
    if "Assistant:" in response:
        response = response.split("Assistant:")[-1].strip()

    return response

# Test it
print(chat_with_model("Explain quantum computing in three sentences"))
print(chat_with_model("What's the most elegant way to sort in Python?"))

Real-World Use Cases

Use Case 1: Local Knowledge Base Q&A

Combine with an embedding model to build a fully local RAG (Retrieval-Augmented Generation) system:

from airllm import AutoModel
# Use with LangChain
from langchain.llms import HuggingFacePipeline
from langchain.embeddings import HuggingFaceEmbeddings
from langchain.chains import RetrievalQA

# Load AirLLM model as LLM backend
model = AutoModel.from_pretrained("Qwen/Qwen3-32B", compression='4bit')

# Pair with a local vector database (e.g. FAISS)
# Enables fully offline document Q&A, code retrieval, etc.

Use Case 2: Edge Device Deployment

Raspberry Pi 5 + USB AI accelerator, or Jetson Nano — edge computing scenarios:

# On an edge device
pip install airllm
# CPU inference mode, no GPU required

from airllm import AutoModel

# Run a small model on CPU
model = AutoModel.from_pretrained("microsoft/Phi-3-mini-4k-instruct")

# Suitable for on-device AI inference on IoT
# Smart home voice control, industrial inspection, etc.

Use Case 3: Local Inference on MacBook

Apple Silicon users can run large models directly:

# On macOS, install mlx first
# pip install mlx

from airllm import AutoModel

# Run on M1/M2/M3 MacBook
model = AutoModel.from_pretrained("Qwen/Qwen3-32B")

# Unified memory architecture delivers near-native inference speed

AirLLM vs Other Local Inference Options

Approach Mechanism Min VRAM Speed Accuracy Ease of Use
AirLLM Layer-wise loading 4GB Slower ✅ Lossless ⭐⭐⭐⭐⭐
llama.cpp GGUF quantization 4GB Fast Quantized loss ⭐⭐⭐⭐
vLLM PagedAttention 80GB+ Blazing ✅ Lossless ⭐⭐⭐
Ollama GGUF wrapper 4GB Fast Quantized loss ⭐⭐⭐⭐⭐
ExLlamaV2 4-bit GPTQ 24GB+ Blazing Quantized loss ⭐⭐⭐

How to choose: - VRAM < 8GB: AirLLM (lossless) or llama.cpp/Ollama (quantized) - VRAM 8–24GB: AirLLM + 4-bit quantization, or ExLlamaV2 - Multi-GPU / data center: vLLM (production-grade serving) - Want simplicity: Ollama (one-click start) or AirLLM (one pip command)

Benchmark References

Community benchmark data (varies by hardware and model version — use as rough guide):

Model Hardware tokens/sec Notes
Llama-3-8B RTX 3060 12GB ~8–12 4-bit quantized
Llama-3-70B RTX 3060 12GB ~1–2 4-bit quantized
Qwen3-32B GTX 1650 4GB ~1–2 No quantization
DeepSeek-V3 671B RTX 4090 24GB ~0.3 ~12GB VRAM used

⚠️ Note: AirLLM's inference speed is indeed slower than full GPU loading (typically 1–3 tokens/sec). But its value lies in letting ordinary hardware run big models — great for development, local experiments, and offline inference.

FAQ

Q: Can AirLLM train models?

No. AirLLM is inference-only — it doesn't support training or fine-tuning. For fine-tuning, use LoRA/QLoRA instead.

Q: Why is my model running so slowly?

This is expected with AirLLM's layer-by-layer loading. Ways to speed things up: 1. Use 4-bit quantization (compression='4bit') 2. Enable prefetching (prefetching=True, on by default) 3. Store model files on an SSD (reduces disk I/O latency)

Q: Why does the first run take so long?

On first run, AirLLM downloads the model and splits it layer by layer into the cache directory. A 70B model downloads ~130GB, and splitting takes time too. Second runs reuse the cache and are much faster.

Q: What if I'm running out of disk space?

# Set delete_original=True to remove the original HuggingFace model after splitting
model = AutoModel.from_pretrained(
    "Qwen/Qwen3-32B",
    delete_original=True  # saves ~50% disk space
)

Summary

AirLLM's core value in one sentence: letting every developer run large language models on their own machine.

Dimension Rating
🎯 Core highlight 4GB VRAM runs 70B, no quantization needed
📦 Install difficulty pip install, zero config
🧪 Model coverage All major open-source models
⚡ Inference speed Slower (1–3 tok/s), but it runs
🔧 Best for Dev/debug, local experiments, edge deployment
📄 License Apache 2.0 / BELLE License

Next steps: - GitHub repo — 23K+ stars, actively maintained - HuggingFace blog — deep technical walkthrough - Colab demo — try it free online

If you found this guide helpful, feel free to share it with fellow developers!