pdf-inspector Complete Guide: Firecrawl's Open-Source High-Performance PDF Parsing Engine in Rust

Introduction: The Pain Points of PDF Parsing

As developers, processing PDFs is an unavoidable requirement — invoice parsing, contract extraction, paper crawling, data archiving. Nearly every backend project bumps into this at some point. But PDF parsing has long been a headache:

• OCR services are expensive: Calling cloud OCR APIs costs a few cents per page, and the bill adds up fast at scale
• Open-source solutions are slow: PyMuPDF4LLM takes 17 seconds to process 200 documents, and MarkItDown takes 16 seconds
• Formatting gets lost: Traditional extraction tools spit out plain text — heading hierarchy, table structure, and code blocks are all gone
• No smart routing: They can't automatically distinguish scanned PDFs from native text PDFs, so even documents that don't need OCR still go through the OCR pipeline

In July 2026, the well-known web crawling platform Firecrawl open-sourced a Rust-based PDF parsing library called pdf-inspector. Within just a few weeks, it racked up 7,900+ Stars and landed in the top three of GitHub Trending's daily list. It promises to complete local PDF classification and text extraction within 200ms, automatically outputting formatted Markdown — no OCR, no external services needed.

Today, let's take a deep dive into this project and see how it manages to outperform mainstream solutions.

What Is pdf-inspector?

pdf-inspector is a PDF classification and text extraction library written in Rust by the Firecrawl team. Its core capabilities include:

1. Smart classification — Detects PDF type in 10–50ms: native text (TextBased), scanned (Scanned), image-based (ImageBased), or mixed (Mixed), returning confidence scores and per-page OCR routing suggestions
2. Text extraction — Position-aware text extraction with font information, X/Y coordinates, and automatic multi-column reading order
3. Markdown conversion — Automatically recognizes H1–H4 headings, lists, code blocks (via monospace font detection), tables, bold/italic, URL links, and page breaks
4. Table detection — Dual-mode table detection (rectangle detection based on PDF drawing operations + heuristic detection based on text alignment), handling financial reports and cross-page tables flawlessly
5. CID font support — ToUnicode CMap decoding, supporting Type0/Identity-H fonts, UTF-16BE, UTF-8, and Latin-1 encoding
6. Multi-language bindings — Provides Python, Node.js, and browser WebAssembly bindings

Why Rust? Rust's zero-cost abstractions and memory safety let pdf-inspector achieve extreme parsing speed without relying on ML models or external services. The entire library has just one external dependency: lopdf (a Rust PDF parsing library).

Performance Comparison

Based on official benchmarks on Apple M4 Pro (200 documents, opendataloader-bench corpus):

pdf-inspector leads across the board in composite score, reading order, table recognition, and speed — taking just 1/36 of PyMuPDF4LLM's time and 1/34 of MarkItDown's.

Installation

pdf-inspector supports multiple installation methods, covering Rust, Python, Node.js, and browser environments.

Rust (Native)

Add the dependency in your Cargo.toml:

[dependencies]
pdf-inspector = "0.2"

Or install via the cargo CLI:

cargo add pdf-inspector

Python

pdf-inspector provides Python bindings via maturin:

pip install maturin
git clone https://github.com/firecrawl/pdf-inspector.git
cd pdf-inspector
maturin develop --release

💡 Note: pdf-inspector hasn't been published to PyPI yet, so you'll need to build from source. The Firecrawl team plans to release a PyPI package once it's stable.

Node.js

npm install @firecrawl/pdf-inspector

Browser WebAssembly

npm install @firecrawl/pdf-inspector-wasm

The Wasm version embeds the full Rust parser in the browser and can run in Web Workers without server round-trips.

Quick Start

Python Example

After installation, it's dead simple — just one function call:

import pdf_inspector

# Process a single PDF file
result = pdf_inspector.process_pdf("document.pdf")

# View the PDF type classification
print(f"PDF type: {result.pdf_type}")
# Output: "text_based", "scanned", "image_based", or "mixed"

# Get the Markdown output
if result.markdown:
    print(result.markdown)

That's it. One line of code completes classification + extraction + Markdown conversion.

Node.js Example

const { processPdf } = require('@firecrawl/pdf-inspector');
const fs = require('fs');

async function analyzePdf(filePath) {
  const pdfBuffer = fs.readFileSync(filePath);
  const result = await processPdf(pdfBuffer);

  console.log('PDF type:', result.pdfType);
  console.log('Confidence:', result.confidence);
  console.log('--- Markdown output ---');
  console.log(result.markdown);
}

analyzePdf('report.pdf');

Rust Native Example

use pdf_inspector::{PdfInspector, ProcessPdfResult};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let inspector = PdfInspector::new();
    let result: ProcessPdfResult = inspector.process_pdf_path("document.pdf")?;

    println!("PDF type: {:?}", result.pdf_type);
    println!("Confidence: {:.2}", result.confidence);

    if let Some(markdown) = result.markdown {
        println!("Markdown:\n{}", markdown);
    }

    Ok(())
}

Core Features Explained

1. Smart Classification: Avoid Unnecessary OCR Overhead

One of pdf-inspector's most practical features is automatic classification. By sampling the PDF content stream, it determines the document type in 10–50ms:

Real-world scenario: Building a smart PDF processing pipeline

import pdf_inspector
# Assuming you have your own OCR function
def smart_pdf_pipeline(pdf_path, ocr_function):
    result = pdf_inspector.process_pdf(pdf_path)

    if result.pdf_type == "text_based":
        # Native text — extract directly: fast, free, accurate
        print("✅ Native text PDF, direct extraction")
        return result.markdown

    elif result.pdf_type == "scanned":
        # Scanned — call OCR
        print("🔍 Scanned PDF, calling OCR")
        return ocr_function(pdf_path)

    elif result.pdf_type == "mixed":
        # Mixed — process page by page
        print("📄 Mixed PDF, per-page routing")
        pages = []
        for page_info in result.page_results:
            if page_info.needs_ocr:
                pages.append(ocr_function_for_page(pdf_path, page_info.page_number))
            else:
                pages.append(page_info.markdown)
        return "\n\n".join(pages)

    else:
        # Pure image-based — cannot extract text
        print("❌ Image-based PDF, cannot extract text")
        return None

This pipeline lets you completely skip OCR for ~54% of PDFs (the native text ones), significantly cutting processing costs and latency.

2. Markdown Conversion: Preserving Document Structure

pdf-inspector's Markdown output isn't just plain text — it intelligently identifies and preserves document structure:

• Heading hierarchy (H1–H4): Automatically inferred from font size ratios
• Lists: Bullet lists, numbered lists, and letter-numbered lists recognized automatically
• Code blocks: Marked as code blocks via monospace font detection
• Tables: Supports both rectangle detection and heuristic detection, handling financial reports and cross-page tables
• Bold/italic: Formatting preserved
• URL links: Automatically identified and converted to Markdown link format
• Page breaks: Separated with --- between pages

Here's what it looks like in practice. Say you have an academic paper PDF:

# Deep Learning Approaches for Natural Language Processing

## 1. Introduction

Natural language processing (NLP) has seen remarkable progress in recent years...

### 1.1 Background

The transformer architecture, introduced by Vaswani et al., has become...

## 2. Methodology

| Model | BLEU Score | Training Time |
|-------|-----------|---------------|
| Transformer | 38.2 | 12 hours |
| BERT | 41.0 | 24 hours |
| GPT-4 | 45.7 | 72 hours |

## 3. Results

The experimental results demonstrate...

> **Note:** All experiments were conducted on...

Heading hierarchy, tables, and block quotes are all preserved intact. That's a huge step up from traditional PDF extractors that only output plain text.

3. Multi-Column and Reading Order Detection

Many PDFs — like papers, newspapers, and magazines — use multi-column layouts. pdf-inspector automatically detects column structures and extracts in the correct reading order:

result = pdf_inspector.process_pdf("newspaper.pdf")

# pdf-inspector automatically detects two-column / three-column layouts
# and outputs in the correct top-to-bottom, left-to-right reading order
print(result.markdown)

It also supports RTL (right-to-left) text, suitable for Arabic, Hebrew, and other scripts.

4. CID Fonts and Encoding Issue Detection

When handling Chinese, Japanese, or Korean PDFs, font encoding issues are common. pdf-inspector supports ToUnicode CMap decoding and can correctly handle:

• Type0 / Identity-H fonts
• UTF-16BE, UTF-8, and Latin-1 encoding
• Automatic flagging of broken font encodings

This means when processing CJK PDFs, it can tell you which pages might have garbled text risk, letting you decide in advance whether to fall back to OCR.

Advanced Practice: Building a PDF Processing Service

Let's build a complete PDF processing microservice that integrates pdf-inspector's smart classification capability.

Scenario: Document Archiving System

Say you're building an internal enterprise document archiving system that needs to process hundreds of PDFs daily (contracts, invoices, reports, etc.):

# pdf_service.py
import os
import json
from datetime import datetime
import pdf_inspector

class PdfProcessingService:
    def __init__(self, output_dir="./processed"):
        self.output_dir = output_dir
        os.makedirs(output_dir, exist_ok=True)

    def process_batch(self, pdf_files):
        """Batch process PDF files"""
        results = []
        for pdf_path in pdf_files:
            result = self.process_single(pdf_path)
            results.append(result)
        return results

    def process_single(self, pdf_path):
        """Process a single PDF file"""
        filename = os.path.basename(pdf_path)
        pdf_result = pdf_inspector.process_pdf(pdf_path)

        # Extract metadata
        metadata = {
            "filename": filename,
            "pdf_type": pdf_result.pdf_type,
            "confidence": pdf_result.confidence,
            "processed_at": datetime.now().isoformat(),
            "pages_count": len(pdf_result.page_results) if hasattr(pdf_result, 'page_results') else 0,
            "needs_ocr": pdf_result.pdf_type in ("scanned", "image_based"),
        }

        # Save Markdown output
        if pdf_result.markdown:
            md_filename = filename.replace(".pdf", ".md")
            md_path = os.path.join(self.output_dir, md_filename)
            with open(md_path, "w", encoding="utf-8") as f:
                f.write(pdf_result.markdown)
            metadata["markdown_path"] = md_path

        # Save metadata
        meta_path = os.path.join(
            self.output_dir,
            filename.replace(".pdf", ".meta.json")
        )
        with open(meta_path, "w", encoding="utf-8") as f:
            json.dump(metadata, f, ensure_ascii=False, indent=2)

        return metadata

# Usage example
service = PdfProcessingService(output_dir="./output")

# Batch processing
pdf_files = [
    "contracts/contract_2026_001.pdf",
    "invoices/invoice_aug_2026.pdf",
    "reports/q2_financial_report.pdf",
]

results = service.process_batch(pdf_files)
for r in results:
    status = "Needs OCR" if r["needs_ocr"] else "Extracted directly"
    print(f"{r['filename']}: {r['pdf_type']} ({status})")

Scenario: Integration with LLM Document Analysis Pipelines

The high-quality Markdown output from pdf-inspector can be fed directly to LLMs for analysis:

import pdf_inspector
# Assuming you use an OpenAI-compatible API
from openai import OpenAI

def analyze_pdf_with_llm(pdf_path, prompt="Summarize the main content of this document"):
    """Analyze PDF content using an LLM"""
    # 1. Extract Markdown with pdf-inspector
    result = pdf_inspector.process_pdf(pdf_path)

    if not result.markdown:
        return "Could not extract text content; OCR may be needed"

    # 2. If the document is long, truncate to the first 8000 characters
    markdown_content = result.markdown[:8000]

    # 3. Call the LLM for analysis
    client = OpenAI(api_key="YOUR_API_KEY")
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": "You are a document analysis assistant."},
            {"role": "user", "content": f"Please {prompt} based on the following document:\n\n{markdown_content}"}
        ]
    )

    return response.choices[0].message.content

# Usage
summary = analyze_pdf_with_llm("annual_report.pdf", "extract key financial metrics")
print(summary)

This kind of pipeline is especially useful in RAG (Retrieval-Augmented Generation) systems — quickly extract structured text with pdf-inspector, then vectorize and store it in a knowledge base.

Comparison with Other Solutions

pdf-inspector vs PyMuPDF4LLM

pdf-inspector vs MarkItDown (Microsoft)

When to Use pdf-inspector vs When to Use OCR?

• Use pdf-inspector: Native text PDFs, reports, papers, contracts, invoices — any document with text content streams
• Use OCR: Scanned documents, pure image PDFs, handwritten documents
• Use both together: Classify with pdf-inspector first, extract text_based directly, route scanned/image_based to OCR — that's exactly the routing strategy pdf-inspector was designed for

Project Info & Community

• GitHub: firecrawl/pdf-inspector
• Stars: 7,900+ (launched July 2026, growing fast)
• License: Apache-2.0 (commercial-friendly)
• Language: Rust
• Ecosystem: By Firecrawl (well-known web crawling platform, also the open-source force behind the firecrawl crawler engine)

Installation Channels

Summary

pdf-inspector fills an important gap — a lightweight, fast, localized PDF parsing solution. It doesn't need ML models or external services. Pure Rust implementation delivers accuracy well beyond comparable tools at a fraction of the time.

Key advantages at a glance:

1. 🚀 Blazing fast — 200ms-level parsing, 36× faster than PyMuPDF4LLM
2. 🧠 Smart classification — Automatically distinguishes text/scanned/image PDFs to optimize OCR routing
3. 📐 Structure preserved — Headings, tables, code blocks, lists all intact as Markdown
4. 🔒 100% local — No external dependencies, no API calls, your data never leaves your machine
5. 🌐 Multi-platform — Rust/Python/Node.js/browser Wasm, full coverage

If you're building document processing pipelines, RAG systems, or any application that needs to parse PDFs, pdf-inspector deserves a spot in your tech stack. Especially for scenarios dominated by native text PDFs, it can save you a ton of OCR cost and latency.

🔗 Related Links - GitHub: firecrawl/pdf-inspector - Python Docs: docs/python.md - Benchmarks: opendataloader-bench - Firecrawl: firecrawl.dev