Firecrawl's pdf-inspector Parses PDFs 30x Faster By Refusing to Use OCR or ML at All

Most of the AI document-parsing conversation in 2026 has been about more — bigger vision models, more expensive OCR passes, more infrastructure to babysit. Firecrawl, the web-scraping-for-LLMs company behind the firecrawl crawler, just open-sourced something that does the opposite. pdf-inspector is a Rust library that reads a PDF's internal structure — its fonts, its text operators, its xref tables — decides in single-digit-to-low-double-digit milliseconds whether the document actually needs OCR, and if it doesn't, extracts clean, structured Markdown without ever touching a neural network.

It picked up over 15,000 GitHub stars and 4,000+ in a single week, which puts it solidly in "newly significant" territory rather than "steady, boring utility library." But the interesting part isn't the star count. It's the bet embedded in the architecture: that the AI industry's default reflex — route every unstructured input through a model — is frequently the wrong call for PDFs, and that a few hundred lines of pattern-matching against PDF content streams can outperform it on speed, and in this benchmark, even on structural accuracy.

What it actually does

pdf-inspector solves a narrower problem than it might sound like from the name. It does not do OCR. It does not do layout understanding via vision transformers. It does one thing: given a PDF, decide what kind of PDF it is — text-based, scanned, image-based, or a mix — and then, for the pages that already contain machine-readable text, extract that text with position awareness and turn it into Markdown.

The reasoning behind this scope, according to the project's README, is a specific number: roughly 54% of PDFs that flow through real-world document pipelines already have embedded, extractable text. They're invoices generated by accounting software, reports exported from Word or LaTeX, contracts drafted in Google Docs and printed to PDF. None of that needs OCR. Yet a huge share of RAG and document-ingestion pipelines treat "PDF" as a single category and send every page through the same expensive OCR or vision-model step regardless, because building a reliable classifier is more work than just always calling the model.

pdf-inspector is that classifier, productized and open-sourced, plus a fast extractor for the pages that don't need the expensive path.

How it works

The library runs a two-stage pipeline, and the design choice that matters most is that both stages share a single document load — the PDF is parsed once and reused, instead of being opened once for classification and again for extraction, which is a common inefficiency in glue-code pipelines that chain a classifier and a separate parser.

Detection stage. Before doing any real extraction work, pdf-inspector scans page content streams for the presence of text operators (Tj/TJ in PDF syntax) versus image operators (Do, referring to embedded XObjects). A page with real text operators and font references is almost certainly extractable natively; a page that's dominated by a single large image XObject and has no text operators is almost certainly a scan. This scan is what runs in single-digit-to-low-double-digit milliseconds — it doesn't decode fonts, resolve CMaps, or lay out text, it just walks the operator stream.

Classification isn't all-or-nothing at the document level, either. The library exposes a pages_needing_ocr list, so a 40-page contract with three scanned exhibit pages at the end can route 37 pages through fast native extraction and only the 3 image pages to whatever OCR service the caller has configured. That per-page routing is arguably the actual product here — most pipelines make one binary decision for an entire file and eat the OCR cost for the whole thing if even one page is a scan.

Classification is configurable via scan strategies: EarlyExit (the default — stop scanning as soon as a non-text page is found, optimized for speed), Full (scan every page, needed to distinguish "Mixed" documents from purely "Scanned" ones accurately), Sample(n) (spot-check n distributed pages on very large PDFs where scanning everything would be wasteful), and Pages(vec) (check specific page numbers only). This is a real trade-off surface, not a toggle for show: EarlyExit is fast but will mis-tag a mixed document as fully text-based if the scanned pages come after the point it stops looking, which matters if you're deciding whether to budget for OCR at all.

Extraction stage. For pages classified as text-based, pdf-inspector goes deeper: resolving fonts (including CID/Type0 fonts with ToUnicode CMap decoding, which matters for anything with non-Latin scripts or embedded subset fonts), tracking X/Y positions of text runs, and reconstructing reading order — including detecting newspaper-style multi-column layouts and right-to-left text. That positioned text then feeds a Markdown converter that infers headings from font-size ratios, detects bold/italic from font name patterns, finds tables via both PDF drawing-operator geometry and text-alignment heuristics, and picks out lists, code blocks, and captions.

Notably, the library also flags broken font encodings — a class of bug where a PDF's font mapping is corrupted or non-standard and text extraction would silently produce garbage. Instead of returning garbled Unicode and calling it done, pdf-inspector detects the encoding failure and signals that the page should fall back to OCR. That's a small design decision, but it's the kind of thing that separates a demo-quality parser from one that's safe to run unattended in a pipeline.

The whole thing is built on a single dependency, lopdf, for low-level PDF object parsing — everything above that (classification heuristics, layout reconstruction, Markdown generation) is Firecrawl's own Rust code. No ONNX runtime, no bundled model weights, no GPU requirement.

Bindings ship for Python (via PyO3), Node.js/Bun (via napi-rs), and — notably — browser WebAssembly, with embedded CMaps so it works fully offline with no server round trip. There's also a native Rust crate on crates.io and two CLI tools, pdf2md and detect-pdf. Same core engine, five different places to call it from.

The benchmark, and what it actually shows

Firecrawl ran pdf-inspector against four other local (non-cloud, non-OCR) parsing engines on the opendataloader-bench corpus — 200 PDFs, scored on reading order (NID), table structure (TEDS), heading detection (MHS), and wall-clock speed, on an Apple M4 Pro, refreshed July 31, 2026:

A few things worth pulling out rather than skimming past. First, the overall-score gap between pdf-inspector and liteparse is tiny (0.875 vs 0.873) — this is essentially a tie on quality, and liteparse actually wins on heading detection. The real separation is speed: pdf-inspector finishes the full 200-document corpus in under half a second, about 1.6x faster than liteparse and over 36x faster than PyMuPDF4LLM. Second, the table-extraction score (TEDS) is where pdf-inspector pulls furthest ahead of everything except liteparse — 0.814 versus 0.401 for PyMuPDF4LLM and 0.273 for MarkItDown. If your pipeline deals with financial statements, invoices, or any document where tables carry the actual information, that gap is the one to pay attention to, not the aggregate score.

Read this benchmark for what it is, though: it excludes cloud/model-based parsers entirely (no Docling, no LlamaParse, no Marker, no AWS Textract or Google Document AI) — it's a comparison among engines that don't use ML models for parsing, which is the category pdf-inspector is explicitly built for. It's also a single 200-document corpus; Firecrawl publishes the full harness so you can reproduce it against your own document mix, which is the right response to "benchmarks always favor the vendor that wrote them," but it's still worth running yourself before trusting it for a specific document type you care about (dense academic papers with equations, government forms, non-Latin-script documents).

What changed versus the rest of the field

The PDF-to-LLM-input space has three broad camps, and pdf-inspector doesn't try to compete in two of them — which is itself the point.

Cloud OCR/document AI (AWS Textract, Google Document AI, Adobe PDF Extract API) — accurate on genuinely scanned or handwritten documents, but every page costs money and a network round trip, and you're sending potentially sensitive documents to a third party.

ML-based local parsers (Docling from IBM, Marker, unstructured) — bundle layout-detection and OCR models that run locally, which is a real privacy and cost win over cloud APIs, but they still pay a model-inference cost on every page, text-based or not, and they typically need more RAM/CPU (or a GPU) and a Python environment with heavier dependencies.

Rule-based/heuristic parsers (PyMuPDF4LLM, MarkItDown, pdf-inspector, liteparse) — no models, pure parsing logic against the PDF's own internal structure. Fast and cheap, but only as good as the heuristics, and structurally incapable of reading an actual scanned image.

pdf-inspector's contribution to that third camp is that it treats classification as a first-class output, not an afterthought. PyMuPDF4LLM and MarkItDown will both dutifully try to extract text from a scanned PDF and hand you back mostly-empty or garbled Markdown; they don't tell you "this page needs OCR," they just fail quietly. pdf-inspector's pages_needing_ocr list and confidence score are the connective tissue that lets you build a pipeline that routes — cheap path by default, expensive path only when the document actually demands it — instead of a pipeline that picks one tool and hopes.

None of this makes pdf-inspector a replacement for Docling or Textract — it can't read a scan, full stop. What it changes is the default: instead of every document going through the ML or cloud path, only the documents that actually need it do, and you find out which ones in milliseconds instead of after a full OCR pass fails to improve on garbled native-text extraction.

Developer experience: what calling it actually looks like

The API surface is intentionally small, and it's the same shape across every binding. In Python:

import pdf_inspector

result = pdf_inspector.process_pdf("document.pdf")
print(result.pdf_type)   # "text_based", "scanned", "image_based", "mixed"
print(result.markdown)   # Markdown string or None

In Node.js, the equivalent is processPdf() from @firecrawl/pdf-inspector, returning .pdfType and .markdown on the same object shape. In the browser, the WebAssembly build exposes an init() call followed by the same processPdf() function operating on raw bytes fetched client-side — no server, no upload. That consistency matters more than it sounds: a team prototyping in Python and shipping a browser-based version of the same feature doesn't have to reconcile two different classification behaviors or two different Markdown output formats, because it's the same Rust engine underneath in both cases.

One detail that's easy to miss on a skim: the result object doesn't just hand back a pass/fail. It returns the PDF type, a confidence score, and — when relevant — the markdown, in one call. There's no separate "call classify, then call extract" round trip required for the common case, though the library does expose that split explicitly (classify_pdf / detect-pdf versus the full process_pdf pipeline) for callers who want to make the OCR-routing decision before paying for extraction on pages they're going to discard anyway.

On the project-health side, at the time of writing pdf-inspector sits at roughly 15,100 stars against 447 commits, 63 open issues, and 87 open pull requests — a ratio that suggests active usage generating real bug reports rather than a quiet library nobody's actually running in anger. That's a reasonable health signal for a library that's a few months into a growth spike, though it's also a reason to expect the API surface to still move before it settles into a 1.0 with firmer stability guarantees.

Why this matters beyond the benchmark table

Cost. If ~54% of a document corpus is genuinely text-based, routing intelligently instead of defaulting to OCR-everything cuts the OCR/vision-model bill roughly in half before you've optimized anything else. At cloud OCR pricing or GPU-hours for a self-hosted vision model, that's not a rounding error at scale.

Latency. For interactive use — a user uploads a PDF and expects a response in seconds, not after a queued OCR job — the difference between "this page classifies and extracts in milliseconds" and "this page waits in an OCR queue" is the difference between a snappy product and a spinner.

Lock-in and privacy. Running entirely offline, including in a browser tab via WebAssembly, means a document never has to leave the user's machine for the common case. That's a meaningfully different privacy posture than "upload to a cloud API," and it removes a vendor dependency for the majority of documents.

Maintainability. A single Rust crate with one dependency is a very different operational surface than a Python service bundling PyTorch and gigabytes of model weights. Fewer moving parts, no GPU provisioning, no model version drift to track.

The trade-off, of course, is that none of this helps you with the ~46% of documents that actually are scanned — you still need an OCR or ML-based fallback for those, which means pdf-inspector is realistically a front door, not a complete pipeline. Its value is entirely in how much traffic it diverts away from that expensive fallback.

Practical use cases

• RAG ingestion pipelines — classify incoming documents and only pay for OCR/vision-model inference on the subset that needs it, per page rather than per file.
• Browser-based document tools — client-side PDF-to-Markdown conversion with no server round trip and no document upload, useful for anything handling sensitive files (legal, medical, financial).
• Pre-flight triage in front of an existing OCR pipeline — drop pdf-inspector in front of an existing Textract/Document AI/Docling setup purely as a cost-control filter, with zero change to the downstream OCR logic.
• CLI batch conversion — pdf2md for one-off or scripted bulk conversion of reports, papers, and contracts to Markdown for feeding into any text-based tool, LLM or otherwise.
• Encoding-failure detection as a signal, not just a fallback trigger — a spike in flagged broken-encoding pages across a document set can indicate a batch of PDFs was generated by a buggy tool, which is useful operational information independent of the OCR question.
• Compliance-sensitive document handling — legal discovery, medical records intake, or financial audit workflows where "this document never left our infrastructure, and for most of the batch it never left the browser" is a real answer to a real question from a security or compliance review, not just a marketing line.
• Cost estimation before committing to a cloud OCR contract — run pdf-inspector's classification over a representative sample of your actual document corpus before signing up for per-page Textract or Document AI pricing, so the OCR budget is sized against the real scanned-page percentage instead of a worst-case assumption that everything needs it.

What the docs and hype don't dwell on

The README is admirably direct about scope, but a few things are worth stating more bluntly than the marketing copy does:

• It cannot read scanned documents, at all. This isn't a limitation to work around later — it's the entire premise. If your document corpus is mostly scans (older archives, faxed contracts, handwritten forms), pdf-inspector's value collapses toward zero and you should reach for an ML-based or cloud parser directly.
• Heuristic Markdown conversion has known failure modes. Font-size-ratio heading detection and text-alignment-based table detection are heuristics, not semantic understanding — dense academic PDFs with unusual typesetting, PDFs generated by unusual toolchains, or documents that abuse font substitution for stylistic effect can confuse any heuristic-based extractor, pdf-inspector included. The benchmark's 0.788–0.811 heading scores (not 1.0) and 0.814 table score reflect that this is good, not perfect.
• The benchmark is one 200-document corpus on one machine. It's a reasonable, reproducible signal, not a guarantee that your specific document distribution behaves the same way. The presence of a reproducible-results branch is a genuine plus here — verify against your own corpus before trusting the numbers for a production cost model.
• EarlyExit is the default and it's a real accuracy trade-off, not just a performance knob — it can misclassify a document with scanned pages late in the file. Anyone deploying this at scale needs to actually read the scan-strategy documentation rather than accepting the default blind, especially if getting the OCR routing decision wrong is costly (e.g., silently skipping OCR on a scanned exhibit at the end of a contract).
• This is infrastructure for a company that sells the next step. pdf-inspector and its sibling project AnyDoc are the open-source foundation under Firecrawl's paid /parse API, which layers custom OCR models on top for the pages this library flags as needing them. That's not a criticism — it's a well-executed open-core strategy, and the open half is genuinely useful standalone — but it's worth naming plainly rather than presenting the open-source release as pure altruism.

An independent read

The most interesting thing about pdf-inspector isn't the Rust performance story, even though "0.47 seconds instead of 17" is a great headline number. It's that Firecrawl built a genuinely useful piece of infrastructure by refusing to reach for a model. In an ecosystem where "AI agent," "AI-native," and "powered by a fine-tuned model" get bolted onto tools that don't need them, this is a library that gets more valuable specifically because it isn't one — no model weights to version, no GPU to provision, no inference cost on the majority path.

The benchmark against liteparse also deserves a note of skepticism rather than uncritical repetition: a 0.875-vs-0.873 overall score is not a meaningful win on quality, and the honest framing of this release is "matches the best rule-based competitor on accuracy, wins clearly on speed" — not "beats everyone at everything," which is not quite the story either README or the surrounding blog coverage leads with. That's a strong result on its own terms; it doesn't need inflating.

The classification-and-routing idea is the part I'd bet travels furthest, independent of this specific library. Treating "does this page need OCR" as a fast, cheap, first-class decision — rather than a binary file-level assumption baked into pipeline architecture — is the kind of unglamorous engineering that saves real money at scale and rarely gets a launch blog post of its own. Firecrawl gave it one anyway, wrapped in a fast Rust parser, and that combination is likely why it's the library that's actually catching on rather than staying a footnote in someone's internal tooling.

Who should try it, wait, or skip it

Try it now if you're running any PDF ingestion pipeline (RAG, document search, contract analysis) that currently sends every file through OCR or a vision model regardless of content — dropping pdf-inspector in as a pre-filter is low-risk and the cost savings are close to immediate. Also worth it if you're building browser-based document tooling where offline, no-upload processing is a feature, not just a nice-to-have.

Wait and watch if your corpus is genuinely mixed and you need airtight Mixed-vs-Scanned classification — run with the Full scan strategy and validate against your own documents before trusting the default EarlyExit behavior in production, and keep an eye on how the project handles edge cases as its issue tracker matures past its current early-growth spike.

Skip it if your documents are predominantly scanned, handwritten, or otherwise non-native-text — you need an ML-based or cloud OCR pipeline as the primary tool here, not a fast pre-classifier for a case that barely applies to you.

What's your current PDF-to-text approach costing you — in OCR API spend, in GPU-hours, or in pipeline complexity — and would a fast, free classification step actually change that bill, or is your document mix mostly scans where this wouldn't move the needle at all?

Sources:

• pdf-inspector on GitHub
• opendataloader-bench corpus
• Firecrawl
• lopdf (Rust PDF parsing dependency)
• PyMuPDF4LLM
• MarkItDown
• Docling
• LlamaParse
• Marker
• unstructured