{
 "measurement": "What preserving PDF structure costs in tokens: pdftotext against firecrawl/pdf-inspector on the same PDF",
 "claude_code_version": "2.1.251 (Claude Code)",
 "subject": {
  "repo": "firecrawl/pdf-inspector",
  "commit": "23cf1ad",
  "stars_at_trial": 17147,
  "license": "MIT",
  "python_package": "pdf-inspector 1.17.0 from PyPI, zero dependencies, 11 MB venv",
  "fixture": "a 10-page text-based PDF, 124,310 bytes, classified text_based with confidence 1.00"
 },
 "why": "Feeding a PDF to a model means extracting text first, and the extractor you pick changes both what survives and what you pay. This site already used pdftotext for its book work. A trending Rust extractor claims structure preservation, so the question is what that structure costs.",
 "results": {
  "speed_median_of_7": {
   "classify_pdf_ms": 0.6,
   "extract_text_ms": 2.9,
   "extract_pages_markdown_ms": 6.4,
   "pdftotext_ms": 14.5,
   "vendor_claims": "classification ~10-50ms, text-based PDFs under 200ms",
   "verdict": "Both claims hold with margin, and classification BEATS the stated band: 0.6ms against a claimed 10-50ms. Plain-text extraction is 2.9ms against pdftotext's 14.5ms, about 5x faster, on this file."
  },
  "what_survives": {
   "pdftotext": {
    "chars": 4454,
    "markdown_headings": 0,
    "table_rows": 0
   },
   "pdf_inspector_markdown": {
    "chars": 4795,
    "markdown_headings": 60,
    "table_rows": 40
   },
   "the_decisive_case": "The PDF contains a colour table. pdf-inspector reconstructs it as |Deep Navy|Teal|Seafoam|Cream| over |#1a2332|#2d8b8b|#a8dadc|#f1faee|, preserving which hex belongs to which name. pdftotext emits the names and the hex codes as separate disconnected lines, so the column-to-value association is destroyed.",
   "a_count_i_had_to_correct": "Grepping '^#' on the pdftotext output returns 40 lines and looks like 40 headings. They are hex colour codes. Counting '^#{1,6} ' gives the real answer: zero. Reported because the wrong count would have flattered pdftotext."
  },
  "what_structure_costs": {
   "pdftotext": {
    "tokens": 1986,
    "bytes": 4454,
    "tokens_per_byte": 0.4459,
    "mode_seen": "6/6"
   },
   "inspector_markdown": {
    "tokens": 2162,
    "bytes": 4795,
    "tokens_per_byte": 0.4509,
    "mode_seen": "5/6"
   },
   "extra_tokens": 176,
   "extra_percent": 8.9,
   "plain_statement": "Keeping the tables and headings costs 176 extra tokens, 8.9%, on this PDF. The per-byte rates are nearly identical (0.4459 against 0.4509), so the extra cost is the extra characters rather than markdown tokenising worse."
  }
 },
 "method": {
  "population": "one extraction as CLAUDE.md in an otherwise empty project, priced against an empty floor from the SAME round",
  "estimator": "guarded mode over 6 rounds",
  "rounds": 6,
  "controls": [
   "paired floor per round",
   "one constant working directory, created and removed per run, because the floor moves with the path",
   "arrival control: reply exactly OK",
   "fixture byte size asserted before every run",
   "--setting-sources project asserted onto the command line",
   "result_guard refuses an unusable result rather than recording a zero",
   "timings are the median of 7 repeats, with the extracted TEXT asserted identical across repeats so a timing is never taken from a run that produced different output"
  ]
 },
 "known_limits": {
  "ONE PDF, AND A SIMPLE ONE": "A 10-page ReportLab-generated, text-based PDF with one table shape. Nothing here covers scanned PDFs, multi-column academic papers, or the OCR path.",
  "THE OCR PATH WAS DELIBERATELY NOT RUN": "The Python package is built with the ocr feature, which pulls in model download over HTTPS and an ONNX runtime. Models resolve lazily and no model cache directory existed before or after this trial, which was checked. Local model inference is out of scope here, so process_pdf_with_ocr was never called and nothing in this corpus describes OCR quality or cost.",
  "THE PYTHON BUILD IS NOT THE RUST DEFAULT": "Cargo.toml sets default = [] and comments that network and inference stay opt-in so 'default library, renderer-only, and browser consumers do not inherit inference or HTTP/TLS'. The PyPI package is built with python = [pyo3, ocr], so a pip install ships the OCR surface that a default cargo build does not. That is a real difference for anyone assuming pip install gives them the local-only library.",
  "TIMINGS ARE ONE MACHINE, WARM": "Medians of 7 repeats in one process on one machine. Absolute milliseconds do not transfer; the 5x ratio against pdftotext on the same file and machine is the transferable part.",
  "MORE CHARACTERS IS NOT AUTOMATICALLY BETTER": "This measures what survives and what it costs. It does not measure whether a model ANSWERS better from the structured version, which is the question a reader actually has and is not tested here.",
  "one_machine_one_model": "Opus on Claude Code 2.1.251, project scope, one machine."
 },
 "runs": [
  {
   "arm": "floor",
   "round": 0,
   "status": "ok",
   "context": 17050,
   "delta": 0,
   "reply_exact_ok": true,
   "fixture_bytes": 0
  },
  {
   "arm": "pdftotext",
   "round": 0,
   "status": "ok",
   "context": 19036,
   "delta": 1986,
   "reply_exact_ok": true,
   "refused_because": null,
   "fixture_bytes": 4454
  },
  {
   "arm": "inspector",
   "round": 0,
   "status": "ok",
   "context": 19212,
   "delta": 2162,
   "reply_exact_ok": true,
   "refused_because": null,
   "fixture_bytes": 4795
  },
  {
   "arm": "floor",
   "round": 1,
   "status": "ok",
   "context": 17050,
   "delta": 0,
   "reply_exact_ok": true,
   "fixture_bytes": 0
  },
  {
   "arm": "pdftotext",
   "round": 1,
   "status": "ok",
   "context": 19036,
   "delta": 1986,
   "reply_exact_ok": true,
   "refused_because": null,
   "fixture_bytes": 4454
  },
  {
   "arm": "inspector",
   "round": 1,
   "status": "ok",
   "context": 19212,
   "delta": 2162,
   "reply_exact_ok": true,
   "refused_because": null,
   "fixture_bytes": 4795
  },
  {
   "arm": "floor",
   "round": 2,
   "status": "ok",
   "context": 17050,
   "delta": 0,
   "reply_exact_ok": true,
   "fixture_bytes": 0
  },
  {
   "arm": "pdftotext",
   "round": 2,
   "status": "ok",
   "context": 19036,
   "delta": 1986,
   "reply_exact_ok": true,
   "refused_because": null,
   "fixture_bytes": 4454
  },
  {
   "arm": "inspector",
   "round": 2,
   "status": "ok",
   "context": 19212,
   "delta": 2162,
   "reply_exact_ok": true,
   "refused_because": null,
   "fixture_bytes": 4795
  },
  {
   "arm": "floor",
   "round": 3,
   "status": "ok",
   "context": 17050,
   "delta": 0,
   "reply_exact_ok": true,
   "fixture_bytes": 0
  },
  {
   "arm": "pdftotext",
   "round": 3,
   "status": "ok",
   "context": 19036,
   "delta": 1986,
   "reply_exact_ok": true,
   "refused_because": null,
   "fixture_bytes": 4454
  },
  {
   "arm": "inspector",
   "round": 3,
   "status": "ok",
   "context": 19212,
   "delta": 2162,
   "reply_exact_ok": true,
   "refused_because": null,
   "fixture_bytes": 4795
  },
  {
   "arm": "floor",
   "round": 4,
   "status": "ok",
   "context": 17050,
   "delta": 0,
   "reply_exact_ok": true,
   "fixture_bytes": 0
  },
  {
   "arm": "pdftotext",
   "round": 4,
   "status": "ok",
   "context": 19036,
   "delta": 1986,
   "reply_exact_ok": true,
   "refused_because": null,
   "fixture_bytes": 4454
  },
  {
   "arm": "inspector",
   "round": 4,
   "status": "ok",
   "context": 18315,
   "delta": 1265,
   "reply_exact_ok": true,
   "refused_because": null,
   "fixture_bytes": 4795
  },
  {
   "arm": "floor",
   "round": 5,
   "status": "ok",
   "context": 17050,
   "delta": 0,
   "reply_exact_ok": true,
   "fixture_bytes": 0
  },
  {
   "arm": "pdftotext",
   "round": 5,
   "status": "ok",
   "context": 19036,
   "delta": 1986,
   "reply_exact_ok": true,
   "refused_because": null,
   "fixture_bytes": 4454
  },
  {
   "arm": "inspector",
   "round": 5,
   "status": "ok",
   "context": 19212,
   "delta": 2162,
   "reply_exact_ok": true,
   "refused_because": null,
   "fixture_bytes": 4795
  }
 ]
}