Document Ingestion

Phase 9 · Document 01 · RAG Prev: 00 — RAG Overview · Up: Phase 9 Index

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

Ingestion is the first stage of RAG and the one that quietly determines your ceiling: garbage in, garbage out. If a PDF is parsed into jumbled text, a table becomes word soup, or you forget to attach source/acl/updated_at metadata, then no amount of clever retrieval, reranking, or prompting downstream can recover — you'll retrieve mangled chunks or leak documents a user shouldn't see. Ingestion is also where the un-glamorous production realities live: parsing dozens of file types, deduplication, incremental updates (re-ingesting only what changed), and capturing the metadata that powers ACL-aware retrieval (10, Phase 8.09) and freshness. Get ingestion right and the rest of the pipeline has a chance; get it wrong and you've capped your quality on day one.

2. Core Concept

Plain-English primer: turn messy sources into clean, tagged records

Ingestion is the offline ETL that converts raw sources (PDFs, web pages, Confluence/Notion exports, Slack, code, databases) into clean text records ready to chunk (02) and embed (03). Four steps:

1. Load — pull bytes from the source (filesystem, S3, an API/connector).
2. Parse/extract — get clean text out of the format (the hard part for PDFs/HTML/tables/images).
3. Clean/normalize — strip boilerplate (nav bars, headers/footers), fix encoding, normalize whitespace.
4. Attach metadata — source, title, acl/permissions, updated_at/version, section/page — the fields that drive filtering, freshness, and citations downstream.

@dataclass
class Document:
    content: str                 # clean text
    metadata: dict               # source, title, acl, updated_at, section, page, doc_id, hash

def ingest(path) -> Document:
    raw  = load(path)            # bytes
    text = parse(raw, path.suffix)   # format-specific extraction (the hard part)
    text = clean(text)           # de-boilerplate, normalize
    return Document(content=text, metadata=build_metadata(path, text))

Parsing is the hard part (especially PDFs)

Plain text/Markdown is easy. The real difficulty is lossy or structured formats:

• PDFs — text may be in arbitrary order, multi-column, or scanned images (need OCR). Tables and figures lose structure. Use a real PDF parser (PyMuPDF, pdfplumber, Unstructured) and, for scans, OCR (Tesseract, or a vision model).
• HTML — strip nav/ads/scripts; keep main content (readability extractors).
• Office docs (docx/pptx/xlsx) — extract text + structure; spreadsheets need special handling.
• Tables — preserve as Markdown/CSV so rows/columns stay meaningful (a table flattened to prose is unretrievable).
• Images/diagrams — caption with a vision model if they carry information.

A parser that produces coherent, reading-order text with structure preserved is worth more than any downstream tweak. Tools like Unstructured, LlamaParse, or Docling specialize in this.

Metadata is a first-class output, not an afterthought

Every record must carry metadata, because downstream stages depend on it:

• source / title / page / section → citations (08).
• acl / permissions / tenant → ACL-aware retrieval (filter so users only see what they're allowed to) (10, Phase 8.09).
• updated_at / version / hash → freshness and incremental updates.
• doc_id → grouping chunks back to their document.

Incremental ingestion + dedup (the production reality)

A corpus changes. Re-embedding everything on every change is wasteful, so ingestion must be incremental: detect what changed (content hash or updated_at), and only upsert new/changed docs and delete removed ones from the vector store (04). Deduplicate near-identical documents (same doc in two places) to avoid retrieving redundant chunks. This is an ongoing pipeline (scheduled or event-driven), not a one-time script — covered operationally in 10.

def incremental_upsert(doc, store):
    h = sha256(doc.content)
    if store.get_hash(doc.metadata["doc_id"]) == h:
        return "unchanged"                 # skip re-embedding
    store.delete_by_doc(doc.metadata["doc_id"])   # remove old chunks
    chunks = chunk(doc)                    # [02]
    store.upsert(embed(chunks), chunks, doc.metadata)  # [03,04]
    store.set_hash(doc.metadata["doc_id"], h)
    return "updated"

Connectors

In production, sources are systems, not folders: S3/GCS, Confluence, Notion, SharePoint, Google Drive, Slack, Jira, databases. Each needs a connector that authenticates, lists/pulls documents, respects permissions (capture the source ACL!), and supports incremental sync (changed-since). Frameworks (LlamaIndex/LangChain loaders, Unstructured connectors) provide many; the key is they bring the metadata and ACL along.

3. Mental Model

   SOURCES (PDF·HTML·docx·Confluence·Slack·DB·code)
        │  CONNECTOR (auth · list · pull · capture ACL · changed-since)
        ▼
   LOAD → PARSE/EXTRACT (the hard part: PDFs/OCR/tables/HTML) → CLEAN (de-boilerplate)
        → ATTACH METADATA {source,title,page,acl,updated_at,version,doc_id,hash}
        ▼
   Document(content, metadata) → [chunk 02 → embed 03 → store 04]

   INCREMENTAL: hash/updated_at → upsert changed, delete removed; DEDUP near-identical
   ★ garbage in = garbage out: parsing + metadata cap your whole pipeline's quality

Mnemonic: load → parse (the hard part) → clean → tag with metadata; ingest incrementally (hash/upsert) and bring ACL along. Parsing + metadata set your ceiling.

4. Hitchhiker's Guide

What to look for first: the parser quality for your dominant format (usually PDF/HTML) and the metadata schema (esp. acl, updated_at, source). These two cap quality and enable compliance/freshness.

What to ignore at first: building connectors for every system. Start with the filesystem/one source; add connectors as you productionize (10).

What misleads beginners:

• Treating PDF extraction as trivial. Bad PDF parsing (jumbled order, lost tables, no OCR for scans) silently destroys retrieval — inspect the extracted text.
• Forgetting metadata. No acl → data leaks; no updated_at → stale answers; no source → no citations (08, 10).
• Full re-ingest every time. Wasteful and slow — use incremental (hash/updated_at) upserts.
• Not deduplicating. The same doc in two places retrieves twice, crowding out diversity.
• Dropping ACL from the source. If you don't capture permissions at ingestion, you can't enforce them at retrieval.

How experts reason: they pick (or buy) a robust parser, inspect extracted text before trusting it, design a metadata schema first (source/acl/freshness/version/doc_id/hash), build incremental, ACL-preserving connectors, and dedup. They treat ingestion as a maintained data pipeline with monitoring, not a script.

What matters in production: parse fidelity (esp. tables/scans), complete + correct metadata (ACL!), incremental sync correctness (no stale/orphan chunks), dedup, and re-ingestion cadence/freshness (10).

How to debug/verify: dump the extracted text and eyeball it (is it coherent, in order, tables intact?); verify every chunk carries the expected metadata; confirm a changed doc updates and a deleted doc disappears from the store; confirm ACL is present.

Questions to ask: what formats dominate, and how well do we parse them (OCR for scans)? what metadata do we need for citations/ACL/freshness? incremental sync + dedup? do connectors preserve source permissions?

What silently gets expensive/unreliable: poor parsing (caps everything downstream), missing ACL (data leak), full re-ingestion (cost), orphaned chunks from deletes not propagating, and stale corpora.

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• Ingestion sets the quality ceiling — bad parsing or missing metadata can't be fixed downstream.
• Parsing is the hard part — PDFs (order/columns/tables/scans→OCR) and HTML (boilerplate) need real parsers, not naïve text reads.
• Tables/structure must be preserved (Markdown/CSV) or they become unretrievable word soup.
• Metadata is a first-class output: source/title/page (citations), acl (ACL-retrieval), updated_at/version/hash (freshness/incremental), doc_id.
• Capture ACL at ingestion from the source — you can't enforce permissions you didn't record (10, Phase 8.09).
• Use incremental sync (hash/updated_at → upsert/delete), not full re-ingestion.
• Deduplicate near-identical docs to keep retrieval diverse.
• Connectors (S3/Confluence/Notion/Slack/DB) must auth, sync incrementally, and preserve permissions.

9. Observations from Real Systems

• Most RAG quality complaints trace upstream to ingestion — a PDF parsed into jumbled text, or a table flattened to prose (00).
• Unstructured / LlamaParse / Docling exist precisely because robust multi-format parsing (esp. tables/scans) is hard and high-value.
• Enterprise RAG lives or dies on ACL — capturing source permissions at ingestion is mandatory for compliant retrieval (Phase 8.09, Phase 14.04).
• Freshness via updated_at + incremental sync is how production assistants avoid stale answers (10).
• Connectors (Confluence/Notion/SharePoint/Slack) are the bulk of real ingestion work — and the place permissions are most often dropped.

10. Common Misconceptions

11. Engineering Decision Framework

INGEST a corpus:
 1. INVENTORY sources + formats; pick parsers per format (real PDF/HTML parser; OCR for scans).
 2. PARSE → CLEAN: produce coherent reading-order text; preserve tables (Markdown/CSV). INSPECT output.
 3. METADATA SCHEMA (design first): source·title·page·section · ACL/tenant · updated_at·version · doc_id·hash.
 4. CONNECTORS: auth, list, pull, CAPTURE PERMISSIONS, support changed-since (incremental).
 5. INCREMENTAL: hash/updated_at → upsert changed, delete removed; DEDUP near-identical.
 6. HAND OFF clean Documents → chunk [02] → embed [03] → store [04].
 7. MONITOR: parse failures, missing metadata, sync lag, orphan chunks.

12. Hands-On Lab

Goal

Build an ingestion pipeline that parses mixed formats into clean, metadata-rich Documents with incremental updates — and prove parsing quality matters.

Prerequisites

• pip install pymupdf unstructured beautifulsoup4; a folder with a few PDFs (incl. one with a table), an HTML page, and Markdown.

Steps

1. Parse per format: extract text from each (PyMuPDF for PDF, BeautifulSoup/readability for HTML, plain read for MD); produce Document(content, metadata) with source, title, updated_at, doc_id, and a content hash.
2. Inspect quality: print the extracted text for the table-containing PDF. Is it coherent and reading-order? Does the table survive? Compare a naïve text dump vs a structure-aware parser (Unstructured/PyMuPDF layout) — note the difference.
3. Add ACL metadata: attach a synthetic acl (e.g., ["team:research"]) to some docs; this is what 10 filters on.
4. Incremental upsert: implement hash-based incremental_upsert (skip unchanged, replace changed, delete removed). Modify one file, re-run, and confirm only it is re-processed; delete one and confirm its records would be removed.
5. Dedup: add the same doc under two paths; detect and drop the duplicate by content hash.

Expected output

A set of clean Documents with complete metadata; a parsing-quality comparison (naïve vs structure-aware) on the table PDF; and a working incremental/dedup pass showing only-changed reprocessing.

Debugging tips

• Garbled/empty PDF text → wrong parser or a scanned PDF (needs OCR).
• Re-processing everything → your change-detection (hash/updated_at) isn't wired.

Extension task

Add OCR for a scanned PDF (Tesseract or a vision model) and confirm text is now extractable.

Production extension

Replace the folder with a real connector (e.g., Confluence/Notion) that preserves ACL + supports changed-since; schedule incremental syncs and monitor parse-failure/missing-metadata rates (10).

What to measure

Parse fidelity (coherent text? tables intact?), metadata completeness (esp. ACL), incremental correctness (only changed reprocessed; deletes propagate), dedup rate.

Deliverables

• An ingestion pipeline producing metadata-rich Documents.
• A parsing-quality comparison (naïve vs structure-aware) on a table PDF.
• An incremental + dedup demonstration.

13. Verification Questions

Basic

1. What are the four steps of ingestion?
2. Why is parsing (esp. PDFs) the hard part?
3. What metadata must every record carry, and why?

Applied 4. How do you preserve a table so it stays retrievable? 5. Design an incremental ingestion that avoids re-embedding unchanged docs.

Debugging 6. Retrieval returns mangled text. Where in ingestion do you look? 7. A user sees a document they shouldn't. What ingestion step failed?

System design 8. Design an ACL-preserving, incremental connector for Confluence feeding a RAG index.

Startup / product 9. Why does ingestion quality (parsing + metadata) cap your product's RAG quality and compliance posture?

14. Takeaways

1. Ingestion sets the quality ceiling — garbage in, garbage out; parsing is the hard part.
2. Preserve structure (tables/reading order); use real parsers + OCR for scans.
3. Metadata is first-class: source/page (citations), ACL (compliance), updated_at/version/hash (freshness/incremental).
4. Capture ACL at the source, ingest incrementally (hash/upsert), and dedup.
5. Connectors bring data and permissions — it's a maintained pipeline, not a one-time script.

15. Artifact Checklist

• An ingestion pipeline producing clean, metadata-rich Documents.
• A metadata schema (source/page, ACL, updated_at/version, doc_id, hash).
• A parsing-quality comparison on a table/complex doc.
• An incremental upsert + dedup demonstration.
• (Production) an ACL-preserving connector + freshness schedule.

Up: Phase 9 Index · Next: 02 — Chunking