90 PDFs In, 0 Fields Out: Putting Document Extraction Under a Data Contract

Picture a driver onboarding job at a ride-hailing company that reads drivers' insurance certificates. Ninety certificates land, ninety rows reach Bronze, and every policy_reference is null. The supplier moved the reference to a new line, the extraction regex stopped matching, and nothing in the pipeline treated "no value found" as a problem. The run is green. Operations finds out when a driver with lapsed cover is approved.

Documents fail differently from tables. A CSV with a missing column breaks loudly; a PDF parser that matches nothing returns an empty field and carries on. And the documents that matter most in a lakehouse (licences, certificates, contracts, invoices) are exactly the ones full of names, dates of birth and identifiers.

This article compares the usual ways to turn documents into governed rows, then shows how RideFlow, a fictional ride-hailing business, declares its extraction, failure handling and personal fields in data contracts. It covers predictable, templated documents. It does not claim one parser handles every layout, or that tagging a field as personal data makes a pipeline compliant.

Why document pipelines fail quietly

• No match is not an error. Extraction code commonly returns None when a pattern misses, so a layout change produces empty columns, not failures.
• The rules live in the script. What counts as a valid licence number sits in a regex inside a notebook, unreviewed by the team that owns onboarding.
• Personal data is untracked. The parsed table has holder_name and date_of_birth, but nothing records that they are personal, so they flow into Silver and Gold like any other column.

The ways teams solve it

Each can be the right answer. Models and cloud services win on varied layouts; people win when every document drives a decision. A gap these approaches can leave is the same: the extraction result is not checked against an agreed shape, and the personal fields are not declared where the next team will see them.

How RideFlow declares it

RideFlow's Operations domain receives insurance certificates as PDFs and driving licences as photos. Both are defined in the reference repository with LakeLogic, an open-source contract runtime.

1. Name the files and the patterns

# contracts/operations/onboarding/contracts/bronze/bronze_onboarding_insurance_certificates.yaml (excerpt)
source:
  type: landing
  path: '{landing_root}/insurance_certificates/**/*.pdf'
  format: pdf
  load_mode: incremental
  watermark_strategy: pipeline_log

extraction:
  provider: document

model:
  fields:
  - name: file_path
    type: string
    description: The document this row was extracted from.
  - name: policy_reference
    type: string
    extraction_task: metadata
    extraction_examples:
    - (FMC-\d{8})
  - name: driver_id
    type: string
    extraction_task: metadata
    extraction_examples:
    - (DRI-\d+)

The document provider reads each file's text (PDFs with or without a text layer, images, Word files) and applies each field's regex, taking the first capture group. It runs offline with no model, so the same file gives the same row every time. Every row keeps file_path, so a value can always be traced to its document. LakeLogic also supports model-based extraction providers for varied layouts; RideFlow does not use them here.

2. Make a miss visible

A pattern that does not match yields null. The contract's rules then decide what that means. RideFlow's Silver contract requires the driver link:

# contracts/operations/onboarding/contracts/silver/silver_onboarding_insurance_certificates.yaml (excerpt)
quality:
  row_rules:
  - name: driver_id_not_null
    sql: driver_id IS NOT NULL
    category: completeness

A certificate with no recognisable driver ID is quarantined in the domain's quarantine schema with that rule as its reason, alongside its file_path, instead of reaching Silver. Separately, if no file in a batch can be read at all, LakeLogic fails the run with the first error in the log, rather than writing an empty table. A pattern that reads the file but matches nothing is not that case: it yields null.

In RideFlow only driver_id is required. policy_reference is not, so the opening failure would still pass Silver with nulls. If a field matters to the business, add a rule for it; the contract only enforces what it says.

3. Declare the personal fields

The licence contract marks which extracted fields are personal data:

# contracts/operations/onboarding/contracts/silver/silver_onboarding_driving_licences.yaml (excerpt)
- name: licence_no
  type: string
  pii: true
  classification: licence_id
- name: date_of_birth
  type: string
  pii: true
  classification: date_of_birth

Tagging records the fact; it does not protect the value. LakeLogic masks a field when it also has a masking strategy (nullify, hash, redact, partial or encrypt), applied to accepted rows before they are written, and stops the run if masking fails rather than writing the values unmasked. A field tagged pii: true with no strategy is written as-is with a warning, and LakeLogic's contract checks flag it. RideFlow's licence fields are tagged but not masked, because Operations needs the licence number to verify drivers; who can read that table is left to Unity Catalog grants. A product built for wider use would add, for example:

- name: date_of_birth
  type: string
  pii: true
  masking: redact   # illustrative; not in the RideFlow contract

When you might not need this

• A few documents a week. People reading them, with a link to the file stored beside the typed fields, may be simpler and safer.
• Many layouts from many issuers. Regex patterns will not keep up; start with a model-based parser and put the validation and review path around it.
• Documents that never leave a restricted operational system. If no downstream team reads the parsed fields, the contract layer adds little.

Limits worth knowing

• Patterns are brittle by design. Deterministic means a changed layout fails the same way every time; it does not mean it keeps working. Test against new samples when a supplier changes a template.
• OCR misreads. Photos and scans add character errors. LakeLogic retries labels tolerant of common letter and digit swaps, but low-quality images still need review.
• Personal data is also in the raw files and quarantine. The landing volume holds the original documents, and quarantined rows keep their values. Permissions on both are a Unity Catalog decision, not a contract one.
• A tag is not compliance. Lawful basis, retention, access and erasure each need their own controls. Compliance as code covers what a contract can drive and what it cannot.

Key takeaways

1. Treat "no value found" as a result to check, not a success: require the fields the business depends on.
2. Keep the file path on every extracted row so each value traces back to its document.
3. Choose deterministic patterns for stable templates and model-based parsing for varied ones; both need validation after.
4. Declare personal fields in the contract, and remember a tag without a masking strategy changes nothing at write time.
5. Raw documents and quarantine tables hold the same personal data; secure them too.

See it in the reference project

The RideFlow repository contains the onboarding contracts for certificates and licences, and the synthetic documents they parse. For where failed rows go and who owns them, read Who owns a negative fare? A governed data mesh on Databricks.