Your Pipeline Drops Bad Rows Silently. Run a Data Contract on a Laptop in Five Minutes

Picture an orders file that lands with six rows. The pipeline loads four of them. The job succeeded, so no one asks where the other two went. One had an email with no domain, one had a refund typed as a negative amount. Later the finance total is off, and no one can explain by how much or why.

Rows disappear like this when validation lives in scattered places, a common pattern: a WHERE amount >= 0 in one transformation, a dropna() in another, a dashboard filter somewhere downstream. Each one is reasonable. Together they mean no one can say which rows were removed, by which rule, or whether the total still adds up.

This guide shows the alternative in about five minutes, on your own machine: write one data contract, run it against a small CSV, and inspect the accepted rows, the quarantined rows and the reason for each rejection. It does not need a cloud account, a cluster or credentials. It is deliberately small; the last sections say what it does not show.

What a data contract is, briefly

A data contract is a file that states what a dataset must look like: its fields and types, which fields are required, and the rules each row must pass. The useful part is that the same file is executed, so the stated expectation and the enforced one cannot quietly drift apart. If the term is new, Data contracts explained covers the background.

The ways teams check incoming rows today

All of these work, and many teams combine them.

Validation libraries differ here: some return failure cases or row samples, but routing every failed row to a separate store with its reason is usually code you write around them. Filters also put the rule in code only its author tends to read.

The gap this guide addresses is narrow: for each incoming row, keep it or set it aside with a recorded reason, and make the counts add up. That is what LakeLogic's contract run does.

Try it: one contract, six rows

LakeLogic Core is an open-source Python package (Apache 2.0). Its base install includes the Polars and DuckDB engines used below. Python 3.10 or later is required.

pip install lakelogic

Save six orders as orders.csv. Four of them are broken on purpose:

order_id,customer_email,amount,status
1001,ana@example.com,42.50,shipped
1002,ben@example,19.99,pending
1003,cara@example.com,-12.00,delivered
1004,dev@example.com,88.10,shippd
,eli@example.com,5.00,pending
1006,fay@example.com,61.25,returned

Save the contract as orders.yaml:

version: 1.0.0
dataset: orders

info:
  title: Orders

model:
  fields:
    - name: order_id
      type: integer
      required: true
    - name: customer_email
      type: string
      required: true
    - name: amount
      type: double
      required: true
    - name: status
      type: string

quality:
  row_rules:
    - name: valid_email
      sql: "customer_email LIKE '%@%.%'"
    - name: non_negative_amount
      sql: "amount >= 0"
    - name: known_status
      sql: "status IN ('pending', 'shipped', 'delivered', 'returned')"

Run it from the command line

lakelogic run --contract orders.yaml --source orders.csv \
  --output-good accepted.csv --output-bad quarantined.parquet

The log ends with a one-line summary: six source rows, two accepted, four quarantined. accepted.csv holds orders 1001 and 1006.

The quarantine file is Parquet on purpose. Each quarantined row carries a list of error messages, and CSV cannot store a list column, so writing quarantine to .csv fails.

Or run it from Python

from lakelogic import DataProcessor

processor = DataProcessor("orders.yaml", engine="polars")
good, bad = processor.run_source("orders.csv")

print(f"accepted: {good.height}, quarantined: {bad.height}")
for row in bad.select("order_id", "status", "_lakelogic_errors").iter_rows():
    print(row)

When we ran this, it printed:

accepted: 2, quarantined: 4
(1002, 'pending', ["Rule failed: valid_email (customer_email LIKE '%@%.%')"])
(1003, 'delivered', ['Rule failed: non_negative_amount (amount >= 0)'])
(1004, 'shippd', ["Rule failed: known_status (status IN ('pending', 'shipped', 'delivered', 'returned'))"])
(None, 'pending', ['Rule failed: order_id_required ("order_id" IS NOT NULL)'])

Notice the last row. Nobody wrote an order_id rule: required: true on the field became one. Change engine="polars" to engine="duckdb" and the result is the same for these rules.

What you just saw

• Nothing disappeared. Two accepted plus four quarantined equals six source rows.
• Each rejection names its rule, in the row itself, so the question "why is order 1003 missing?" has an answer you can query.
• The contract is the documentation. The rules a reviewer reads are the rules that ran.

When you might not need this

• A one-off analysis where you will look at the data yourself. A few filters are fine.
• Your bad rows should stop the job, not be set aside. A test that fails the build (dbt, an assertion) is simpler.
• You already run a validation library and have built quarantine and reporting around it. Adding a second rule format costs more than it saves.

Limits of this example

• Six rows prove the mechanism, not your pipeline. Real files bring encodings, late data and schema changes this example does not touch.
• The email rule is a smoke check. LIKE '%@%.%' catches obvious mistakes; it is not email validation.
• Engines are not identical for every expression. These rules behave the same on Polars and DuckDB; a regex or date function may not. Test data contracts locally before Databricks covers where they differ.
• Quarantine is a holding area, not a fix. Someone still decides whether order 1003 is a refund or a bug.

Key takeaways

1. A validation step should account for every row: accepted plus quarantined equals source.
2. Store the failure reason on the rejected row, not only in a log.
3. Declaring required: true is itself an enforced rule; you do not need to write it twice.
4. Keep the rules in a file that is executed, so documentation and enforcement cannot drift.

Next step

You wrote six test rows by hand. The next guide generates them from the same contract, including deliberately broken ones: Prove your quality rules work before production data arrives.