Ten Perfect Test Rows Prove Nothing: Generate Failing Data from Your Contract

Picture the test fixture for a trips pipeline at RideFlow (a fictional ride-hailing business; a trip is one completed ride). It has ten rows. Every fare is positive and every rating is between 1 and 5, so the suite stays green. Then the source starts sending unrated trips with an empty rating, and in production those trips land in quarantine. The rule was ambiguous all along; the fixture never asked.

Hand-written fixtures go stale because they are a second copy of the schema. When a field becomes required or a new status appears, someone has to remember to update the fixture, and the rows added are often happy-path ones. Teams also often cannot use production samples for tests, for privacy reasons, so the fixture is all they have.

This guide shows a practical alternative: generate test rows, including deliberately broken ones, from the same contract that governs the dataset, then run them back through the contract and treat every disagreement as something to investigate. The worked example found a real ambiguity in its own contract on the first run. The code below runs as written against LakeLogic Core; the last sections cover what synthetic data cannot tell you.

The ways teams get test data today

Hand-written fixtures drift when the contract changes and seldom cover failure paths. A masked copy may not contain the edge case you need.

All of these are legitimate, and a test suite often combines a few: curated fixtures for known incidents, plus something that generates volume and variety.

The question these leave open is: when the contract changes, what makes the test data change with it, including the rows that are supposed to fail? That is what contract-driven generation answers.

Generate valid and broken rows from the contract

LakeLogic Core is open source (Apache 2.0). DataGenerator reads a contract and produces rows that respect its types, required fields, ranges and accepted values. Install the synthetic extra for realistic names and emails via Faker:

pip install "lakelogic[synthetic]"

The example contract describes completed trips for RideFlow. Save it as contracts/trips.yaml:

version: 1.0.0
dataset: trips

info:
  title: Completed trips
  target_layer: silver

model:
  fields:
    - name: trip_id
      type: string
      required: true
    - name: rider_id
      type: string
      required: true
    - name: fare_amount
      type: double
      required: true
      min: 0
      max: 500
    - name: rating
      type: integer
      min: 1
      max: 5
    - name: status
      type: string
      accepted_values: [completed, refunded]
    - name: completed_at
      type: timestamp
      required: true

primary_key: [trip_id]

quality:
  row_rules:
    - name: rating_in_range
      sql: "rating IS NULL OR rating BETWEEN 1 AND 5"

Generate 1,000 rows, 10% of them deliberately invalid, and run them through the same contract:

import polars as pl
from lakelogic import DataGenerator, DataProcessor

trips = DataGenerator("contracts/trips.yaml", seed=42).generate(
    rows=1_000, invalid_ratio=0.10
)

good, bad = DataProcessor("contracts/trips.yaml", engine="polars").run(trips)

print(f"accepted={good.height} quarantined={bad.height}")
print("broken rows accepted:", good.filter(pl.col("_is_invalid")).height)
print("valid rows quarantined:", bad.filter(~pl.col("_is_invalid")).height)

Each generated row carries two test columns. _is_invalid says the generator broke it on purpose; _test_case_types says how (for example RANGE_VIOLATION, NOT_NULL_VIOLATION or ACCEPTED_VALUE_VIOLATION). They describe the generator's intent. The contract run is the judge.

The first run disagreed with itself

We expected 900 accepted and 100 quarantined. We got this:

accepted=888 quarantined=112
broken rows accepted: 0
valid rows quarantined: 12

Every deliberately broken row was caught, which is good. But 12 rows the generator considered valid were quarantined. All 12 had a null rating, and each failed a rule called rating_range that we never wrote. It comes from min: 1 and max: 5 on the field, and that generated check treats a missing rating as out of range. The IS NULL OR guard in our own row rule never got a say.

That is the opening scenario, found before production. The generator assumed an optional field may be empty; the contract, as written, said otherwise. Neither tool was wrong in isolation. The contract was ambiguous, and a fixture of ten perfect rows would never have shown it.

We decided unrated trips are allowed, so we removed min and max from the rating field and kept the explicit row rule, which says what we mean. The second run:

accepted=900 quarantined=100
broken rows accepted: 0
valid rows quarantined: 0

This is the pattern worth keeping: assert that intended-valid rows are accepted and intended-invalid rows are quarantined, and investigate every row where they disagree. Counting that the job finished proves nothing.

Make it reproducible

The same seed reproduced every value except completed_at: timestamps are drawn relative to the current time, so two runs a moment apart differ by milliseconds. To make a whole run repeatable, pin the time window as well:

from datetime import datetime
from lakelogic import DataGenerator

trips = DataGenerator("contracts/trips.yaml", seed=42).generate(
    rows=1_000,
    invalid_ratio=0.10,
    window_start=datetime(2026, 1, 1),
    window_end=datetime(2026, 1, 2),
)

With the window fixed, two runs produced identical frames. The RideFlow reference project's test-data notebook pins its generation window for the same reason. Pin your LakeLogic version too: a new release may generate different values from the same seed.

Related tables without orphaned keys

Trips reference riders. Generating each table separately produces rider_id values that exist nowhere. generate_related() reads the key declarations, generates parents first and draws child keys from them. Save a small parent contract as contracts/riders.yaml:

version: 1.0.0
dataset: riders

info:
  title: Riders

model:
  fields:
    - name: rider_id
      type: string
      required: true
    - name: home_city
      type: string

primary_key: [rider_id]

Then:

from lakelogic import DataGenerator

data = DataGenerator.generate_related(
    contracts={"riders": "contracts/riders.yaml", "trips": "contracts/trips.yaml"},
    rows={"riders": 50, "trips": 500},
    seed=42,
)
orphans = data["trips"].join(data["riders"], on="rider_id", how="anti")
print(orphans.height)  # 0

The matching rider_id column name was enough for LakeLogic to detect the relationship. When names differ, pass relationships=[...] explicitly. With invalid_ratio above zero, it also produces deliberate orphans so you can test referential checks.

The same generator offers generate_stream() for time-windowed batches (useful for incremental loads and freshness rules) and DataGenerator.from_file() to start from a sample file's shape. A sample file still needs the privacy review any data file needs.

A CI check in five lines of intent

1. Generate rows from the version-controlled contract with a fixed seed and window.
2. Run them through the contract.
3. Fail the build if any _is_invalid row is accepted.
4. Fail the build if any intended-valid row is quarantined, and read why.
5. Run it whenever the contract or the pipeline changes.

Keep your hand-written fixtures for incidents you have actually seen. Generated data widens coverage around them; it does not replace them.

When you might not need this

• Your rules are few and stable. A handful of curated fixture rows is cheaper to maintain than a generator.
• You are testing a transformation function, not a table's acceptance rules. Hypothesis or dbt unit tests fit better.
• You need realistic distributions for performance or analytics testing. A learned synthesizer or a governed masked copy is closer to the target.

What synthetic data does not prove

• That production looks like this. Generated values follow the contract, not your customers' real distribution or correlations.
• Performance at scale. A thousand rows say nothing about a billion.
• That your rules are the right rules. It tests that rules behave as written. Whether a negative fare should be a refund is a business question.
• Privacy or compliance. Synthetic rows are not an anonymised copy of anything, and LakeLogic does not claim they are.

Key takeaways

1. Generate test data from the contract, so a schema change changes the fixture.
2. Generate failing rows on purpose; the quarantine path needs tests too.
3. Compare intended validity with actual routing. Every disagreement is a finding, often an ambiguous rule.
4. Pin the seed, the time window and the library version for repeatable runs.
5. Keep curated fixtures for real incidents alongside generated volume.

Try it

Run the code above, or open the Data Generation notebook in Google Colab, which also covers streaming batches and related tables. If you are wondering whether an LLM could write these test cases for you, AI-generated test data: what to trust looks at that question separately.