Who Owns a Negative Fare? A Governed Data Mesh on Databricks

Picture a ride-hailing app (RideFlow, a fictional business) that records a completed trip, one passenger's ride, with fare_amount: -12.50. Is it a refund, a correction, or a bug in the source system? The platform team running the pipeline cannot know. The team that understands trips can, but when quality rules sit with the platform team, a common pattern, that team never sees the record. So someone filters negatives out of a dashboard, a driver sees a minus on their earnings, and a question in #data-help gets no reply.

This is the practical problem a data mesh is meant to solve. Giving domain teams ownership of their data puts business decisions with the people who understand them. Without shared mechanics, though, each team can end up inventing its own rules, failure handling and evidence, and other teams find its output hard to trust.

This article follows that one bad record through a working Databricks data mesh: who owns the rule, how it is applied, where the failed row goes, what a batch-level limit records, and what tells you which products are affected. RideFlow is published as an open reference project you can run.

The question underneath

The useful question is not "how do we build a data mesh?" It is:

How can domains own their data without making it harder for everyone else to trust?

Teams answer it in different ways, and several work well:

Databricks documents a separate quarantine pattern for keeping dropped rows; it is code you add around expectations.

RideFlow uses the last approach with LakeLogic, an open-source contract runtime. The rest of this article shows what that looks like in practice.

Who owns what

RideFlow has seven domains: Marketplace, Payments, Operations, Marketing, Business, Reference and Shared. Each owns named source systems and the Bronze, Silver and Gold products built from them, and each publishes into its own Unity Catalog catalog.

RideFlow reference project, synthetic data. One catalog per domain, each with its own layers, quarantine and run-log schemas.

A catalog per domain keeps each team's tables, failed rows and run evidence together, which makes ownership and grants easier to reason about. It is a namespace, not access control: who can read each catalog still depends on the Unity Catalog grants you write. Our post on Unity Catalog layouts for a data mesh covers that in detail.

The work divides like this:

That boundary matters. Declaring retention: P7D records an intended seven-day retention period; it does not delete anything by itself. Declaring that GDPR applies does not prove compliance.

Five steps from ownership to trusted data

Step 1: Name the owner and shared limits

Each domain keeps its ownership and defaults in _domain.yaml. This is Marketplace's, trimmed:

domain: marketplace

ownership:
  domain_owner: Platform Data Engineering
  team: data_engineering
  contacts:
  - name: Platform Data Lead
    role: oncall
    email: ${DOMAIN_CONTACT_MARKETPLACE}

slo:
  freshness:
    bronze:
      max_delay_minutes: 60
    silver:
      max_delay_minutes: 180
  quality:
    min_good_ratio: 0.92
    max_quarantine_ratio: 0.08

LakeLogic merges three levels of configuration, and the most specific value wins:

That gives every product a shared baseline while letting a domain set a stricter rule for one important product.

Step 2: Turn business knowledge into a contract rule

The Silver contract for completed trips declares the fields and the rules Marketplace owns. An excerpt:

# contracts/marketplace/rideflow/contracts/silver/silver_rideflow_trip_completed.yaml
info:
  target_layer: silver
  domain: marketplace
  system: rideflow

quality:
  row_rules:
  - name: fare_amount_range_min
    sql: fare_amount IS NULL OR fare_amount >= 0
  - name: fare_amount_range_max
    sql: fare_amount IS NULL OR fare_amount <= 1000000

Marketplace decides what a valid fare means. LakeLogic applies the rule the same way in every run, on Spark in Databricks or on Polars or DuckDB locally.

Step 3: Keep the failed row, with its reason

With quarantine enabled, one bad row does not reject every valid trip. LakeLogic separates accepted rows from failures and records why each failed. trip_48291 is left out of the Silver table and written to the domain's quarantine schema with an error such as:

[pre] Rule failed: fare_amount_range_min (fare_amount IS NULL OR fare_amount >= 0)

RideFlow reference project, synthetic data. Quarantined rows in rideflow_dev_marketplace.quarantine, each with _lakelogic_errors, _lakelogic_categories and its source.

Quarantine is not a bin where errors disappear. The row keeps enough context for the owner to investigate, fix the source and replay it. Inline data quality gates goes deeper on where to place these checks.

Step 4: Decide when a batch is unsafe

One negative fare in a large batch may be an isolated problem. Thousands suggest the source or the business process has changed. The domain's quality limits separate the two:

slo:
  quality:
    min_good_ratio: 0.92
    max_quarantine_ratio: 0.08

LakeLogic counts accepted and quarantined rows and checks them against these limits. When the good-row ratio falls below 92% or the quarantine ratio goes above 8%, the breach is recorded on the run and can raise an alert. The run that loaded the data is not stopped. RideFlow also runs a separate service-level job, pl_monitor_slo, that fails on a breach by default (fail_on_breach: true); downstream jobs can be made to depend on it.

Two options stop work. Setting quarantine.fail_on_quarantine: true fails the run as soon as a table quarantines any row; valid rows have already been written by then, and RideFlow leaves it off. Or your orchestration reads the recorded breach and decides whether downstream jobs run.

Step 5: Show what depends on the product

The system's registry declares what each contract reads. The daily trip KPIs read completed trips:

# contracts/marketplace/rideflow/_system.yaml (excerpt)
- layer: silver
  entity: silver_rideflow_trip_completed
  depends_on:
  - bronze_rideflow_trip_completed
- layer: gold
  entity: gold_fact_trip_completed_daily
  depends_on:
  - silver_rideflow_trip_completed

Those declarations build the dependency graph, set the run order, and tell you which products a bad Silver batch could affect. pipeline.visualize_dag() draws it:

RideFlow reference project, synthetic data. The top of the pipeline.visualize_dag() graph for the Marketplace system, rendered from its contracts. The same graph appears in the notebook during a Databricks run.

The graph shows declared dependencies, not relationships hidden inside arbitrary code. Unity Catalog's runtime lineage remains the record of what actually ran; use both.

What LakeLogic does, and what it does not

LakeLogic reads contracts, applies the checks they declare, separates accepted and failed rows, evaluates run-level limits, and writes structured evidence about each run.

It does not replace Unity Catalog permissions, workspace administration, compute policy, organisational decisions about ownership, legal review, or the processes that enforce retention and notifications. The domain defines the agreement, LakeLogic runs the controls, Databricks provides the platform, and people stay accountable for decisions.

When you might not need this

• One team owns all the data. A central team with good tests and clear runbooks may not need contracts per domain.
• You already run declarative pipelines everywhere and are happy to build failed-row retention and evidence on top of expectations.
• A handful of sources that rarely change. The overhead of a contract layer pays off as domains and cross-team products grow.

What makes this a data mesh

Domain folders alone do not make a data mesh. RideFlow also has:

1. Decision rights: domains own business meaning and product changes; the platform team owns shared mechanisms.
2. Data products: schemas, rules, dependencies and service expectations are version-controlled and testable.
3. Federated governance: shared defaults apply broadly, with explicit overrides where a domain needs them.
4. Self-service infrastructure: teams use common contract and deployment tooling instead of requesting custom pipelines.
5. Operational ownership: failed rules and breached limits have named owners and a route to resolution.

Start with one product

1. Name one producing domain and an accountable owner.
2. Write one schema and a small set of business-critical rules.
3. Test valid and invalid records locally (how to test contracts before Databricks).
4. Configure accepted and quarantine targets in a non-production catalog.
5. Set the dataset-level limit, and decide in orchestration whether a breach stops downstream work.
6. Declare the product's direct upstream and downstream dependencies.
7. Connect alerts, permissions, retention and audit through the systems that enforce them.

A negative fare makes a good first test: it should be missing from Silver, present in quarantine with fare_amount_range_min as its reason, and counted in the run totals.

Key takeaways

1. Put business rules with the domain that understands them, and run them with one shared mechanism.
2. Keep failed rows with their reason; dropping them silently loses the evidence the owner needs.
3. Separate row failures from batch breaches: quarantine the first, record and alert on the second, and wire any stop into orchestration.
4. Declared dependencies tell you what a bad batch can affect before it spreads.
5. Contracts describe and enforce data rules; access, retention and accountability still need their own controls.

Explore the reference project

The RideFlow repository contains every contract, the Databricks Asset Bundle and the quarantine shown here. The contract runtime is the open-source LakeLogic Core. Further patterns built on the same contracts:

• External Python logic and SCD Type 2 in Gold
• PDF extraction and PII at the contract boundary