Data Product Quality Metrics That Matter: Measure Each Question Separately
A single quality score can be green on the day half the data never arrived. The data product quality metrics and KPIs worth tracking, which consumer question each one answers, the tools teams use to measure them, and what LakeLogic records per run.
Data Product Specification Template (YAML + Markdown), Based on a Real Contract Format
A data product specification written in a document template drifts from the table it describes. A copyable template in two parts: YAML that a validator and a pipeline can read, and a short Markdown page for what no machine checks. Every YAML field comes from the Open Lakehouse Contract format, and the article says which parts are checked.
Data Product Testing: How to Test Pipelines and Contracts Before Consumers Do
Green pipeline tests prove the code does what the engineer intended. They do not prove the data product still keeps its promises. A test strategy in five layers, what each one catches, the tools for each, and where LakeLogic fits.
Data Product Versioning: Ship Breaking Changes Without Breaking Consumers
A version number only protects consumers if something checks it against the change. Which schema changes break readers, the ways teams version data products today, and how LakeLogic derives the minimum bump from the contract diff.
Data Products in a Medallion Architecture: Which Bronze, Silver and Gold Tables Are the Product?
A medallion lakehouse tells you how refined a table is. It does not tell you which tables a team has promised to someone. How teams map data products onto bronze, silver and gold, the common ways to record that mapping, and a worked example where the product lists its outputs and the contracts point back.
Data Products vs Medallion Architecture: What Each One Organises, and When It Matters
Medallion architecture organises tables by how refined they are. A data product organises them by who promises what to whom. The two are often argued about as rivals. What each concept is, where they differ, the strongest case on each side, and how to tell which framing your estate needs.
Data Governance in Databricks: A Contract-Driven Framework
A worked example of ownership, quality rules, lineage and personal-data retention, with clear lines between what the contract checks and what Unity Catalog enforces.
Synthetic Test Data for a Databricks Test Environment: Fill Dev Without Copying Production
A test workspace with empty tables, or a masked copy of production nobody has refreshed in months, cannot tell you whether the platform works. The common ways to fill a non-production environment, where each fits, and how LakeLogic seeds one from data contracts without reading production rows.
A Catalog per Team Is Not an Access Boundary: Unity Catalog Layouts for a Data Mesh
A catalog per domain gives each Databricks team a clear namespace and a natural place for grants. It does not give them access isolation until you write the grants. A worked example from the RideFlow reference project.
Let Your AI Assistant Propose Data Quality Rules, Not Merge Them
Claude, Cursor or Copilot can write a uniqueness check in seconds. What they cannot tell you is what it would reject or who agreed to it. The common ways to govern assistant-written changes, and how LakeLogic's MCP server adds a sandbox test and a person's decision.
Quarantine Bad Rows, Check Every Batch: Data Quality Inside the Pipeline
Row rules decide what happens to one bad record; dataset checks decide whether a whole batch is fit to use. The common ways to do both, their trade-offs, and how LakeLogic keeps the failing rule on every quarantined row.
Running Databricks and Fabric? What a Data Product Can Share and What It Can't
When a data product has to run on Databricks and Microsoft Fabric, some of it can be written once and some cannot. How teams split the work today, where the business rules drift, and what a portable contract does and does not keep in sync.
The Agent Was Right About the Wrong Data: Check Inputs Before AI Agents Read Them
A good model and a careful prompt still give a confident wrong answer when the table behind them is stale or half-loaded. Where data checks belong in an AI application, the common approaches, and what LakeLogic contracts add.
Ten Perfect Test Rows Prove Nothing: Generate Failing Data from Your Contract
How to test that quality rules and quarantine actually work without production data: generate valid and deliberately broken rows from the contract, run them back through it, and investigate every disagreement.
The Trip Your Revenue Report Dropped: SCD2 History and Unknown Members in Gold
A fact row that cannot find its dimension disappears from an inner join or lands on a null key. How teams handle SCD Type 2 history, late dimensions and calendar logic in Gold, and how a contract can declare the policy and count the misses.
90 PDFs In, 0 Fields Out: Putting Document Extraction Under a Data Contract
Insurance certificates and licences arrive as PDFs and photos full of personal data. Compare the common ways to extract them, then see how a contract makes the parse deterministic, the misses visible and the personal fields explicit.
Green Locally, Red on Databricks: What a Laptop Run Can and Cannot Prove
Run data contracts on Polars or DuckDB for fast feedback, then know exactly which checks still need Spark. A worked example where one regex rule passes on one engine and errors on another.
Your Pipeline Drops Bad Rows Silently. Run a Data Contract on a Laptop in Five Minutes
Six rows, one YAML file, two commands. See what a data contract does to bad data, where the rejected rows go and why each was rejected, before you connect anything to a warehouse.
Who Owns a Negative Fare? A Governed Data Mesh on Databricks
Follow one bad trip record through a Databricks data mesh: which team owns the rule, where the failed row goes, what a batch-level limit does and does not stop, and what the platform team should not have to decide.
The Erasure Request Nobody Can Close: Masking, Erasure and Retention From Contracts
A right-to-erasure request turns into a hunt through every table that might hold the person. Compare the usual approaches, then see how contracts can declare personal fields, drive a dry-run-first erasure and record evidence, and what they leave to the platform.
Testing a Data Quality Rule Shouldn't Need a Cluster. Proving It on Spark Still Does.
Waiting for a Spark cluster to test one quality rule is a tax on every change. How teams iterate locally, why a local pass can still be wrong on Spark, and how one contract run on Polars, DuckDB and Spark keeps the two honest.
Your Data Alerts Channel Is Muted: Route Alerts by Owner and Page Once per Problem
A shared #data-alerts channel that pings hundreds of times a day gets muted, and the one alert that matters is missed. The common fixes, their trade-offs, and how LakeLogic routes alerts from the contract's owner and folds repeats into one episode.
An LLM Says user+tag@example.com Is Invalid. Where AI Test Data Helps, and Where It Can't Decide
Language models are good at plausible values and poor at deciding what is valid. How to use AI for test data without letting it become the judge of your quality rules.
dbt Contracts Check Your Models. What Checks the Data on the Way In?
dbt model contracts, dlt schema contracts and platform expectations each guard one point in a pipeline. Where row-level rules belong, what each tool does with a bad row, and when a contract runtime at ingestion is worth adding.
Data Contracts Explained: What ODCS Standardises and What Still Has to Run
A data contract is an agreement a machine can read. What the Open Data Contract Standard covers, the ways teams enforce contracts today, and where an executable contract fits.