LakeLogic
Loading…
Home/Blog

Engineering essays

from the data trust front

LakeLogicBlog
Engineering

Data Product Quality Metrics That Matter: Measure Each Question Separately

BlogEngineering

A single quality score can be green on the day half the data never arrived. The data product quality metrics and KPIs worth tracking, which consumer question each one answers, the tools teams use to measure them, and what LakeLogic records per run.

Oct 31, 2026LakeLogic Team
LakeLogicBlog
Data Contracts

Data Product Specification Template (YAML + Markdown), Based on a Real Contract Format

BlogData Contracts

A data product specification written in a document template drifts from the table it describes. A copyable template in two parts: YAML that a validator and a pipeline can read, and a short Markdown page for what no machine checks. Every YAML field comes from the Open Lakehouse Contract format, and the article says which parts are checked.

Oct 31, 2026LakeLogic Team
LakeLogicBlog
Engineering

Data Product Testing: How to Test Pipelines and Contracts Before Consumers Do

BlogEngineering

Green pipeline tests prove the code does what the engineer intended. They do not prove the data product still keeps its promises. A test strategy in five layers, what each one catches, the tools for each, and where LakeLogic fits.

Oct 31, 2026LakeLogic Team
LakeLogicBlog
Data Contracts

Data Product Versioning: Ship Breaking Changes Without Breaking Consumers

BlogData Contracts

A version number only protects consumers if something checks it against the change. Which schema changes break readers, the ways teams version data products today, and how LakeLogic derives the minimum bump from the contract diff.

Oct 31, 2026LakeLogic Team
LakeLogicBlog
Engineering

Data Products in a Medallion Architecture: Which Bronze, Silver and Gold Tables Are the Product?

BlogEngineering

A medallion lakehouse tells you how refined a table is. It does not tell you which tables a team has promised to someone. How teams map data products onto bronze, silver and gold, the common ways to record that mapping, and a worked example where the product lists its outputs and the contracts point back.

Oct 31, 2026LakeLogic Team
LakeLogicBlog
Engineering

Data Products vs Medallion Architecture: What Each One Organises, and When It Matters

BlogEngineering

Medallion architecture organises tables by how refined they are. A data product organises them by who promises what to whom. The two are often argued about as rivals. What each concept is, where they differ, the strongest case on each side, and how to tell which framing your estate needs.

Oct 31, 2026LakeLogic Team
LakeLogicBlog
Engineering

Data Governance in Databricks: A Contract-Driven Framework

BlogEngineering

A worked example of ownership, quality rules, lineage and personal-data retention, with clear lines between what the contract checks and what Unity Catalog enforces.

Oct 31, 2026LakeLogic Team
LakeLogicBlog
Engineering

Synthetic Test Data for a Databricks Test Environment: Fill Dev Without Copying Production

BlogEngineering

A test workspace with empty tables, or a masked copy of production nobody has refreshed in months, cannot tell you whether the platform works. The common ways to fill a non-production environment, where each fits, and how LakeLogic seeds one from data contracts without reading production rows.

Oct 31, 2026LakeLogic Team
LakeLogicBlog
Engineering

A Catalog per Team Is Not an Access Boundary: Unity Catalog Layouts for a Data Mesh

BlogEngineering

A catalog per domain gives each Databricks team a clear namespace and a natural place for grants. It does not give them access isolation until you write the grants. A worked example from the RideFlow reference project.

Oct 31, 2026LakeLogic Team
LakeLogicBlog
Engineering

Let Your AI Assistant Propose Data Quality Rules, Not Merge Them

BlogEngineering

Claude, Cursor or Copilot can write a uniqueness check in seconds. What they cannot tell you is what it would reject or who agreed to it. The common ways to govern assistant-written changes, and how LakeLogic's MCP server adds a sandbox test and a person's decision.

Oct 15, 2026LakeLogic Team
LakeLogicBlog
Engineering

Quarantine Bad Rows, Check Every Batch: Data Quality Inside the Pipeline

BlogEngineering

Row rules decide what happens to one bad record; dataset checks decide whether a whole batch is fit to use. The common ways to do both, their trade-offs, and how LakeLogic keeps the failing rule on every quarantined row.

Jul 31, 2026LakeLogic Team
LakeLogicBlog
Engineering

Running Databricks and Fabric? What a Data Product Can Share and What It Can't

BlogEngineering

When a data product has to run on Databricks and Microsoft Fabric, some of it can be written once and some cannot. How teams split the work today, where the business rules drift, and what a portable contract does and does not keep in sync.

Jul 31, 2026LakeLogic Team
LakeLogicBlog
Engineering

The Agent Was Right About the Wrong Data: Check Inputs Before AI Agents Read Them

BlogEngineering

A good model and a careful prompt still give a confident wrong answer when the table behind them is stale or half-loaded. Where data checks belong in an AI application, the common approaches, and what LakeLogic contracts add.

Jul 31, 2026LakeLogic Team
LakeLogicBlog
Engineering

Ten Perfect Test Rows Prove Nothing: Generate Failing Data from Your Contract

BlogEngineering

How to test that quality rules and quarantine actually work without production data: generate valid and deliberately broken rows from the contract, run them back through it, and investigate every disagreement.

Jul 22, 2026LakeLogic Team
LakeLogicBlog
Data Contracts

The Trip Your Revenue Report Dropped: SCD2 History and Unknown Members in Gold

BlogData Contracts

A fact row that cannot find its dimension disappears from an inner join or lands on a null key. How teams handle SCD Type 2 history, late dimensions and calendar logic in Gold, and how a contract can declare the policy and count the misses.

Jul 21, 2026LakeLogic Team
LakeLogicBlog
Data Contracts

90 PDFs In, 0 Fields Out: Putting Document Extraction Under a Data Contract

BlogData Contracts

Insurance certificates and licences arrive as PDFs and photos full of personal data. Compare the common ways to extract them, then see how a contract makes the parse deterministic, the misses visible and the personal fields explicit.

Jul 21, 2026LakeLogic Team
LakeLogicBlog
Data Contracts

Green Locally, Red on Databricks: What a Laptop Run Can and Cannot Prove

BlogData Contracts

Run data contracts on Polars or DuckDB for fast feedback, then know exactly which checks still need Spark. A worked example where one regex rule passes on one engine and errors on another.

Jul 21, 2026LakeLogic Team
LakeLogicBlog
Data Contracts

Your Pipeline Drops Bad Rows Silently. Run a Data Contract on a Laptop in Five Minutes

BlogData Contracts

Six rows, one YAML file, two commands. See what a data contract does to bad data, where the rejected rows go and why each was rejected, before you connect anything to a warehouse.

Jul 14, 2026LakeLogic Team
LakeLogicBlog
Data Mesh

Who Owns a Negative Fare? A Governed Data Mesh on Databricks

BlogData Mesh

Follow one bad trip record through a Databricks data mesh: which team owns the rule, where the failed row goes, what a batch-level limit does and does not stop, and what the platform team should not have to decide.

Jul 14, 2026LakeLogic Team
LakeLogicBlog
Governance

The Erasure Request Nobody Can Close: Masking, Erasure and Retention From Contracts

BlogGovernance

A right-to-erasure request turns into a hunt through every table that might hold the person. Compare the usual approaches, then see how contracts can declare personal fields, drive a dry-run-first erasure and record evidence, and what they leave to the platform.

Jul 12, 2026LakeLogic Team
LakeLogicBlog
Engineering

Testing a Data Quality Rule Shouldn't Need a Cluster. Proving It on Spark Still Does.

BlogEngineering

Waiting for a Spark cluster to test one quality rule is a tax on every change. How teams iterate locally, why a local pass can still be wrong on Spark, and how one contract run on Polars, DuckDB and Spark keeps the two honest.

Jul 11, 2026LakeLogic Team
LakeLogicBlog
Engineering

Your Data Alerts Channel Is Muted: Route Alerts by Owner and Page Once per Problem

BlogEngineering

A shared #data-alerts channel that pings hundreds of times a day gets muted, and the one alert that matters is missed. The common fixes, their trade-offs, and how LakeLogic routes alerts from the contract's owner and folds repeats into one episode.

Jul 10, 2026LakeLogic Team
LakeLogicBlog
AI

An LLM Says user+tag@example.com Is Invalid. Where AI Test Data Helps, and Where It Can't Decide

BlogAI

Language models are good at plausible values and poor at deciding what is valid. How to use AI for test data without letting it become the judge of your quality rules.

Jul 09, 2026LakeLogic Team
LakeLogicBlog
Data Contracts

dbt Contracts Check Your Models. What Checks the Data on the Way In?

BlogData Contracts

dbt model contracts, dlt schema contracts and platform expectations each guard one point in a pipeline. Where row-level rules belong, what each tool does with a bad row, and when a contract runtime at ingestion is worth adding.

Jul 08, 2026LakeLogic Team
LakeLogicBlog
Data Contracts

Data Contracts Explained: What ODCS Standardises and What Still Has to Run

BlogData Contracts

A data contract is an agreement a machine can read. What the Open Data Contract Standard covers, the ways teams enforce contracts today, and where an executable contract fits.

Feb 10, 2026LakeLogic Team

Ready to build trusted data products?

See LakeLogic in action.