LakeLogic
Loading…

Open Source · Apache 2.0

Data contracts that execute, not just document.

Define your data product in YAML. LakeLogic enforces it at pipeline time. Bad rows are quarantined the moment they appear, not three dashboards later. Same contract runs on Polars, Spark, or DuckDB.

30K+ downloads · Apache 2.0

Run QuickstartGitHubDocs
lakelogicoss.ipynbPython 3.12
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
"""
E-Commerce Orders → silver layer.
 
Executes the contract end-to-end: schema + PII + row & dataset quality rules.
Bad rows are quarantined with reasons; good rows promoted to silver.
Same contract runs on DuckDB, Spark, or Polars.
"""
 
# pip install lakelogic
 
import lakelogic as ll
 
contract = """
version: 1.0.0
dataset: orders
info:
  title: E-Commerce ▌

One governed definition

Define every data product once.

Open Lakehouse Contract defines what each data product must deliver.

LakeLogic Core applies the same contract across supported engines, from efficient single-node processing to distributed workloads.

Design inputs

  • Business requirements
  • Source schemas
  • Governance policies
  • Ownership
  • Service expectations
  • Expected outputs

Open Lakehouse Contract

Open specification · Apache 2.0

One governed data-product definition

Schema · quality · ownership · classification · lineage · SLOs

LakeLogic Core

Open source · Apache 2.0

Applies the contract and records the evidence

Validation · execution · quarantine · run evidence

PolarsFast columnar processing
SparkDistributed data processing
DuckDBEmbedded analytical SQL

One definition · the right engine for every workload · consistent evidence

  • Open Lakehouse Contract

    One portable definition.

    Schema and business meaning · Ownership and contacts · Quality expectations · Lineage and dependencies · Service levels · Privacy and retention · Transformations and outputs

    Explore the open specification

  • LakeLogic Core

    Run it across engines.

    Apply the same contract on every engine · Validate before and during a run · Quarantine failing records · Capture runtime evidence · Report the result back to the contract

    View LakeLogic Core

Define once. Execute across supported engines.

One contract. Three engines.

Apache 2.0

LakeLogic auto-discovers the best engine for your environment, and the same SQL-first quality gates run from a laptop to petabytes.

  • Runtime Data Contracts

    Enforce schema, quality rules, and business logic during data movement, not after. YAML, version-controlled, engine-agnostic.

  • Quarantine Bad Data

    Automatically detour invalid rows in real-time. Bad data is isolated, tagged, and triaged, never silently dropped.

  • Medallion Architecture

    Quality gate between Bronze → Silver → Gold. Clean the front door of your lakehouse with shift-left data quality.

  • Materialization Strategies

    Append, merge, SCD2, or overwrite. Native Spark operations at petabyte scale. Same contract, different strategies.

  • Polars

    Fast columnar processing

  • Spark

    Distributed data processing

  • DuckDB

    Embedded analytical SQL

Runnable proof

8 notebooks · no install

Each notebook is self-contained, runs in Google Colab in under a minute, and maps to one pillar of LakeLogic’s value. Open one and it runs, no install, no sales call.

  • Quickstart

    5-minute quickstart

    One YAML file, and every row accounted for: kept, or set aside with the reason.

    Open in Google Colab

  • Trust

    Data quality checks

    Catch bad data at the door, and see each rejected row with the rule it broke.

    Open in Google Colab

  • Compliance

    Personal data and erasure

    GDPR-ready pipelines: erase a person in two lines and keep the audit trail.

    Open in Google Colab

  • Scale

    One contract, three engines

    Run the same data contract on Polars, DuckDB or Spark, with no code changes.

    Open in Google Colab

  • DX

    From failure to root cause

    From pipeline failure to root cause in minutes, not hours.

    Open in Google Colab

  • AI

    Test data from a data contract

    Realistic test data that catches bugs, with no production samples needed.

    Open in Google Colab

  • Integrations

    dbt, dlt and streams

    Add data contracts to what you already run: dbt, dlt, streams, cloud storage and databases.

    Open in Google Colab

  • End-to-End

    Bronze to Gold with dlt and Prefect

    A production Bronze, Silver and Gold pipeline in one notebook, on Delta Lake or any database.

    Open in Google Colab

Two products · one contract

Outgrow the OSS? Bring it to the platform.

LakeLogic Open Source is free forever. When you need Zeus AI, visual lineage, governance, and team workflows, drop your YAML straight into the LakeLogic Platform.

Talk to us about the platform
Stay on OSS

Apache 2.0 · runs on Polars, Spark, or DuckDB · 90+ runnable examples · zero vendor lock-in.

Browse the repo
Migration path

Already using the OSS reference architecture? Drag and drop your docs/contracts/*.yaml folder straight into the LakeLogic Platform.

Bulk import contracts

Ready to build trusted data products?

See LakeLogic in action.