DuckLake DataFrame: Reading and Writing Lakehouse Data with Pandas, Polars, and PySpark

The DuckLake spec is so simple, even a “clanker” (robot) could build a DataFrame reader/writer for it.

What is DuckLake?

DuckLake v1.0 is the open lakehouse format specification released by the DuckDB team. Its core philosophy can be summarized in one sentence: DuckLake is just a collection of Parquet files plus a metadata catalog.

Compared to traditional data lake formats like Apache Iceberg or Delta Lake, DuckLake takes an extreme minimalism approach:

• On write: You tell DuckLake which Parquet files you produced
• On read: DuckLake tells you which Parquet files to read
• Metadata management: Optional backend supporting DuckDB, SQLite, PostgreSQL, etc.

This design makes DuckLake significantly easier to implement than Iceberg or Delta Lake. As the DuckDB team puts it: “All the heavy lifting is done by battle-tested systems — either a Parquet reader/writer (e.g., PyArrow) or a DBMS catalog (e.g., PostgreSQL).”

Project Background: Building a DuckLake Implementation with AI

In May 2026, the DuckDB team launched an interesting experiment: build a DuckLake DataFrame implementation from scratch with AI assistance.

The project lead, Pedro Holanda, set a simple goal — implement a DuckLake reader/writer in Python supporting Pandas, Polars, and PySpark. The “engineer” tasked with this mission was Dr. Peter van Holland, a database engineer known for his remarkably fast development speed.

The result was stunning: in just six days, Dr. Van Holland completed the full read/write implementation and published the ducklake-dataframe library on PyPI. This library achieves full compatibility with the DuckLake v1.0 specification, supports three DataFrame libraries (Pandas, Polars, PySpark), and works with three catalog backends (DuckDB, SQLite, PostgreSQL).

Core Architecture

The core architecture of ducklake-dataframe is elegantly simple:

┌─────────────────────────────────────────────────┐
│ DataFrame Layer │
│ ┌──────────┐ ┌──────────┐ ┌──────────────┐ │
│ │ Pandas │ │ Polars │ │ PySpark │ │
│ └────┬─────┘ └────┬─────┘ └──────┬───────┘ │
│ └──────────────┼───────────────┘ │
│ ▼ │
│ ┌─────────────────────┐ │
│ │ ducklake-dataframe │ │
│ │ (Parquet I/O) │ │
│ └────────┬────────────┘ │
│ ▼ │
│ ┌─────────────────────┐ │
│ │ Catalog Backend │ │
│ ┌──────┤ DuckDB/SQLite/PG ├──┐ │
│ │ └─────────────────────┘ │ │
│ │ │ │
│ ┌─▼──────────┐ ┌─────▼─────────┐ │
│ │ Parquet │ │ Parquet Files │ │
│ │ (PyArrow) │ │ (S3/GCS/local)│ │
│ └────────────┘ └───────────────┘ │
└─────────────────────────────────────────────────┘

Key insight: DuckLake is a specification, not a tool. This means:

1. DuckDB is not required at runtime
2. The catalog backend can be any relational database
3. Parquet I/O can use any mature library (PyArrow is the most common choice)

Quick Start

Installation

pip install ducklake-dataframe pandas pyarrow

Creating a DuckLake Catalog

First, create a DuckLake catalog using DuckDB’s ducklake extension:

import duckdb

# Connect and create DuckLake
conn = duckdb.connect()
conn.execute("""
 ATTACH 'ducklake:sqlite:catalog.ducklake' AS lake
 (DATA_PATH 'data/');
""")

# Create a table
conn.execute("""
 CREATE TABLE lake.users (
 id INTEGER, 
 name VARCHAR, 
 email VARCHAR,
 score DOUBLE, 
 active BOOLEAN
 );
""")

# Insert data
conn.execute("""
 INSERT INTO lake.users VALUES
 (1, 'Alice', '[email protected]', 95.5, true),
 (2, 'Bob', '[email protected]', 87.3, true),
 (3, 'Carol', '[email protected]', 72.1, false),
 (4, 'Dave', '[email protected]', 91.0, true),
 (5, 'Eve', '[email protected]', 88.8, true);
""")

conn.close()

Reading DuckLake Data with Pandas

Now, without needing DuckDB at runtime, read the data using ducklake-dataframe:

from ducklake_dataframe.pandas import DuckLakePandas

# Connect to existing DuckLake
dl = DuckLakePandas("ducklake:sqlite:catalog.ducklake", data_path="data/")

# Read like a regular table
df = dl.read("users")
print(df)

Output:

 id name email score active
0 1 Alice [email protected] 95.5 True
1 2 Bob [email protected] 87.3 True
2 3 Carol [email protected] 72.1 False
3 4 Dave [email protected] 91.0 True
4 5 Eve [email protected] 88.8 True

Writing to DuckLake with Pandas

import pandas as pd

# Create a new Pandas DataFrame
new_data = pd.DataFrame({
 'id': [6, 7, 8],
 'name': ['Frank', 'Grace', 'Henry'],
 'email': ['[email protected]', '[email protected]', '[email protected]'],
 'score': [93.2, 85.7, 79.4],
 'active': [True, True, False]
})

# Append to DuckLake
dl.append("users", new_data)

# Re-read to verify
df_updated = dl.read("users")
print(len(df_updated)) # Output: 8

Cross-Engine Interoperability

This is where it gets truly powerful — different DataFrame engines can seamlessly read and write the same DuckLake:

# Write with Polars
import polars as pl
from ducklake_dataframe.polars import DuckLakePolars

dl_polars = DuckLakePolars("ducklake:sqlite:catalog.ducklake", data_path="data/")
polars_df = pl.DataFrame({
 'id': [9, 10],
 'name': ['Ivy', 'Jack'],
 'email': ['[email protected]', '[email protected]'],
 'score': [96.1, 82.3],
 'active': [True, True]
})
dl_polars.append("users", polars_df)

# Read Polars-written data with Pandas
dl_pandas = DuckLakePandas("ducklake:sqlite:catalog.ducklake", data_path="data/")
df_all = dl_pandas.read("users")
print(len(df_all)) # Output: 10!

Comparison with Traditional Data Lake Solutions

Technical Highlights

1. Read-Write Parity

Dr. Van Holland’s core objective was: whatever DuckDB writes, he should be able to read; whatever he writes, DuckDB should be able to read. This goal was fully achieved.

2. Minimal Dependency Footprint

The core dependencies of ducklake-dataframe are:

• DataFrame library (Pandas / Polars / PySpark)
• PyArrow (for Parquet I/O)
• Catalog backend (SQLite / PostgreSQL / DuckDB — all optional)

No complex runtime components, no JVM dependency, no external services.

3. Seamless Integration with DuckDB Ecosystem

While ducklake-dataframe can run without DuckDB, it is fully compatible with the DuckDB ecosystem:

# Query DuckLake data with DuckDB
import duckdb
conn = duckdb.connect()
conn.execute("ATTACH 'ducklake:sqlite:catalog.ducklake' AS lake")
result = conn.execute("SELECT * FROM lake.users WHERE score > 90").fetchdf()
print(result)

4. Data Inlining Support

DuckLake supports storing small data directly inside Parquet files rather than as separate files. This is particularly useful for small tables or frequently updated dimension tables.

Summary

The core idea behind DuckLake is: data lakes shouldn’t be that complicated. By offloading complexity to battle-tested Parquet format and existing relational database catalogs, DuckLake provides a minimalist yet fully functional data lake implementation.

The greatest significance of the ducklake-dataframe project lies not in how polished it is (the authors themselves acknowledge it is not production-ready), but in what it proves: the simplicity of the DuckLake specification — a simple spec, paired with mature open-source libraries, can build a fully functional lake format implementation in a matter of days.

With DuckLake v1.0 officially released and more ecosystem tools emerging, we have every reason to believe that the future of data lakes may be simpler and more efficient than we imagined.

📚 Further Reading:

• ducklake-dataframe GitHub Repository
• DuckLake v1.0 Specification
• DuckDB Official DuckLake Documentation

本文信息

如发现错误，欢迎通过 GitHub Issue 或邮件 [email protected] 反馈。