#562: DuckLake: The Lakehouse That's Just SQL and Parquet

How many files does your query read before it reads any data? On some data lakes, you go through JSON and metadata files first, just to learn which Parquet files matter. DuckLake asks one SQL question instead. The metadata lives in a real database. The data stays in plain Parquet. That's the entire format.

Pedro Holanda joined DuckDB in 2018, when it was still a research prototype at CWI. He's the lead DuckLake developer. Guillermo Sanchez Dionis works on DuckLake and the new Quack protocol.

With Quack as the catalog, DuckLake handles 200 transactions a second under heavy contention. No other open table format comes close.

Episode sponsors

Six Feet Up
Talk Python Courses

Links from the show

Guests
Pedro Holanda: pedroholanda.org
Guillermo Sanchez: linkedin.com

PhD on progressive indexes: ir.cwi.nl
SQLite: www.sqlite.org
Litestream: litestream.io
boring hardware: talkpython.fm
DuckDB: duckdb.org
episode 491: talkpython.fm
Iceberg: iceberg.apache.org
manifesto: ducklake.select
DuckLake: ducklake.select
spec: ducklake.select
this diagram: blobs.talkpython.fm
Data inlining: ducklake.select
ducklake-dataframe: github.com
Polars course: training.talkpython.fm
CSV parser: duckdb.org
Zero-copy Arrow: duckdb.org
ART index: duckdb.org
async I/O: duckdb.org
v1.0: ducklake.select
Git-like branching: ducklake.select

Watch this episode on YouTube: youtube.com
Episode #562 deep-dive: talkpython.fm/562
Episode transcripts: talkpython.fm

Theme Song: Developer Rap
🥁 Served in a Flask 🎸: talkpython.fm/flasksong

---== Don't be a stranger ==---
YouTube: youtube.com/@talkpython

Bluesky: @talkpython.fm
Mastodon: @talkpython@fosstodon.org
X.com: @talkpython

Michael on Bluesky: @mkennedy.codes
Michael on Mastodon: @mkennedy@fosstodon.org
Michael on X.com: @mkennedy