Skip to content

// data lakes and query engines · 20 October 2026 · 5 min read

An open table format is only as open as the tool that wrote it

I saved the same table as Delta and Iceberg. The format was never what made them differ.

Delta Lake and Apache Iceberg gave me almost the same result. What changed the result, three separate times, was the program that wrote the files.

That matters because “open format” sounds like a promise: save it once, open it anywhere. It is a promise about the rules. It says very little about the tool that follows them.

First, the vocabulary

A few terms come up a lot, so here is each one in a line.

  • A table format is a set of rules for how the files behind a table are laid out, plus a small bookkeeping record of which files are current. Delta Lake and Apache Iceberg are the two big ones.
  • Parquet is the file type that holds the actual rows. Both formats use it.
  • A writer is the program that saves the table. A reader is a program that opens it later.
  • An open format means any reader can open what any writer saved.

The setup

I took real data: Wikipedia’s hourly page-view counts, which Wikimedia publishes for anyone to use. I wrote the first 2,000,000 rows of one hour as a table in both formats, then added 500,000 more rows, deleted one page, and changed the columns.

The writers were two Python libraries, delta-rs for Delta and PyIceberg for Iceberg. The readers were DuckDB on my laptop and a Databricks SQL warehouse in the cloud. Everything ran on one table, so treat the results as observations, not a verdict.

Surprise one: Delta looked 26% bigger. It wasn’t.

My first run said the Delta data took 28,162,105 bytes and the Iceberg data 20,838,814, for the same rows. I almost wrote that sentence down as a finding.

Then I looked inside the Parquet files. delta-rs had compressed with Snappy and PyIceberg with ZSTD. Different compression, nothing to do with the formats. With both set to ZSTD the files came out at 20,667,639 and 20,838,814 bytes.

Compression is a writer setting. Compare two formats on default settings and you may be comparing two defaults.

What each format leaves on disk

Delta keeps the Parquet files plus a folder called _delta_log. Each change adds one small JSON file, about 1 KB here.

Iceberg keeps the Parquet files plus a metadata folder. Each change added three or four small files, about 9 to 14 KB in total.

By the end of the run (a write, an append, a delete and a column change), Delta’s log was 4,046 bytes and Iceberg’s metadata about 39 KB. Both are tiny next to 46 MB of data. I did not measure what happens after thousands of changes, so I can’t say how that grows.

Deleting one row rewrites a whole file

I deleted every row for one page. In both formats the tool wrote a new file of nearly the whole table, about 19.4 MB for Delta and 20.8 MB for Iceberg, for one deleted page. The old files stayed on disk until a cleanup, because older versions of the table still point at them.

Both formats have newer features meant to avoid this rewrite. I did not try them, so this is not a verdict on either one.

Changing columns

In Iceberg, adding a column and renaming one each wrote a single small metadata file and touched no data. The old rows stayed readable under the new name.

In Delta, adding a column wrote one small log entry. Renaming failed, but only because my Python library has no rename call. Delta itself can rename, through a feature called column mapping, which its docs describe as an upgrade you can’t undo. I didn’t test that route.

Surprise two: the Iceberg folder wouldn’t open in Databricks

I uploaded both folders to Databricks and queried them by path. The Delta table came back with 2,499,999 rows and 10,318,574 total views, exactly what DuckDB had said.

The Iceberg folder returned no columns at all.

My best explanation is paths. The Iceberg bookkeeping files store full addresses, like D:/oss/open-table-formats/.... The Delta log stores only file names, relative to its own folder. Copy a Delta folder and it still works. Copy an Iceberg folder written to a local disk and it points at a drive that doesn’t exist anywhere else. I didn’t rewrite the paths and retry, so I can’t call that proven.

It also showed why the bookkeeping matters. Reading the Iceberg data folder as plain Parquet gave 4,499,999 rows instead of 2,499,999, because it counted the old files that were no longer part of the table. The files hold the rows. The metadata says which files count.

Surprise three: Databricks wrote Iceberg with cloud paths, and DuckDB still couldn’t read it

Next I let Databricks write the Iceberg table itself. This time the addresses were cloud addresses, s3://..., as you’d hope. The table worked inside Databricks.

Then I tried to read it from DuckDB on my laptop, through Databricks’ Iceberg endpoint for outside engines. DuckDB logged in, found the table, and followed the address to the file in S3. S3 answered 403 Forbidden.

The login worked for the catalog, which lists the tables. It didn’t work for the storage, which holds the files. Think of a library that lets you search its catalog but keeps the stacks locked.

Databricks’ docs say outside engines need “external data access” turned on and a privilege on the schema. I turned the setting on and tried again: still 403. Then I tried the privilege, and Databricks refused it as “not applicable” for that schema, which uses default storage. That fits the idea that default storage doesn’t allow outside readers. I didn’t prove it, because I didn’t try other storage setups.

What I’d take from it

  • Before comparing two formats, check the writer’s defaults. Compression alone made a false 26% gap.
  • Before moving a table, check whether its paths are relative or absolute, and whether they point at your disk or at the cloud.
  • “Open” means the rules are public. It doesn’t mean every reader can reach the files. Permissions are a separate question.

Years of building payment systems taught me that two systems can both claim to follow the same standard and still fail to reconcile on the details. Tables are no different.

What I didn’t test

Snowflake, because it needs a cloud storage bucket I didn’t have. Deletion vectors and Iceberg delete files. Cleanup of old files. Thousands of changes. Databricks’ own routes for reading Iceberg that isn’t managed by it. Delta’s column mapping. Writers other than these two libraries. It is one table on one laptop and one trial workspace.

Same format, different writer, different result. Check the writer first.


Related reading: What Is Inside an Iceberg Table? walks through the metadata layers of an Iceberg table using DuckDB.

Muhammad Mairaj is a Senior Backend Engineer with 8 years building payment and data systems. He writes about data engineering, measured. More about him.