// data platforms · 18 October 2026 · 13 min read
Databricks vs Snowflake: what actually happens when you build the same pipeline on both
Real numbers from building the same medallion pipeline twice, not another opinion piece.
Most "Databricks vs Snowflake" content online is opinion wearing a comparison chart. Someone picks a side, usually the one they already sell or already use, and backfills the reasoning. I wanted actual evidence instead, so I built the same pipeline, on the same dataset, with the same rules, on both platforms in parallel, and recorded what happened.
The dataset is the CDC’s BRFSS public health survey, 2011 through 2015: 2,380,047 rows across five yearly files, with the business question "how does diabetes prevalence vary by US state." The pipeline is a standard medallion architecture: bronze (raw, untouched), silver (typed and validated), quarantine (anything that fails a rule, with a reason, never silently dropped), and gold (the actual analytical answer). Same layers, same validation rules, same schema contract, implemented independently on Databricks and on Snowflake.
The point of this piece isn’t to crown a winner. It’s to show you what you actually hit when you build on both: where they agree down to the decimal, where the SQL quietly means something different, what the real cost and lock-in picture looks like once you dig past the marketing pages, and where a real platform limitation showed up that neither vendor’s docs mention.
What matched exactly
This is the part that actually matters, because it’s the part that’s checkable. Loading 441,456 rows (the 2015 file) into bronze, then applying the same validation rules, produced identical row counts on both platforms at every single layer: 440,421 clean, 1,035 quarantined (all for the same rule, a BMI value outside a sane range), 0 duplicate keys, 53 states resolved in gold. Extended to all five years, that becomes 2,380,047 rows loaded identically on both, with 29,402 of 29,402 gold-table values matching value by value across platforms, and all 11 of 11 CDC-published health indicators I checked against reproduced exactly.
The strongest piece of evidence here isn’t even the platform parity, it’s that I built this pipeline’s logic twice, in two structurally different ways: once as a hand-sequenced sequence of CREATE TABLE AS SELECT statements (Databricks Free Edition), and once as a declarative Lakeflow pipeline where the engine works out execution order from the SQL itself rather than me running things in a fixed sequence (Databricks Trial).
Both independently-built pipelines agree on the five-year national prevalence trend to the decimal: 9.8% → 10.18% → 10.27% → 10.54% → 10.5%, 2011 through 2015. When two differently-built implementations of the same logic produce identical output, that’s much stronger evidence the logic is correct than either one agreeing with itself.
Platform architecture, not just this pipeline’s architecture
Both platforms can run the exact same medallion pattern, but they get there differently, and the difference is worth understanding before you pick one.
Databricks is built around separating compute from an open storage format. Data lives as Delta Lake tables on your own cloud object storage (S3, ADLS, GCS), and compute (a Spark cluster, or, increasingly, serverless SQL warehouses) attaches to that storage on demand. Unity Catalog sits on top as the governance layer across catalogs, schemas, and tables. This is the "lakehouse" pitch: one copy of data, in an open format, that both data-engineering (Spark) and BI/SQL workloads can query without duplicating it into a separate warehouse.
Snowflake is a fully-managed platform where storage and compute are also separated, but both live inside Snowflake’s own system. You don’t pick a cloud bucket and point a cluster at it; you load data in, and Snowflake manages storage internally in its own optimized, historically proprietary format. Compute is a "virtual warehouse," a cluster you can resize or suspend in seconds, and multiple warehouses can read the same data concurrently without contending for the same compute. There’s no equivalent of "bring your own Spark cluster" because there’s no cluster to bring, there’s only Snowflake’s own query engine.
The practical difference I hit building on both: Databricks gives you more levers (cluster sizing, Spark configuration, choice of engine) and more moving parts to manage. Snowflake gives you fewer levers and a more uniform experience, at the cost of being more fully inside one vendor’s own execution engine. Neither is "simpler," they’re simple in different places.
Where the platforms actually diverge, found by building, not by reading docs
This is where a side-by-side build earns its keep: you find the differences that don’t show up in a feature comparison table, because they only show up when the same SQL means something different.
Two real SQL dialect differences, both found live, not anticipated:
- Snowflake has no
FILTER (WHERE ...)clause on aggregates, which Databricks/Spark supports. Not a style preference, a real syntax error. The portable fix is the olderCOUNT(CASE WHEN ... THEN 1 END)pattern, which works on both. - Snowflake’s
CONCAT_WSreturnsNULLthe moment any argument isNULL. Databricks/Spark’s version skipsNULLs and concatenates whatever’s left. Tested directly:CONCAT_WS(',', NULL, 'X', NULL)returnsNULLon Snowflake,'X'on Databricks. Fixed withARRAY_TO_STRING(ARRAY_CONSTRUCT_COMPACT(...), ',')on the Snowflake side, which does drop nulls the way you’d expect.
An ANSI-mode date function that crashes instead of returning NULL. The build assumed to_date() returns NULL for an unparseable date, matching the documentation at the time. That assumption was never actually tested, because the 2015-only dataset happens to have zero impossible dates. The moment the 2011 data (63 rows of dates like 09312011, since September has no 31st) entered the pipeline, to_date() under this runtime’s ANSI SQL mode threw a real error and crashed the whole CREATE TABLE, instead of nulling the bad value the way the quarantine logic was built to expect.
try_to_date() is the version that actually returns NULL on a bad date. The fix was a one-word swap, but finding it required data that actually exercised the edge case.
A hardening gap that generalizes past this one project. Testing the pipeline’s resilience to schema drift (deliberately renaming a column, dropping a column, changing a type) surfaced something worse than the test itself was looking for: every numeric CAST in the silver build was still a plain CAST, not try_cast. That means a single malformed upstream value anywhere in a numeric column would crash the entire run, not get caught and quarantined the way the project’s own "nothing is silently dropped" rule requires.
A crash isn’t "dropped with a reason," it’s the whole pipeline failing. This is the kind of bug that’s invisible until you go looking for it on purpose, because the data on hand never happened to exercise it.
Databricks Free Edition’s real limits, confirmed by hitting them directly, not by reading a limitations page: no ability to create clusters at all (serverless-only), no cluster policies (nothing to apply them to), a single SQL warehouse whose auto-stop setting can’t be edited ("Editing warehouse is not available to you"), and, most relevant to the cost section below, no system.billing.usage table at all.
The same actions (editing a warehouse, creating a pipeline) succeeded immediately on a Trial workspace. If you’re evaluating Databricks on its free tier, know you’re evaluating a meaningfully different product than the paid tiers, not a smaller version of the same thing.
Cost: the two billing models aren’t shaped the same way
Pricing changes often enough, and varies enough by cloud, region, and commitment level, that quoting a specific dollar figure here would be stale by the time you read it. What’s stable is the shape of each model, and that shape matters more than the exact number.
Databricks bills in DBUs (Databricks Units), a normalized measure of compute consumption, metered by the second, with the DBU rate itself varying by compute type (classic vs serverless), pricing tier, and cloud. Classic compute shows up as two separate bills: the DBU charge from Databricks, plus a separate cloud-infrastructure bill from AWS/Azure/GCP for the underlying VMs. Serverless compute folds the infrastructure cost into one DBU rate, so you see a single number, but you’re trading transparency for simplicity.
There’s no up-front cost on the pay-as-you-go plan; committing to usage in advance unlocks a lower rate (Databricks’ own pricing page lays out the mechanism, with exact SKU rates in its per-cloud price list, since those numbers move).
Serverless is the more expensive per-unit option on both platforms, by design, and that’s easy to miss until you’re running at real scale. Databricks’ own price lists put serverless SQL warehouse DBUs at roughly 2-3x the rate of a classic warehouse (list-price guides put classic around $0.22/DBU against serverless around $0.70-0.75/DBU on AWS Premium), because you’re paying to not manage infrastructure or wait for a cluster to spin up.
That trade is a clear win for bursty, intermittent workloads with real idle time between queries, since a classic cluster billed for its full uptime wastes money sitting idle. It stops being a clear win the moment a workload runs near-continuously: at sustained high utilization, a classic warehouse on committed/reserved pricing typically comes out cheaper than paying the serverless premium around the clock.
Snowflake has the mirror version of this problem with its own serverless features. Snowpipe, automatic clustering, and materialized view maintenance all bill as separate serverless compute outside your regular warehouse credits, and teams that lean on those features heavily at scale are often surprised by that line item specifically, since it doesn’t show up in the "warehouse size" mental model most people budget against.
The practical rule on both platforms: serverless is the right default to start with, and the wrong default to leave unexamined once a workload’s shape stops being bursty.
Snowflake bills in credits, and the model is more uniform: every warehouse size step doubles the credit-per-hour rate (an X-Small is the base rate, a Large is roughly 8x that, and so on up through 6X-Large), billing is per-second once a warehouse is running, with a 60-second minimum charge every time a warehouse starts, resumes, or is resized upward. A suspended warehouse costs nothing.
The dollar value of a credit itself depends on your edition (Standard, Enterprise, Business Critical) and region. Compute is reported to be the large majority of most customers’ bills, which means warehouse sizing and auto-suspend settings decide your spend far more than data volume does.
The real finding from actually trying to measure both: Snowflake’s cost tooling works immediately, even on a plain trial account. CREATE RESOURCE MONITOR ... TRIGGERS ON 50 PERCENT DO NOTIFY ON 100 PERCENT DO SUSPEND gave a genuine, enforced credit ceiling with zero setup friction.
Databricks Free Edition, by contrast, has no billing table to query at all; cost on that tier isn’t just unmeasured, it’s unmeasurable by design, because there’s no bill behind a free tier. If your actual evaluation plan involves a free-tier proof of concept before committing budget, know that Databricks’ free tier can’t show you a real cost signal the way Snowflake’s trial can.
Vendor lock-in: closer than it used to be, not identical
This is the question everyone asks and almost nobody checks recently, because both platforms have moved in the last two years.
Databricks’ pitch is openness at the storage layer. Delta Lake is an open-source table format, and Databricks ships UniForm, which lets a Delta table also be read as an Apache Iceberg table without duplicating the data, specifically so you aren’t locked into reading your own data only through Databricks’ engine.
In June 2024, Databricks open-sourced Unity Catalog itself, framing the move explicitly around avoiding vendor lock-in, and the open-sourced catalog supports both the Iceberg REST Catalog and Hive Metastore interfaces, meaning other engines can in principle talk to it too.
Snowflake has moved toward openness too, from the opposite direction. Historically, Snowflake managed data in its own internal, proprietary storage format, full stop. That’s changed: Snowflake’s Iceberg Tables reached general availability in June 2024, and by the 2026 Snowflake Summit the platform was reporting GA support for Iceberg v3 with full read/write interoperability across external query engines via its Horizon Catalog (built on Apache Polaris), meaning a non-Snowflake engine can now read and write data that Snowflake manages, not just read it.
The honest caveat on both sides: a feature existing isn’t the same as a vendor not wanting you locked in. lakeFS published a critical piece in 2026 arguing Unity Catalog is becoming the new choke point for Databricks’ otherwise-open stack, reporting that some customers were told future features would be Unity-Catalog-specific. Worth noting that lakeFS sells a competing product, so weigh the framing accordingly, but the underlying claim (that the catalog, not the file format, is where lock-in risk is migrating) is a reasonable thing to ask your account rep about directly, on either platform.
The safest practical read: both platforms now support genuinely open table formats at the storage layer, which is real progress over a few years ago, but the governance/catalog layer on top is where you should actually scrutinize a contract before assuming you can walk away cleanly.
The wall nobody writes about: Delta Sharing’s undocumented restriction
Phase 4 of this build was testing whether an external consumer, with no Databricks account and no Spark, could read the gold table directly. I built the real infrastructure: a SHARE, a MATERIALIZED VIEW add to that share (not a plain ADD TABLE, a real distinct syntax requirement for Lakeflow materialized views), an OPEN RECIPIENT, and an activation-link credential, then connected with a genuinely external Python client using the open-source delta-sharing library.
It worked, partially: the client connected and listed the shared tables in 2.25 seconds, real proof the share/recipient/credential chain functions end to end.
Querying actual data failed, with a real, specific error: DS_GATEKEEPER_DISABLED: "This feature is disabled for the account. Please contact your data provider." This isn’t a documented public setting. It’s not fixed by the account-console "external sharing" toggle, which was already enabled. It’s a real platform restriction that doesn’t show up until you actually try to do the thing the feature is for.
Where this comparison is honestly limited
A fair comparison says what it didn’t measure, not just what it did:
- Region mismatch. This build ended up with Databricks in AWS Ohio and Snowflake in AWS Mumbai, and a Snowflake account’s region can’t be changed after signup. In-platform query timings (server-side execution time) are still fair to compare; anything involving network distance from a client (load times, client-side query latency) isn’t, and is labeled as such throughout.
- Scale. 2.38 million rows is a real dataset, not a toy, but it’s nowhere near the hundred-million-row range where cost-per-row and warehouse-sizing tradeoffs actually start to bite differently between platforms. Extrapolating this build’s cost behavior to a much larger one would be a guess, not a measurement.
- One engineer, not two. The same person built and measured both platforms, so there’s no independent cross-check between builders, only between the two independently-built pipeline implementations described above.
- A single load, not a benchmark suite. Each load ran once, because repeating a load burns real credits. A single number has no error bars.
What this actually tells you, if you’re the one choosing
Not "Databricks wins" or "Snowflake wins," because that’s exactly the opinion-first framing this whole exercise was trying to avoid. What held up, concretely:
- If your evaluation starts on a free tier, check what that tier can actually tell you before trusting it. Databricks Free Edition cannot expose cost data at all; it’s a genuinely different product shape than the paid tiers, not a smaller version of the same one.
- "Same SQL" is not a safe assumption across these two engines. Budget real time for dialect differences in anything with aggregates, string functions, or date parsing, especially ones that fail silently (wrong result) rather than loudly (a syntax error): the
CONCAT_WSNULL behavior is the dangerous kind precisely because it doesn’t crash. - Vendor lock-in is a catalog/governance question now, more than a file-format one. Both platforms support genuinely open table formats at the storage layer in 2026. Push on the catalog layer specifically when you’re evaluating a long-term commitment.
- Features that work in the console can still fail in production for undocumented reasons. The Delta Sharing wall here passed every documented setup step and still failed on the one thing it exists to do. Build a real end-to-end test before you depend on a feature, not just a setup checklist.
The full build, every SQL file, and the schema contract both platforms were held to are public: github.com/thisismairaj/data-lab2.
Muhammad Mairaj is a Senior Backend Engineer with 8 years building payment and data systems. He writes about data engineering, measured. More about him.