Skip to content

// data quality · 2 November 2026 · 4 min read

The scarce skill in data engineering is noticing a plausible wrong answer

Photo by Ibrahim Rifath on Unsplash

Knowing a tool is cheap now. What’s scarce is a process that notices when an answer looks right and isn’t.

Two things I read this week made that clear, and one old result of mine sits right between them.

The same setup scored 61%, 100% and 67%

dbt Labs ran 22 coding-agent runs on a Formula 1 dbt project in Snowflake: about 700,000 rows, including 589,000 lap times, all loaded as plain text. The agents had to clean, type and model it across a sequence of tickets, up to 12 of them. After each ticket the harness injected new data (late races, corrected lap times, duplicate rows), rebuilt the project, and graded every required model row by row against an answer key. The agent never saw its score.

The first round was easy. Every run but one passed all 43 checks, with or without extra instructions. So they built a harder second round, and that’s where it gets interesting.

One setup, run three times, scored 61%, 100% and 67%. Run-to-run variance was bigger than any effect of the extra instructions they gave the agents. 12 of 19 Claude 5.5 runs passed every check. The agents rarely opened the instructions at all. And a newer model generation moved the score more than anything else: the older Sonnet 4.5 scored 38% without the instructions and 17% with them forced on it.

The author’s read is that the gap between good runs and bad ones was process, not dbt knowledge. The failures he lists: a test that froze today’s answers, ignored definitions, lookback windows set wrong. One ticket only mentioned a Ferrari bug, but the harness had also resent a race’s standings rows. An agent that fixed what it was told about and never looked wider got a plausible number and the wrong total.

He calls it not conclusive, and I’d say the same. It’s one afternoon or two of work on one project. But it matches what I’ve seen.

A passing test only describes what you asked

I once rebuilt a hand-written pipeline in dbt and planted the bugs myself. 258 orders came in. 250 passed and 5 went to quarantine. That’s 255.

Three orders were gone. They were exact duplicates, and one step dropped them under a comment that said “nothing is silently dropped.” All 31 tests passed anyway.

The check that would have caught it was rows in, equal to rows passed plus rows quarantined. My hand-written version had it. The dbt version didn’t, because I never wrote it. dbt runs every test you write, every time. It doesn’t write the one you forgot.

That habit isn’t from data engineering. It’s from building a settlement engine in fintech, where “every unit accounted for” was the whole job. I carried it into the same pipeline built on both Databricks and Snowflake, 2.38 million rows of health-survey data. Both platforms matched value for value, and the year-by-year row count was the check I trusted most, because a number that looks fine can still be a number from the wrong set of rows.

To be fair to the comparison: both of those were human-written pipelines, not agent runs. I’m not claiming they prove anything about agents. I’m saying the failure has the same shape.

Show me what you understood before you build it

Ananth Packkildurai, writing in Data Engineering Weekly, makes the sharpest version of this in Do we still need data models? An agent finds a click-based definition of “response” and builds the pipeline. The records arrive and the tests pass. But marketing wanted purchases within seven days of the email. A valid definition answered the wrong question.

His fix is to make the agent state its interpretation first, before any code. Keep guesses (this recipient ID is probably that customer ID) apart from confirmed facts. Leave open questions, like how to treat returns, visible instead of quietly chosen. And check what the agent says it built against the code, the data and the tests, because a plausible summary can still describe something different.

That’s quarantine at a bigger scale. A bad row gets a reason and stays where you can see it. A guess should get the same treatment.

His piece is a proposal, not a measurement, so read it as an argument. I find it convincing mostly because it describes a failure I’ve watched happen with no agent involved.

What I’d do with this

  • Write the reconciliation check first. Rows in, rows out, rows quarantined. If those don’t add up, nothing else your tests say matters.
  • Plant bugs where you know the count. Tests passing on clean data proves very little. Tests passing on data where you know exactly what’s broken proves they can see.
  • If an agent builds it, make it state its definitions before it starts. Then compare them to the code afterwards, not to its summary.
  • Look wider than the ticket. The fix you were asked for is rarely the only thing that changed.

The tool knowledge is free now. The habit worth keeping is counting the rows.

p.s. The project with the planted defects is on GitHub.

Muhammad Mairaj is a Senior Backend Engineer with 8 years building payment and data systems. He writes about data engineering, measured. More about him.