// data quality · 18 November 2026 · 4 min read
How my data checks lied: a row count passed with damaged bytes
One day of Wikipedia pageviews on one laptop, and the nine failures that showed which checks to trust.
Loading rows is the easy part. Knowing none of them went missing is the part people skip, and the first checks I wrote to prove it were fooled.
Here’s the project, the nine ways I broke it on purpose, and the one failure that changed how I think about “the counts match.”
The setup
pageviews-on-a-laptop loads Wikimedia’s hourly pageview dumps on one ordinary laptop: an Intel i5–10310U with 4 cores and 31.6 GB of RAM, no GPU. Every number below was measured on that machine.
One full day of data, New Year’s Day, all 24 hourly files:
- 143,351,375 rows in, 143,351,375 rows loaded, 0 quarantined.
- Download: 174 seconds over two connections.
- Load: 146 seconds, about 0.98 million rows a second, including writing Parquet and checksums.
- A second run of the same day: 2 seconds, nothing changed.
Fast enough. But speed was never the point.
The one rule everything hangs on
Every input line ends up in exactly one of two places: the good Parquet file, or a quarantine file with the raw line and a reason. Nothing is dropped silently.
So the rule is simple: rows in = rows loaded + rows quarantined.
The trick is who counts. “Rows in” is counted by plain Python reading the gzip, not by the loader. “Loaded” and “quarantined” are read back from the files on disk. The loader gets checked against a number it didn’t produce. If the sums don’t match, it deletes its output and marks the hour failed, instead of leaving a plausible-looking partial result behind.
That last bit matters. A crash is easy to notice. A table that’s 97% complete and looks fine is what ends up in a dashboard.
Breaking it on purpose
A check you’ve never seen fail is a check you’re trusting on faith. So I wrote a chaos script that builds a fresh data directory from real hourly files, breaks one thing, and records what happened. Nine breaks:
- kill the process mid-download,
- kill it mid-load,
- truncate a raw file,
- flip a single bit in a raw file,
- the same content under two names,
- a missing hour,
- 600 garbage lines inside a valid file,
- bit rot inside a Parquet file after it was loaded,
- two loaders running at the same time.
Kills are real kills. On Windows that’s TerminateProcess, so no cleanup code runs in the victim.
The first run: 5 of 9 passed.
Failure 1: my own harness
Most of the first failures were mine, not the loader’s.
The check that every day has 24 hours reported 21 missing hours. Correct! The test only loaded 3. The tempting fix is to loosen the check. I added an –expected-hours option instead, so the test says what it expects and the check stays strict.
If your test harness is wrong, every result it gives you is a guess. Check the checker first.
Failure 2: the row count that passed with damaged bytes
This is the one worth the whole post.
I damaged 64 bytes in the middle of a good Parquet file. Then I ran reconcile. It said every row was accounted for.
It was right about the rows. Parquet keeps its row count in the file’s footer, and the footer was intact. The count matched because the count never looked at the data.
A row count proves the shape. It doesn’t prove the content.
The fix: record a sha256 of each Parquet file when it’s written, and re-check it during reconcile. I measured the cost on the full day: about 9 extra seconds on the 146-second load, and a 4-second reconcile over 1.3 GB. Cheap, for knowing the bytes are the bytes you wrote.
I’d been treating “the counts match” as the end of the question. It’s the start. This is the same trap I wrote about in the scarce skill in data engineering is noticing a plausible wrong answer: a check that passes is not the same as a check that’s right.
Failure 3: two loaders, one lucky pass
Two loaders at once passed the first time. That pass was luck.
The next run, one loader hit Windows “Access is denied” renaming a file the other had open, and the hour got marked failed. Behind that was a second bug: only download failures were ever retried. A failed load stayed failed forever.
Two fixes:
- An exclusive lock per data directory, held as a SQLite lock on its own file. The operating system releases it even if the holder is killed. A PID file would go stale after a kill and block every run after it.
- A separate load_failed status that the next load retries.
Now the second loader refuses to start, exits with code 3, and the first one finishes.
Two small bugs the tests caught first
Before any real file hit them:
- A one-byte delimiter collision. The first loader used one rare byte to read each line as a single value. A test with that byte inside a page title made DuckDB refuse the whole file. Now it’s a three-byte sequence.
- An exception I didn’t catch. My gzip check caught OSError and EOFError. A flipped bit inside the stream raises zlib.error, which is neither. A one-bit test found it.
What this doesn’t cover
Disk full. Power loss in the middle of a rename. A source that quietly serves different bytes under the same name. One machine, so it won’t scale past one disk. Peak memory and CPU during the load: not measured yet.
Checksums tell you something changed after you wrote it. They can’t tell you the source was right in the first place.
All nine breaks pass now, with the full evidence for each in the repo’s results file. If you’ve read the small file problem, measured on disk and in Azure, this is the other half of the same habit: measure the thing, then break the thing.
A row count is a promise about shape. Make your checks read the bytes.
Muhammad Mairaj is a Senior Backend Engineer with 8 years building payment and data systems. He writes about data engineering, measured. More about him.