The Databricks Data Engineer
The Databricks pipeline that ran green for a year and was wrong the whole time
8:29
A revenue-attribution pipeline runs overnight, bronze to silver to gold, and lands a clean set of numbers the whole company uses to decide where the money goes. Every morning: green. Every morning, everyone moves on with their day. For a year, nobody looked any closer.
The job succeeded every single night, so everyone trusted the data. But "finished" and "correct" are two different words, and…