The Databricks Data Engineer

Two Databricks jobs, one Delta table, same moment: 3 endings (and the 1 that needs a human)

11:41

Two in the morning. A nightly merge is folding yesterday's orders into a Delta table. A backfill someone kicked off before logging off is correcting March. Neither job knows the other exists, and nothing in Delta would tell them.

A lot of design around this runs on superstition. Either "it's Delta, it's ACID, relax." Or "never let two jobs touch that table." Both are guesses, and the tidy…