The Databricks Data Engineer
Databricks broadcast joins: when the memo beats the meeting (and when it kills a task)
10:48
Two engineers on the same team join the same big orders table to the same small lookup table. Same cluster, same data, one line of code different. Sarah's finishes in the time it takes to get a coffee. Mike's dies, and the error isn't about the data at all. A task ran out of memory building a hash map.
Broadcast joins get passed around as a tip instead of a model: small table equals fast, flip…