The Databricks Data Engineer
The Spark Shuffle is baggage claim: why your job waits instead of computes (and more workers won't fix it)
11:09
Your Spark job has been running for forty minutes. The dashboard shows your cluster isn't even busy. So you do the obvious thing: add more workers. And it changes nothing.
Here's why. During a shuffle, Spark is barely computing at all. It's tagging every row by destination, piling rows together, spilling the overflow to disk, and hauling data across the network between executors. It's an…