The Databricks Data Engineer
Data skew is a checkout line - how to unjam Databricks jobs without bigger clusters
12:08
Your nightly Databricks job ran long, so you open the Spark UI. Two hundred tasks. One hundred ninety-nine finished in about a minute. One is still running, and it's been running for forty. You double the cluster and rerun it. Same forty minutes. Nearly twice the bill.
The cluster was never the problem. The rule that keeps your joins and group-bys correct is the same rule that lets one task…