← Problems/Find the Skewed PartitionMedium

Find the Skewed Partition

SparkPartitioning

Given a Spark DataFrame of 1B user events, the current partition scheme creates severe data skew — 80% of data lands in 3 partitions. Your task is to identify the skewed partitions by computing partition sizes, then implement a salting strategy to distribute the load evenly across all executors.

Constraints

•Use PySpark 3.5+ DataFrame API
•No hardcoding partition counts
•Handle up to 256 partitions
•Execution must complete within 30s on 8-core cluster

Expected Output

Partition sizes: [23401, 19876, 45621, 3210, ...]
Max size: 45621, Min size: 3210
After fix - Max: 5123, Min: 4890

Related Challenges

Solution

Output

Run your code to see output here.

Difficulty

Medium

Submissions

4,281

Acceptance

68%