Ends in00:00:00
2,341 participants
90 min
Spark Shuffle Optimisation
SparkKafkaSQL
Given a Spark DataFrame of 1B user events, optimise the repartitioning strategy to minimise shuffle spill. Implement a custom partitioner that distributes keys by hash range to reduce data skew.
Test Cases
Input:
numPartitions=200Expected:
Shuffle spill < 10MBInput:
numPartitions=400Expected:
Shuffle spill < 5MBContest Info
Duration
90 min
Participants
2,341
Prize
🏆 $200 + Certification
Difficulty
Medium
Leaderboard
1YouYou
3/4 solved · 45 min
2Ananya S.
3/4 solved · 52 min
3Rahul V.
3/4 solved · 61 min
4Priya K.
2/4 solved · 28 min
5Amit R.
2/4 solved · 43 min
Solution
1
Output
Run your code to see output here.