Spark AQE: Adaptive Query Execution Deep Dive

AQE rewrites your physical plan at runtime using actual partition statistics. This tutorial walks through each of the three AQE features — partition coalescing, join conversion, and skew join handling — with live Spark UI screenshots.

Animated Explanation

STEP 1 / 5

Normally Spark plans the entire query before execution. AQE changes this: after each shuffle stage completes, Spark inspects the actual partition sizes and may re-optimize the remaining stages.

Key Concepts

Runtime plan rewritingShuffle statisticsJoin conversionSkew partition splitting

🎯 You'll Learn

Enable and tune AQE for your production jobs

📋 Prerequisites

Spark query plan basics

Stack

Apache Spark

5 tutorials

Progress

Step 1 of 520%

How Spark Repartitions a DataFrame Across 4 Executors

▶ 8 min12.4k views

How Spark's Catalyst Optimizer Rewrites Your Query Plan

▶ 15 min3.8k views

When to Use Broadcast Joins and How Spark Decides

▶ 8 min9.1k views

Liked this tutorial?

View full Apache Spark path →