How Spark's Catalyst Optimizer Rewrites Your Query Plan

dTechStack Team

May 4, 2026

8 min read·Apache Spark

SparkCatalystQuery OptimizationDataFrame

A step-by-step breakdown of Catalyst's four phases — analysis, logical optimisation, physical planning, and code generation.

📬 Weekly Newsletter

Get one deep-dive article every Saturday. No spam. Unsubscribe anytime.

When you write a Spark DataFrame transformation, you're not describing how to compute the result — you're describing what result you want. Spark's Catalyst optimizer takes that intent and figures out the most efficient physical plan to execute it.

Understanding Catalyst is the difference between a data engineer who tunes Spark by guessing and one who reads a query plan and knows exactly what to fix.

The Four Phases of Catalyst

Catalyst transforms your logical description into a physical execution plan through four phases:

1. Analysis — Resolves column names against the schema. If you reference a column that doesn't exist, this phase throws an AnalysisException.

2. Logical Optimization — Applies rule-based transformations: predicate pushdown (move filters as early as possible), column pruning (read only columns you actually use), constant folding (evaluate 1 + 1 to 2 at compile time), and boolean simplification.

3. Physical Planning — Translates the logical plan into one or more physical plans. Catalyst uses a cost model to pick the best one — choosing between sort-merge join, broadcast join, or shuffle hash join based on table statistics.

4. Code Generation — Compiles the chosen physical plan into JVM bytecode using Janino. This whole-stage code generation fuses multiple operators into a single tight loop, eliminating virtual function call overhead.

Reading EXPLAIN Output

The fastest way to see Catalyst in action is EXPLAIN:

sql

-- Run this in Spark SQL or spark.sql(...)
EXPLAIN EXTENDED
SELECT customer_id, SUM(amount) as total
FROM orders
WHERE status = 'completed'
GROUP BY customer_id
HAVING SUM(amount) > 1000;

⚡dTechStack Pro

Get unlimited access to all Apache Spark tutorials & animated diagrams

1,000+ animated tutorials · 100+ AI tools · Mock interviews · Certifications

Try Pro free →

In the output, look for:

== Parsed Logical Plan == — Your query as written, before any resolution.

== Analyzed Logical Plan == — Columns resolved to their actual data types.

== Optimized Logical Plan == — After predicate pushdown and column pruning. Notice the Filter node appears above the Aggregate — Catalyst moved the WHERE filter before the GROUP BY.

== Physical Plan == — The actual execution strategy. Look for:

`*(1) Filter` — pushed down

`*(2) HashAggregate` — partial aggregation before shuffle

`Exchange hashpartitioning` — the shuffle boundary

`*(3) HashAggregate` — final aggregation after shuffle

When Catalyst Can't Optimize

Catalyst has limits:

UDFs are black boxes. A Python or Scala UDF can't be optimized by Catalyst — the optimizer can't see inside it. Always prefer built-in functions over UDFs.

Complex outer joins. Catalyst sometimes can't push predicates through outer joins correctly (it would change semantics). Check your EXPLAIN output if a filter appears after a join.

Stale statistics. The cost-based optimizer (CBO) relies on table statistics for join ordering. Run ANALYZE TABLE to update them, especially after large data loads.