Overview
This lecture explains how to tune Spark executors by choosing the optimal number of executors, cores, and memory for efficient Spark job performance, with examples and best practices.
Understanding Executors
- An executor in Spark is a process that performs tasks and stores data on a worker node.
- Configuring executors involves setting their number, CPU cores, and memory per executor at Spark submit.
- Allocating resources without following proper logic can lead to inefficient or slow jobs.
Executor Sizing Approaches
- Three approaches: fat executors (large resource share), thin executors (minimal resources), and optimally sized executors (balanced).
- Example cluster: 5 nodes, each with 12 cores and 48GB RAM.
Fat Executors
- Fat executors use most resources on a node (e.g., 1 executor per node with 11 cores and 47GB RAM).
- Advantages: High task-level parallelism, good for memory-intensive tasks, enhanced data locality, fewer executors to manage.
- Disadvantages: Idle resources if not fully utilized, lower fault tolerance (large recomputation if one fails), possible HDFS throughput and garbage collection issues.
Thin Executors
- Thin executors use minimal resources (e.g., 1 core and ~4GB RAM per executor, 11 executors per node).
- Advantages: High executor-level parallelism, greater fault tolerance due to smaller workload per executor.
- Disadvantages: Increased network traffic, reduced data locality, not suitable for memory-heavy tasks.
Optimally Sized Executors
- Best practice is to balance between thin and fat executors.
- General rules:
- Leave 1 core and 1GB RAM per node for OS and system processes.
- For YARN Application Master, leave 1 core and 1GB RAM (or subtract one executor) at the cluster level.
- Assign 3-5 cores per executor to minimize garbage collection and maintain throughput.
- Exclude memory overhead (max of 384MB or 10% of executor memory) from executor memory configuration.
Examples of Optimal Sizing
Example 1: 5 nodes, each 12 cores, 48GB RAM
- After system and YARN allocation: 54 cores, 234GB RAM cluster-wide.
- Assign 5 cores per executor: 10 executors total.
- Memory per executor (after overhead): 20GB.
Example 2: 3 nodes, each 16 cores, 48GB RAM
- After system and YARN allocation: 44 cores, 140GB RAM.
- Assign 4 cores per executor: 11 executors total.
- Memory per executor (after overhead): 11GB.
Key Terms & Definitions
- Executor — Spark process running tasks and storing data on a worker node.
- Fat Executor — Executor configured with a large share of a node's resources.
- Thin Executor — Executor configured with minimal resources.
- Application Master (YARN) — Component managing resource negotiation with the cluster manager.
- Data Locality — Degree to which data needed for processing is available on the same node.
- Garbage Collection (GC) — JVM process of clearing unused memory, can pause program execution.
Action Items / Next Steps
- Practice calculating executor configurations for your cluster setup.
- Review partition sizing to ensure each core gets adequate memory.
- Remember to leave resources for system and YARN processes in your calculations.