When processing large datasets in Databricks using PySpark, performance depends heavily on how well your cluster resources are utilized — specifically, executors, cores, and partitions. Databricks using PySpark executors, cores, and partitions In this blog, we’ll break down exactlyhow to calculate the number of parallel tasks, understand cluster behavior, and see a real-world example with performance observations. how to calculate the number of parallel tasks real-world example Concept Overview Concept Overview Before diving into the calculations, let’s understand the key Spark components: Term

Description



Executor

A JVM process launched on a worker node is responsible for running tasks.



Core

Each core represents a single thread of execution — 1 task runs per core.



Task

The smallest unit of work in Spark (usually processes one partition).



Parallelism

The number of tasks   Spark can execute simultaneously.



Partition

Logical chunk of data   Spark processes in parallel — one task per partition. Term

Description



Executor

A JVM process launched on a worker node is responsible for running tasks.



Core

Each core represents a single thread of execution — 1 task runs per core.



Task

The smallest unit of work in Spark (usually processes one partition).



Parallelism

The number of tasks   Spark can execute simultaneously.



Partition

Logical chunk of data   Spark processes in parallel — one task per partition. Term

Description Term Term Term Description Description Description Executor

A JVM process launched on a worker node is responsible for running tasks. Executor Executor Executor A JVM process launched on a worker node is responsible for running tasks. A JVM process launched on a worker node is responsible for running tasks. Core

Each core represents a single thread of execution — 1 task runs per core. Core Core Core Each core represents a single thread of execution — 1 task runs per core. Each core represents a single thread of execution — 1 task runs per core. Task

The smallest unit of work in Spark (usually processes one partition). Task Task Task The smallest unit of work in Spark (usually processes one partition). The smallest unit of work in Spark (usually processes one partition). Parallelism

The number of tasks   Spark can execute simultaneously. Parallelism Parallelism Parallelism The number of tasks   Spark can execute simultaneously. The number of tasks   Spark can execute simultaneously. Partition

Logical chunk of data   Spark processes in parallel — one task per partition. Partition Partition Partition Logical chunk of data   Spark processes in parallel — one task per partition. Logical chunk of data   Spark processes in parallel — one task per partition. In short: More cores = more parallel tasks = faster processing (up to a point). More cores = more parallel tasks = faster processing Example Cluster Configuration Example Cluster Configuration Parameter

Description

Value



Number of Worker Nodes

Total compute nodes (excluding the driver)

10



Executors per Node

Executors running on each node

4



CPU Cores per Executor

Number of CPU cores allocated per executor

5



Memory per Executor

Memory allocated per executor

16 GB Parameter

Description

Value



Number of Worker Nodes

Total compute nodes (excluding the driver)

10



Executors per Node

Executors running on each node

4



CPU Cores per Executor

Number of CPU cores allocated per executor

5



Memory per Executor

Memory allocated per executor

16 GB Parameter

Description

Value Parameter Parameter Parameter Description Description Description Value Value Value Number of Worker Nodes

Total compute nodes (excluding the driver)

10 Number of Worker Nodes Number of Worker Nodes Total compute nodes (excluding the driver) Total compute nodes (excluding the driver) 10 10 10 Executors per Node

Executors running on each node

4 Executors per Node Executors per Node Executors running on each node Executors running on each node 4 4 4 CPU Cores per Executor

Number of CPU cores allocated per executor

5 CPU Cores per Executor CPU Cores per Executor Number of CPU cores allocated per executor Number of CPU cores allocated per executor 5 5 5 Memory per Executor

Memory allocated per executor

16 GB Memory per Executor Memory per Executor Memory allocated per executor Memory allocated per executor 16 GB 16 GB 16 GB Step-by-Step Calculation Step-by-Step Calculation Total Executors
Total Executors  = Nnode  x Eper_node
= 10 x 4 = 40 executors
Each of the 10 nodes runs 4 executors, giving 40 total executors.


Total CPU Cores
Total CPU Cores = Total Executors x Cper_executor
= 40 x 5 = 200 cores
That means your cluster can process 200 tasks in parallel.


Number of Parallel Tasks
In Spark, each task uses one CPU core:
Parallel Tasks x Total CPU Cores = 200
So 200 partitions can be processed at the same time. Total Executors
Total Executors  = Nnode  x Eper_node
= 10 x 4 = 40 executors
Each of the 10 nodes runs 4 executors, giving 40 total executors. Total Executors Total Executors Total Executors  = Nnode  x Eper_node Total Executors  = Nnode  x Eper_node = 10 x 4 = 40 executors = 10 x 4 = 40 executors Each of the 10 nodes runs 4 executors, giving 40 total executors. 40 total executors Total CPU Cores
Total CPU Cores = Total Executors x Cper_executor
= 40 x 5 = 200 cores
That means your cluster can process 200 tasks in parallel. Total CPU Cores Total CPU Cores Total CPU Cores = Total Executors x Cper_executor Total CPU Cores = Total Executors x Cper_executor = 40 x 5 = 200 cores = 40 x 5 = 200 cores That means your cluster can process 200 tasks in parallel. 200 tasks in parallel Number of Parallel Tasks
In Spark, each task uses one CPU core:
Parallel Tasks x Total CPU Cores = 200
So 200 partitions can be processed at the same time. Number of Parallel Tasks Number of Parallel Tasks In Spark, each task uses one CPU core: each task uses one CPU core Parallel Tasks x Total CPU Cores = 200 Parallel Tasks x Total CPU Cores = 200 So 200 partitions can be processed at the same time. So 200 partitions can be processed at the same time. Cluster Visualization Cluster Visualization The diagram below shows the concept of executors, cores, and tasks in a simplified Databricks cluster executors, cores, and tasks Data & File Example (Real-time Scenario) Data & File Example (Real-time Scenario) Let’s assume we are processing a Parquet file stored in ADLS with the following details: Parquet file Parameter

Description



File Format

Parquet



File Size

100 GB



Number of Rows

250 Million



Columns

60



Cluster Type

Databricks Standard (10-node cluster) Parameter

Description



File Format

Parquet



File Size

100 GB



Number of Rows

250 Million



Columns

60



Cluster Type

Databricks Standard (10-node cluster) Parameter

Description Parameter Parameter Parameter Description Description Description File Format

Parquet File Format File Format Parquet Parquet File Size

100 GB File Size File Size 100 GB 100 GB Number of Rows

250 Million Number of Rows Number of Rows 250 Million 250 Million Columns

60 Columns Columns 60 60 Cluster Type

Databricks Standard (10-node cluster) Cluster Type Cluster Type Databricks Standard (10-node cluster) Databricks Standard (10-node cluster) Partition Calculation and Parallelism Partition Calculation and Parallelism By default, Spark creates partitions automatically. However, for large datasets, it’s better to define a target partition size — typically between 128 MB and 512 MB per partition. target partition size 128 MB and 512 MB Let’s calculate: Number of Partitions = 100 GB ⁄256 MB = 102400 ⁄ 256 = 400 partitions With 200 cores, Spark will process: 200 cores 200 tasks in the first wave
200 tasks in the second wave 200 tasks in the first wave 200 tasks in the first wave 200 tasks in the second wave 200 tasks in the second wave Total = 400 tasks processed in 2 waves 400 tasks processed in 2 waves Performance Observation (Approximate) Performance Observation (Approximate) Stage

Description

Approx Time

Remarks



Stage 1 (Read & Filter)

Reading Parquet and applying filters

~3 mins

Data is distributed evenly across executors.



Stage 2 (Transformations)

Joins and aggregations

~5 mins

CPU-heavy but parallelized well due to 200 cores.



Stage 3 (Write Stage)

Writing output as Delta format

~2 mins

Write parallelism depends on output partitions.



Total Job Runtime

—

~10 mins

Efficient partitioning and balanced task distribution.



If only 50 partitions

—

~25 mins

Underutilization — fewer tasks → idle cores.



If 2000 partitions

—

~12–13 mins

Slight slowdown due to scheduling overhead. Stage

Description

Approx Time

Remarks



Stage 1 (Read & Filter)

Reading Parquet and applying filters

~3 mins

Data is distributed evenly across executors.



Stage 2 (Transformations)

Joins and aggregations

~5 mins

CPU-heavy but parallelized well due to 200 cores.



Stage 3 (Write Stage)

Writing output as Delta format

~2 mins

Write parallelism depends on output partitions.



Total Job Runtime

—

~10 mins

Efficient partitioning and balanced task distribution.



If only 50 partitions

—

~25 mins

Underutilization — fewer tasks → idle cores.



If 2000 partitions

—

~12–13 mins

Slight slowdown due to scheduling overhead. Stage

Description

Approx Time

Remarks Stage Stage Stage Description Description Description Approx Time Approx Time Approx Time Remarks Remarks Remarks Stage 1 (Read & Filter)

Reading Parquet and applying filters

~3 mins

Data is distributed evenly across executors. Stage 1 (Read & Filter) Stage 1 (Read & Filter) Stage 1 (Read & Filter) Reading Parquet and applying filters Reading Parquet and applying filters ~3 mins ~3 mins Data is distributed evenly across executors. Data is distributed evenly across executors. Stage 2 (Transformations)

Joins and aggregations

~5 mins

CPU-heavy but parallelized well due to 200 cores. Stage 2 (Transformations) Stage 2 (Transformations) Stage 2 (Transformations) Joins and aggregations Joins and aggregations ~5 mins ~5 mins CPU-heavy but parallelized well due to 200 cores. CPU-heavy but parallelized well due to 200 cores. Stage 3 (Write Stage)

Writing output as Delta format

~2 mins

Write parallelism depends on output partitions. Stage 3 (Write Stage) Stage 3 (Write Stage) Stage 3 (Write Stage) Writing output as Delta format Writing output as Delta format ~2 mins ~2 mins Write parallelism depends on output partitions. Write parallelism depends on output partitions. Total Job Runtime

—

~10 mins

Efficient partitioning and balanced task distribution. Total Job Runtime Total Job Runtime Total Job Runtime — — ~10 mins ~10 mins ~10 mins Efficient partitioning and balanced task distribution. Efficient partitioning and balanced task distribution. If only 50 partitions

—

~25 mins

Underutilization — fewer tasks → idle cores. If only 50 partitions If only 50 partitions If only 50 partitions — — ~25 mins ~25 mins ~25 mins Underutilization — fewer tasks → idle cores. Underutilization — fewer tasks → idle cores. If 2000 partitions

—

~12–13 mins

Slight slowdown due to scheduling overhead. If 2000 partitions If 2000 partitions If 2000 partitions — — ~12–13 mins ~12–13 mins ~12–13 mins Slight slowdown due to scheduling overhead. Slight slowdown due to scheduling overhead. Performance Insights Performance Insights Configuration Change

Performance Impact



Increase executors or cores

Improves parallelism, reduces runtime



Too few partitions

CPU underutilized, slower



Too many small partitions

Task scheduling overhead increases



Balanced partitions (~256 MB each)

Best performance



Use Delta Format

Faster reads/writes with optimized layout Configuration Change

Performance Impact



Increase executors or cores

Improves parallelism, reduces runtime



Too few partitions

CPU underutilized, slower



Too many small partitions

Task scheduling overhead increases



Balanced partitions (~256 MB each)

Best performance



Use Delta Format

Faster reads/writes with optimized layout Configuration Change

Performance Impact Configuration Change Configuration Change Configuration Change Performance Impact Performance Impact Performance Impact Increase executors or cores

Improves parallelism, reduces runtime Increase executors or cores Increase executors or cores Improves parallelism, reduces runtime Improves parallelism, reduces runtime Too few partitions

CPU underutilized, slower Too few partitions Too few partitions CPU underutilized, slower CPU underutilized, slower Too many small partitions

Task scheduling overhead increases Too many small partitions Too many small partitions Task scheduling overhead increases Task scheduling overhead increases Balanced partitions (~256 MB each)

Best performance Balanced partitions (~256 MB each) Balanced partitions (~256 MB each) Best performance Best performance Use Delta Format

Faster reads/writes with optimized layout Use Delta Format Use Delta Format Delta Format Faster reads/writes with optimized layout Faster reads/writes with optimized layout Key Takeaways Key Takeaways Parallel Tasks = Total CPU Cores
1 Task = 1 Partition = 1 Core
Tune partitions close to the number of available cores for best performance.
Monitor Spark UI → Stages tab to analyze task distribution and identify bottlenecks.
Don’t blindly increase partitions or cores — find the sweet spot for your workload. Parallel Tasks = Total CPU Cores Parallel Tasks = Total CPU Cores 1 Task = 1 Partition = 1 Core 1 Task = 1 Partition = 1 Core Tune partitions close to the number of available cores for best performance. partitions available cores Monitor Spark UI → Stages tab to analyze task distribution and identify bottlenecks. Spark UI → Stages tab Don’t blindly increase partitions or cores — find the sweet spot for your workload. sweet spot Example Summary Example Summary Metric

Value



Worker Nodes

10



Executors per Node

4



CPU Cores per Executor

5



Total Executors

40



Total CPU Cores / Parallel Tasks

200



Input File Size

100 GB



Row Count

250 Million



Partitions

400



Execution Waves

2



Approx Runtime

~10 minutes Metric

Value



Worker Nodes

10



Executors per Node

4



CPU Cores per Executor

5



Total Executors

40



Total CPU Cores / Parallel Tasks

200



Input File Size

100 GB



Row Count

250 Million



Partitions

400



Execution Waves

2



Approx Runtime

~10 minutes Metric

Value Metric Metric Metric Value Value Value Worker Nodes

10 Worker Nodes Worker Nodes 10 10 Executors per Node

4 Executors per Node Executors per Node 4 4 CPU Cores per Executor

5 CPU Cores per Executor CPU Cores per Executor 5 5 Total Executors

40 Total Executors Total Executors Total Executors 40 40 Total CPU Cores / Parallel Tasks

200 Total CPU Cores / Parallel Tasks Total CPU Cores / Parallel Tasks Total CPU Cores / Parallel Tasks 200 200 Input File Size

100 GB Input File Size Input File Size 100 GB 100 GB Row Count

250 Million Row Count Row Count 250 Million 250 Million Partitions

400 Partitions Partitions 400 400 Execution Waves

2 Execution Waves Execution Waves 2 2 Approx Runtime

~10 minutes Approx Runtime Approx Runtime Approx Runtime ~10 minutes ~10 minutes ~10 minutes Final Thoughts Final Thoughts This real-time breakdown shows how cluster configuration, task parallelism, and partition strategy directly impact Spark job runtime. cluster configuration task parallelism partition strategy Whether you’re optimizing ETL pipelines or tuning Delta writes —understanding these fundamentals can drastically improve Databricks performance and cost efficiency. understanding these fundamentals

This story contains AI-generated text. The author has used AI either for research, to generate outlines, or write the text itself. 

Walkthroughs, tutorials, guides, and tips. This story will teach you how to do something new or how to do something better.

Understanding Parallelism and Performance in Databricks PySpark

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

A Practical Guide to the Saga Pattern in Spring Boot Microservices

All About Parquet Part 02 - Parquet's Columnar Storage Model

Picking the Wrong Spark File Format Could Hurt Your Data Strategy

Building a CI Pipeline with Databricks dbx Tool and GitLab

Building an ETL Pipeline to Load Data Incrementally from Office365 to S3 using ADF and Databricks

Optimize Your Data Engine With Data as a Service (DaaS) and Multi-Tenancy

A Practical Guide to the Saga Pattern in Spring Boot Microservices

All About Parquet Part 02 - Parquet's Columnar Storage Model

Picking the Wrong Spark File Format Could Hurt Your Data Strategy

Building a CI Pipeline with Databricks dbx Tool and GitLab

Building an ETL Pipeline to Load Data Incrementally from Office365 to S3 using ADF and Databricks

Optimize Your Data Engine With Data as a Service (DaaS) and Multi-Tenancy

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps