paint-brush
PySpark Over Pandas: The Obsession of Every Data Scientistby@tusharml
767 reads
767 reads

PySpark Over Pandas: The Obsession of Every Data Scientist

by Tushar GoelNovember 20th, 2022
Read on Terminal Reader
Read this story w/o Javascript
tldt arrow

Too Long; Didn't Read

Pandas perform operations on a single machine, whereas PySpark performs operations on multiple machines. This makes it 100x times faster** than Pandas for large datasets. Pandas DataFrames are incapable of constructing a scalable application, but PySparks are ideal for developing scalable applications. As Data is increasing, the need for more frameworks like PySpARK is increasing. For small datasets of 10–12 GB, you can prefer Pandas over the same runtime and with less complexity.

Companies Mentioned

Mention Thumbnail
Mention Thumbnail
featured image - PySpark Over Pandas: The Obsession of Every Data Scientist
Tushar Goel HackerNoon profile picture


In recent years, we have seen an increase in data, and as a result, it increases computational time and memory. Therefore, tools like Pandas which works sequentially, fail to achieve the result in the required time in large datasets.


Some packages, like Dask, Swift, Ray, etc., can parallelize the panda's operation. Parallelizing the panda’s processes causes a lot of speed up. Still, there are specific memory limitations caused by your system as Pandas load your data frame into the memory, even if it is not required at that particular instance. This can be a massive problem in the case of desktop systems, as we need to keep the UI live.


We need a framework to solve the above problems and achieve parallelization under the given memory limitations. Spark solves some of these problems under some given thresholds. In addition, it can perform faster processing speeds. It uses a concept called lazy evaluation (as the name suggests, it evaluates Data when required), which solves some of the limitations caused by memory.

Advantages of PySpark over Pandas

  1. Pandas perform operations on a single machine, whereas PySpark performs functions on multiple devices, making it 100x times faster than Pandas for large datasets.
  2. Pandas adhere to Eager Execution, meaning tasks are completed as soon as possible. In contrast, Pyspark follows Lazy Execution, which means that a job is not executed until an action is performed.
  3. Pandas DataFrames cannot construct a scalable application, but PySpark DataFrames are ideal for developing scalable applications.
  4. The Pandas DataFrame does not guarantee fault tolerance, but PySpark DataFrame assures fault tolerance.

When to use PySpark or Pandas?

As I mentioned earlier, PySpark is 100x faster than Pandas; it is just a half-truth. Pandas and Pyspark have the same runtime for the initial GB of data, as shown in the benchmark below.

For the initial phase, up to 20 GB, they have the same slope, but as file size increased, Pandas goes out of memory, and PySpark was able to complete the job successfully.

                          Benchmark Pandas and PySpark - [Source](https://hirazone.medium.com/benchmarking-pandas-vs-spark-7f7166984de2)


Therefore, for small datasets of 10–12 GB, you can prefer Pandas over PySpark due to the same runtime and less complexity, and above that, you have to work using PySpark.

Limitations of PySpark

Everything good at something lacks behind in the other areas. This is also the case with PySpark. Some of the limitations of Pyspark are:


  1. PySpark has a higher latency, which results in lower throughput.
  2. The consumption of memory is very high.
  3. Fewer algorithms and libraries are developed to work with PySpark.



From the above discussion, we can conclude that as Data is increasing, the need for more frameworks like PySpark is rising, which can easily handle such processes. Therefore, if you are an Aspiring Data Scientist, you should start learning about PySpark.


Originally published here.