Which library should I use? Apache Spark, Dask, and Pandas Performance Compared(With Benchmarks)

Python offers many data-processing libraries to choose from, each with its own strengths and weaknesses. In this post, we learn about Dask, Apache Spark, and pandas. We'll go over their features, benefits, and drawbacks. We'll also run some benchmarks on Dask, pandas, and Apache Spark to see how they perform on various tasks with different difficulty levels.

What, Where, and Why?

This section will look at what these libraries are, how they're used, and what benefits they provide.

What is Dask?

Dask is a library for managing collections of (possibly big) arrays distributed among different machines to take advantage of multiple CPUs or GPUs. It takes care of figuring out which data resides on which machine and distributing the responsibilities fairly across different machines. It also speeds up performance by dividing computations into smaller chunks that can be processed on the individual nodes where the corresponding data is stored.

If you're planning to run your application or build your website on a group of machines, Dask is the best way to utilize all the cores in your cluster. Dask is used by companies like Palantir, Airbnb, and Spotify.  Later in this post, we'll look at some real-world examples.

Dask's Features

• Supports a wide range of tasks
• Dynamic task scheduling
• It's easy to set up and run on your local machine
• Uses TLS/SSL certificates to authenticate it and to provide encryption
• Runs reliably on clusters with tens of thousands of cores

‍

Dask's Benefits

• It offers parallel computing with pandas.
• Dask has a syntax comparable to the pandas API, making learning easier.
• It has dynamic task scheduling and can handle a variety of workloads.

Dask Limitations

• Unlike Apache Spark, Dask does not provide a standalone mode that you may use to test the tool before forming a cluster.
• No Scala and R support.

Installation

//conda
conda install Dask
//pip
python -m pip install "Dask[complete]" 

Explore Dask Docs

What is pandas?

Pandas is an open-source Pythonic data analysis library. It promises to be the core high-level building block for doing realistic, real-world data analysis in Python by providing tools for working with tabular data.

Pandas is used in various industries, including data science, stock prediction, advertising, big data, and a lot more. Pandas can handle a lot of data, making it easier for you to understand the huge amounts of data required to construct a solid recommendation system(For Eg. Netflix, Prime Video, etc.).

Pandas Features

• Several changes and pivots may be made to data sets
• Slicing, indexing, and subsetting of big data sets depending on labels
• With default and customizable indexing, the DataFrame object is quick and efficient
• Offers Time Series functionality

‍

pandas Benefits

• It can process large amounts of data and save a lot of time. 
• It has a lot of customization options, making it easy to get the most out of your data.
• The use of pandas speeds up the data-handling process. In simple terms, there will be less writing and more work done.

pandas Limitations

• When dealing with 2D matrices, pandas are an excellent choice, but when working with 3D matrices, you'll need to use NumPy or another library.
• Some people may find the syntax challenging to understand. 
• The learning curve and pandas docs may challenge new developers and data scientists.

Installation

$ pip install pandas

You can learn more about pandas by referring to pandas documentation

Recommended Reading: Installing pandas 

What is Apache Spark

Apache Spark DataFrame is a distributed, in-memory analytics platform that uses SQL to work with real-time data generated from remote web servers and social media sources. Apache Spark is designed to scale across data centers and clusters, efficiently storing ever-growing amounts of data generated by Big Data applications. It supports various analytics tools, including machine learning algorithms, graph processing systems, and statistical models.

Apache Spark is used in various industries, including gaming, streaming services, healthcare, e-commerce, etc. It is used by firms like Alibaba, eBay, TripAdvisor, Pinterest, and Riot Games for their applications.

Apache Spark Features

• Uses a schematic perspective of data to handle structured data
• Catalyst is a data processing optimizer that works across many languages
• CSV, XML, JSON, RDDs, Cassandra, Parquet, and more data formats are supported
• It provides a custom memory management option to prevent overload and increase efficiency

‍

Apache Spark Benefits

• Apache Spark performs computations in-memory (RAM)
• Apache Spark provides an easy-to-use API for working with huge datasets
• It features about 90 high-level operators that may be used to build parallel programs

Apache Spark Limitations

• There is no such thing as an automated optimization process
• It does not offer many algorithms

Installation

pip install pyApache Spark

Learn more about installation: Apache Spark documentation

The Comparison

This section will compare Dask, pandas, and Apache Spark on various parameters.

Language 

• Apache Spark is written in Scala, with some Python and R compatibility. It works nicely with other programs
• Dask is written in Python and works with that language. Users are not restricted to Python when using the library, but using Python makes handling large amounts of data easier.
• pandas is a Python library, and supports many integrations

Focus

• Apache Spark is a very general-purpose cluster framework, and its massive computing power allows one to solve large amounts of data problems quickly and efficiently. It works on top of Hadoop components such as Hive, YARN, Oozie, etc.—all designed to process huge datasets at high speeds with minimal resource consumption
• Dask is a library for processing large volumes of data in Python. It has wide applicability in machine learning, optimization, statistics, etc.
• pandas is a library that helps you make sense of the data – extract key information easily. If you process enormous datasets, you will not want to go through this task manually as it can be very time-consuming and laborious

DataFrames

• The memory concept and API for Apache Spark DataFrame are unique. It also includes support for a substantial quantity of the SQL language. For complicated queries, Apache Spark features a high-level query optimizer
• Dask DataFrame reuses the pandas API and memory model. It doesn't have SQL or a query optimizer. Random access, time-series operations, and other operations are all possible.

Performance

Performance can vary depending on your application or how you use the library. However, all three are known to have scalability and efficiency issues. 

• Apache Spark is used as an intermediate processing tool that can perform actions on those extracted data. Apache Spark allows for running the operations within the cluster via its API.
• Using Dask, which uses batching techniques, makes the job much easier and gives you one-click access to your data. Dask can handle multiple types of data in a single stream. The object passed into the Dask function is an RDD. This RDD holds all the data and the processing rules to be applied. Dask takes this and presents it to you in Python so that you can use it however you wish.
• Pandas cannot be utilized to create a scalable application. Pandas also has a popular data visualization library called ggplot built on top of matplotlib to provide beautiful data visualizations - which Dask doesn't have.

Now, let's compare and analyze Apache Spark, Dask, and pandas. This benchmark regularly runs against the most recent versions of these packages and is updated automatically: Data Source. 

• Basic Work

So, the following result appears when the data size is roughly 0.5 GB, and the code is given a basic task. Note: The result might not be the same for you, as your datasets or size might be different. 

When the basic task was repeated twice, Dask and pandas produced nearly identical results, while Apache Spark produced a difference of about 2s. You can see in the graph below that Apache Spark takes an average of 13 seconds, whereas Dask and pandas take 11 and 7 seconds, respectively. As a result, pandas takes the lead when dealing with simple tasks with small data sets.

‍

If you change the data size for the same basic task, the outcome may differ. So let's say the size is around 5GB. As seen in the figure below, when dealing with large amounts of data, it's clear that Apache Spark takes less time, whereas Dask takes roughly 170 seconds.

‍

• Advance Work

Let's look at what happens when the task is executed at the advanced level. Both Apache Spark and Dask fall behind when the work is advanced, and pandas takes roughly 84 seconds. So for heavy/advanced work, pandas is the clear winner. 

‍

For most tasks, all three libraries show similar performance characteristics. Still, there are significant differences in speed and memory usage depending upon problem size, input type, and whether the task needs full or split computations.

Real World Use Cases

• Apache Spark is used in various companies like Tencent, Riot Games, Trip Advisor, Alibaba, etc. It assists banks in automating analyses through machine learning. It is also used in the gaming industry to detect trends in real-time in-game activities and assist developers in providing in-game monitoring, user retention, comprehensive insights, and other services.
• Dask, like Apache Spark, is used in a wide range of industries, from retail to banking. For example, Walmart uses Dask to anticipate demand for a million different store items. Their objective is to ensure that a popular item is accessible in adequate quantities across all locations. Barclays uses it to model financial systems, commonly known as credit risk modeling. The emphasis here is on complicated systems rather than huge data. Apache Spark demands consistent data, but Dask thrives in volatility, making it the perfect choice for this application.
• Pandas are widely used in various tasks, from small projects to large applications. Many advertising companies use pandas to determine exactly what the consumer wants; it helps them learn more about customers and products. Companies like Instacart, Trivago, and Tokopedia use pandas to understand their customers. Pandas is mostly used in the analysis field.
  
  

Cost

So, why are we bringing up the topic of cost? Aren't these libraries free to use? Well, all the libraries are free to use on your machine. However, the entire cost of ownership must be considered, including maintenance, hardware and software purchases, and the recruitment of a cluster management staff. You can use your local machine and execute tasks on it, but cost and maintenance may be an issue. The best solution is to install the software with a provider like Apache Spark for DataBricks or run processes on the cloud with AWS.

The cost may vary based on how many clusters you use and what sort of application you have. For AWS, the hourly rate is around $0.102 for four vCPU and 8 GB of memory. Simply multiply the cost by the number of clusters you utilize to get an hourly cost. So your total cost can be calculated like :

Instance Type (No. Of cluster/CPU etc.) * Hourly price (AWS) * Time taken (For task) = Cost

Check - AWS On-Demand Pricing

Just tell me which one to try

It depends entirely on the type of application you have, but you may get a rough idea in this section. You can choose :

Dask

• If your use case is complicated or does not match the Apache Spark or pandas architecture well
• If you prefer Python Code, or you don't want to rebuild things from scratch
• Installing several packages isn't a problem for you
• If you like parallel computing on top of the existing NumPy, pandas, and other ecosystems

Recommended Reading: Why Do Data Scientists Love Dask?

Apache Spark

• If Scala or SQL is your preferred programming language
• If you're in the business of doing light machine learning work, Apache Spark might be the right option for you
• You want a business solution that is well-known and trustworthy
• You've mostly Legacy systems and JVM infrastructure

pandas

• If you need to work with a large amount of data, both pandas and Apache Spark may be a good fit
• If you like to create custom functionality, pandas comes with a lot of customization possibilities
• If you want fast execution for your complex work
  

Learn how to Improve AI outcomes

Quick comparison

Final points

• This code allows you to compare APIs and do benchmarks on your own
• Performance depends on your use case; if you redo a task, you may obtain a different result, and there is no clear winner in terms of performance
• Apache Spark offers a library and high-level API support, which is best for NLP and Computer vision applications
• Pandas comes with Quick Execution, which implies the task is done right away. While Apache Spark is Lazy Execution - a task will not be completed unless action is taken
• The majority of MLOps tools support all three libraries and are easy to integrate

Here are some other alternatives for you to consider:

• Vaex - Vaex is a Python module for visualizing and exploring large tabular datasets using lazy Out-of-Core DataFrames (similar to pandas)
• Ray - Ray is an open-source project that enables scaling any compute-intensive Python job — from deep learning to production model serving — laughably simple
• Datatable - A Python library for handling tabular data structures in two dimensions
• Modin - Change a single line of code to speed up your pandas operations
• cuDF (RapidAI) - Provides a pandas-like API that data engineers and data scientists will be acquainted with, allowing them to rapidly accelerate their workflows without learning CUDA programming

Conclusion

Choosing a library to perform data analysis can be challenging. Apache Spark, pandas, and Dask provide unique features and learning opportunities. Apache Spark is a general-purpose cluster computing system while pandas lets you work with Python data frames, and Dask allows for programming in Python’s parallel, distributed environment. Each library has its benefits and drawbacks. You may try them out for yourself to see which one is suitable for the application. I hope you liked this post!