RDD vs Dataframe vs Dataset

Before delving deep into the differences among the three mentioned above, let's take a look at what these are.

All of them are data abstraction APIs provided by Apache Spark for data processing and analytics. In terms of functionality, all are the same and provide the same output for any given input.

They differ in terms of handling and processing data. They vary in performance, user convenience, and language support.

Users can choose to work with any API while working with Spark.

1) RDD -

RDD stands for Resilient Distributed Dataset. An RDD is an immutable distributed collection of datasets partitioned across a set of nodes of the cluster that can be recovered if a partition is lost, thus providing fault tolerance. RDDs are Spark's fundamental data structure and provide a high-level API for performing distributed data processing tasks.

• Resilient - RDDs are immutable, partitioned collections of records that can be recovered if a partition is lost.
• Distributed - RDDs are a static set of items distributed across clusters to allow parallel processing.
• In-built memory computing - RDDs provide in-built memory computing and reference datasets stored in external storage systems.

RDD provides an OOP-style API, and here we tell the Spark engine "How to do" basically, how to achieve any particular task. And, since here we tell the Spark engine how to achieve a task, optimization is in our hands

2) Dataframes -

DataFrames are distributed collections of data organized into rows and columns. The concept of DataFrames remains similar across all programming languages, but Spark DataFrames differ in functionality compared to Pandas. They are conceptually equivalent to a table in a relational database or a data frame in R/Python, but with richer optimizations under the hood.

Since RDDs provide OOP-style programming, which can be somewhat challenging to work with, the DataFrames API was created to enable a broader audience to work with Spark.

As an extension to the existing RDD API, DataFrames feature:

• The ability to scale from kilobytes of data on a single laptop to petabytes on a large cluster
• Support for a wide array of data formats and storage systems
• Seamless integration with all big data tooling and infrastructure via Spark
• APIs for Python, Java, Scala, and R (in development via SparkR)

DataFrames provides SQL style API and here, we tell spark engine "What to do" and, spark engine will use optimization through the Spark SQL Catalyst optimizer to achieve the cost-effective way to accomplish the task.

3) Datasets:

These are the newest data abstraction provided by spark. It is the combination of best of dataframes and best of datasets. It possess best of RDDs - OOP style + type saftety and best of Dataframes - structured format + optimization + memory management.

Similiarity among all:

• Fault tolerant
• Distibuted
• In-memory paraller processing
• Immutable
• Lazy Evaluation
• Internally processed as RDD API

Differences:

1. Both RDDs and Datasets provide an OOP-style API, while DataFrames provide a SQL-style API.
2. In RDDs, we specify to the Spark engine how to achieve a certain task, whereas with DataFrames and Datasets, we specify what to do, and the Spark Engine takes care of the rest. This is why DataFrames and Datasets inherently have optimization techniques.
3. In RDDs, only on-heap objects are used, while in DataFrames and Datasets, both on-heap and off-heap memory can be utilized. Off-heap objects are employed when there is additional data in memory.
4. Since RDDs use only on-heap objects, serialization is unavoidable because additional data needs to be transferred from RAM to disk. This is avoidable in DataFrames and Datasets due to the presence of off-heap space.
5. In RDDs, *garbage collection (GC) impacts performance, but in DataFrames and Datasets, GC impact is resolved.

*GC - Garbage Collector: When memory is full in RDD, GC will start scanning entire memory and it will start removing the data which is old and obselete.

6) RDD and Datasets provide strong type safety that is at the time of you writing the code it'll give the error if something is wrong and thus they provide run-time compilation error. But, in the case of, DataFrames there's no type safety, so error will be known only once the code is executed and thus, they provides error at compile team.

In summary, Apache Spark's trio of data abstraction APIs—RDDs, DataFrames, and Datasets—offers a flexible framework for distributed data processing. While sharing common traits like fault tolerance and in-memory parallel processing, they diverge in API styles, optimization strategies, and memory management. RDDs, with their OOP-style API, enable users to explicitly guide the Spark engine, whereas DataFrames, featuring a SQL-style API, focus on user-friendly interactions and seamless language integration. Datasets combine the strengths of RDDs and DataFrames, incorporating OOP style, type safety, and efficient memory handling. The choice among these abstractions hinges on specific task requirements, programming preferences, and optimization needs, highlighting Spark's adaptability in catering to diverse data engineering and analytics scenarios

#spark #rdd #dataframe #dataset #dataengineering #apachespark