Apache Spark

Apache Spark is an in-memory data processing framework designed for large-scale distributed data processing. Known for its speed, it significantly outperforms traditional Hadoop MapReduce, making it a popular choice for big data processing.

Key Libraries and Components

• #MLlib :

Description: MLlib is Spark's scalable machine learning library.

Key Features: It includes common algorithms and utilities like classification, regression, clustering, collaborative filtering, dimensionality reduction, and underlying optimization primitives.

• #Spark SQL

Description: Spark SQL is a module for working with structured data.

Key Features: It allows querying data using SQL as well as the Apache Hive variant of SQL called HQL. It also supports various data sources like Parquet, JSON, and ORC.

• #Structured Streaming

Description: Structured Streaming is a scalable and fault-tolerant stream processing engine built on the Spark SQL engine.

Key Features: It allows you to process and analyze streaming data in real-time. It provides a high-level API for stream processing and integrates seamlessly with batch processing.

• #GraphX

Description: GraphX is Spark's API for graphs and graph-parallel computation.

Key Features: It simplifies the process of graph processing and provides a set of graph algorithms (e.g., PageRank) and utilities to manipulate graphs.

Apache Spark Characteristics

Speed

• Spark processes data in-memory, which significantly enhances its speed compared to traditional disk-based systems like Hadoop MapReduce.
• This makes it ideal for handling large-scale data processing efficiently.

Ease of Use

• Spark provides high-level APIs in Java, Scala, Python, and R.
• It also includes an interactive shell for exploring data, making it accessible and user-friendly for a wide range of developers and data scientists.

Modularity

• Spark's architecture is modular, meaning it consists of various components that can work independently or together.
• This modularity allows for flexibility in handling different types of data processing tasks, from batch processing to stream processing and machine learning.

Extensibility

• Spark is designed to be highly extensible.
• Its open-source nature and active community contribute to a wide range of extensions and integrations with other tools and frameworks.

• Developers can easily add new functionalities or customize existing ones to suit their specific needs.

Key Components of Apache Spark

Spark SQL and DataFrames/Datasets

• Spark SQL is a module for structured data processing.
• DataFrames and Datasets are the primary APIs to work with structured data.
• DataFrames are distributed collections of data organized into named columns, similar to tables in a relational database.
• Datasets, introduced in Spark 1.6, are a type-safe, object-oriented API for working with structured data.
• Both APIs provide optimized query execution through the Catalyst query optimizer and Tungsten execution engine.
• Spark SQL can run SQL queries, provide SQL-like functions, and work with various data sources like Hive tables, Parquet, JSON, and JDBC.

Spark Streaming (Structured Streaming)

• Structured Streaming is Spark’s scalable and fault-tolerant stream processing engine.
• It builds on Spark SQL and allows for real-time data processing with high-level APIs similar to batch processing.
• Users can query streaming data the same way they query static data, and Spark ensures fault-tolerance and exactly-once processing.
• Structured Streaming seamlessly integrates with Spark’s batch processing, allowing for continuous, end-to-end data processing pipelines.
• It supports various data sources and sinks, including Kafka, Kinesis, HDFS, and more.

Machine Learning (MLlib)

• MLlib is Spark’s machine learning library, providing scalable and easy-to-use implementations of common algorithms and utilities.
• It includes algorithms for classification, regression, clustering, collaborative filtering, and dimensionality reduction.
• MLlib also offers feature extraction, transformation, and selection, as well as underlying optimization primitives.
• The library supports both DataFrames and RDD-based APIs, making it versatile for various machine learning workflows. With MLlib, users can build and train machine learning models at scale, leveraging Spark’s distributed computing capabilities.

Graph Processing (GraphX)

• GraphX is Spark’s API for graphs and graph-parallel computation. It combines the advantages of both graph processing frameworks and data-parallel frameworks, allowing users to analyze graphs alongside other Spark workloads.
• GraphX introduces the concept of Resilient Distributed Property Graphs (RDG), which are directed multigraphs with user-defined properties attached to each vertex and edge.
• It provides a collection of graph algorithms, including PageRank, Connected Components, and Triangle Counting.
• GraphX simplifies graph processing tasks and enables iterative graph computations using Spark’s fault-tolerant, in-memory computation engine.

Supported Programming Languages

#Scala

Scala is Spark’s native language, offering concise syntax and functional programming capabilities. It allows developers to write efficient, expressive, and type-safe code. Spark APIs are first designed in Scala, making it the most feature-complete language for Spark development.

#SQL

Spark SQL allows users to query structured data using SQL syntax. It supports a subset of the ANSI SQL standard and integrates with Spark’s Catalyst optimizer, providing efficient query execution. Spark SQL can be used interactively in the Spark shell or through programmatic APIs.

Python

PySpark is the Python API for Spark, enabling developers to write Spark applications using Python. PySpark provides bindings for Spark’s core functionalities, including Spark SQL, DataFrames, Datasets, and MLlib. It’s popular among data scientists for its simplicity and integration with libraries like pandas and numpy.

Java

Spark provides Java APIs for its core functionalities, allowing developers to build Spark applications in Java. While the Java API is slightly more verbose than Scala or Python, it offers type safety and seamless integration with existing Java codebases.

R

SparkR is the R API for Spark, designed for data scientists who prefer using R for statistical analysis and data visualization. SparkR provides bindings for Spark’s DataFrame and MLlib functionalities, enabling scalable data processing and machine learning in R.

Apache Spark Installation on Windows

Setup Apache Spark (PySpark) on Windows PC where we have installed JDK, Python, Hadoop and Apache Spark. Please find the below installation links/steps:

PySpark installation steps on MAC: https://sparkbyexamples.com/pyspark/h...

1. Download JDK: https://www.oracle.com/in/java/techno...

2. Download Python: https://www.python.org/downloads/

3. Download Spark: https://spark.apache.org/downloads.html

Winutils repo link: https://github.com/steveloughran/winu...

Environment Variables: HADOOP_HOME- C:\hadoop JAVA_HOME- C:\java\jdk SPARK_HOME- C:\spark\spark-3.3.1-bin-hadoop2 PYTHONPATH- %SPARK_HOME%\python;%SPARK_HOME%\python\lib\py4j-0.10.9-src;%PYTHONPATH%

Required Paths: %SPARK_HOME%\bin %HADOOP_HOME%\bin %JAVA_HOME%\bin