Understanding Spark on YARN Architecture
Sachin D N ????
Data Consultant @ Lumen Technologies | Data Engineer | Big Data Engineer | AWS | Azure | Apache Spark | Databricks | Delta Lake | Agile | PySpark | Hadoop | Python | SQL | Hive | Data Lake | ADF | Data Warehousing | DLT
Apache Spark is a powerful, in-memory data processing engine with robust and expressive development APIs. It enables data workers to efficiently execute streaming, machine learning, or SQL workloads that require fast iterative access to datasets. Hadoop YARN (Yet Another Resource Negotiator), on the other hand, is a resource management platform responsible for managing compute resources in clusters and using them for scheduling users' applications.
When Spark runs on YARN, it provides the resource management, while Spark provides data processing tasks. This allows Spark to leverage YARN's powerful features for managing and scheduling resources across the cluster.
Spark on YARN Architecture
In a Spark on YARN setup, the architecture consists of the following components:
1. Client: This is the machine where the Spark job is initiated.
2. Resource Manager (RM): The Resource Manager is the master process in YARN, responsible for resource assignment and management among all the applications.
3. Node Manager (NM): The Node Manager is the per-machine agent who is responsible for launching the applications' containers, monitoring their resource usage (CPU, memory, disk, network), and reporting the same to the ResourceManager.
4. Application Master (AM): The Application Master negotiates resources from the Resource Manager and works with the Node Manager to execute and monitor the tasks. In the context of Spark on YARN, the Application Master is responsible for the execution of the Spark job.
Running Spark on YARN
When a Spark job is submitted from a client machine, the following steps occur:
1. The client contacts the Resource Manager to create an application, and the Resource Manager responds with an application ID.
2. The client then launches the Application Master in a container. The Application Master registers with the Resource Manager and requests resources.
领英推荐
3. The Resource Manager allocates containers for the Application Master, and the Application Master then communicates with the corresponding Node Managers to launch the containers.
4. Once the containers are launched, the Spark tasks run on these containers.
5. The Application Master continually requests resources from the Resource Manager to run all Spark tasks. It also monitors the task status and provides updates to the client.
6. When the job is done, the Application Master deregisters with the Resource Manager and shuts down, freeing up the resources for other tasks.
Spark on YARN Modes
Spark on YARN can run in two modes:
1. Cluster Mode: In this mode, the Spark driver runs inside the Application Master on a cluster host. This mode is suitable for production jobs as it allows the application to be completely run within the YARN cluster.
2. Client Mode: In this mode, the driver runs on the client machine. This mode is suitable for interactive and debugging workloads where you want to see your application's output immediately.
In conclusion, running Spark on YARN allows we to take advantage of the resource management capabilities of YARN while benefiting from the data processing power of Spark. This combination is particularly useful in a Hadoop ecosystem where we can run Spark alongside other YARN applications.