Pinpoint Spark Jobs to Optimize Using the Spark UI

• 举报此文章

Sai Prasad Padhy

Sai Prasad Padhy

Senior Big Data Engineer | Azure Data Engineer | Hadoop | PySpark | ADF | SQL

发布日期: 2025年1月20日

+ 关注

When working with Spark, knowing which jobs to optimize can save a lot of time and resources. The Spark UI is a powerful tool to help you identify the jobs that need attention.

We can do this by adding Logging to Spark jobs

Here is how you can do this:

Spark UI looks like below:

要查看或添加评论，请登录

Sai Prasad Padhy的更多文章

• Why Are RDDs Immutable?

  2025年1月16日

  Why Are RDDs Immutable?

  RDDs are immutable - they cannot be changed once created and distributed across the cluster's memory. But why is…

  7
• Automating Databricks Infrastructure Provisioning with Databricks API

  2024年12月6日

  Automating Databricks Infrastructure Provisioning with Databricks API

  Databricks, a powerful platform for big data and AI, offers APIs to automate infrastructure provisioning, saving time…

  3
• Global view vs Temp view in the context of Databricks

  2024年11月23日

  Global view vs Temp view in the context of Databricks

  Temp View Available within the context of a single notebook. Can be used to share data across different language REPLs…

  4
• What is Parquet file format & Why it is special?

  2024年2月19日

  What is Parquet file format & Why it is special?

  Parquet known for its efficiency and performance benefits. In this article, we'll discuss about internals of Parquet…

  20
• Spark Deployment Modes: Client Mode vs Cluster Mode

  2024年2月13日

  Spark Deployment Modes: Client Mode vs Cluster Mode

  Apache Spark provides powerful distributed computing capabilities for processing large-scale data. When deploying Spark…

  23

  3 条评论
• Adaptive Query Execution & Power of EXPLAIN command in Spark

  2024年1月9日

  Adaptive Query Execution & Power of EXPLAIN command in Spark

  Optimization is the key to unlocking the true potential of Apache Spark. EXPLAIN is one of the tools that is available…

  9
• Cache vs Persist in Apache Spark

  2024年1月4日

  Cache vs Persist in Apache Spark

  In this article, we'll go through Cache and Persist in Spark and understand the significance of using these techniques…

  8
• Understanding Repartition, Coalesce, and Making the Right Choice in Spark

  2023年12月26日

  Understanding Repartition, Coalesce, and Making the Right Choice in Spark

  Apache Spark excels in processing massive datasets by breaking them into manageable chunks called partitions. In this…

  16
• Apache Spark Transformations and Actions

  2023年12月22日

  Apache Spark Transformations and Actions

  In this detailed guide, we'll explore Transformations and Actions in detail, breaking down the complexities and…

  18
• Apache Spark Architecture: A Comprehensive Guide

  2023年12月19日

  Apache Spark Architecture: A Comprehensive Guide

  Apache Spark has emerged as a powerful and versatile open-source, distributed computing system, revolutionizing the way…

  10

  1 条评论

Show more

See all articles