Apache Spark

Want to learn more about #pyspark #memory profiling? Have questions about PySpark #UDFs, profiling hot loops of a UDF, profile memory of a UDF? This #AMA is a follow-up to the popular post How to Profile PySpark https://lnkd.in/gQ2MzXG5 #apachespark

pandas on Spark makes it easy to read Excel files into Spark DataFrames. Just import pandas on Spark and read the files with read_excel(). Spark makes it easy to read data stored in a variety of data sources - even Excel files!

Spark makes it easy to collapse rows and collect values in a list. This is a great way to reduce the number of rows in a DataFrame without losing data. Don't underestimate the power of storing data in PySpark ArrayType columns!

PySpark has several array helper functions that make it easy to perform common array processing tasks. For example, array_union returns the distinct elements in two arrays. array_intersect returns the elements in both arrays.

Spark lets you accelerate count-distinct operations with the HyperLogLog algorithm. Computing the number of distinct items in a column can be an expensive operation because many rows of data must be scanned. The HyperLogLog algorithm stores the unique values of a column in a sketch, which allows for a quick approximate distinct count. The HyperLogLog isn't precise and returns an approximate result. It sacrifices some accuracy for much more speed. HyperLogLog sketches can be unioned with other sketches, so they're useful for incrementally updated pipelines in production data settings. PySpark 3.5+ supports HLL functions out of the box.

It's best to avoid Spark User Defined Functions (UDFs) when you can simply use Spark native functions. The following example shows a poorly defined UDF that blows up with NULL input. You always want to handle the NULL case in your UDF. The second snippet shows how to define a UDF that gracefully handles the NULL case, which is an improvement. However, the third snippet shows that you don't need a UDF in this case. It's easy to append to a string column using native Spark functions. Native Spark functions gracefully handle the NULL case and can be optimized. UDFs are a black box from the optimizer's perspective, so they're harder to optimize.

Spark makes it simple to snake_case all of the columns in a DataFrame. Just create a function that downcases the string, replaces spaces with underscores, and apply that function to each column. You could also run withColumn() in a loop, but it's more performant to run a single select(). PySpark allows for some really nice abstractions.

You can easily single space a string with PySpark using the built-in trim and regexp_replace functions. The following example shows a column with irregular spacing and a single_space function that cleans the whitespace. It's nice to wrap code in functions so it's readable and easier to unit test. PySpark allows for clean code!

It's straightforward to run PySpark in Jupyter notebooks. Start by creating and activating a virtual environment with PySpark installed. Then create a Spark Session and you can run any Spark code. For example, you can easily create a Parquet table and read the table contents. It's a good idea to set the log level to ERROR to limit the WARN level terminal outputs. You can even use the JupyterLab Desktop application to run the Spark code in a separate application!

Spark makes it easy to calculate the month start and month end dates. Use trunc() to compute the first day of the month. Use last_day() to compute the last day of the month. Spark makes it easy to work with date and datetime values.