pandas udf on spark

In PySpark, Pandas UDFs (user-defined functions) allow you to apply custom Python code that utilizes the Pandas library to transform data in a Spark DataFrame. This can be particularly useful when you need to perform complex data manipulation that is difficult or impossible to express with Spark SQL or the built-in PySpark functions.

To create a Pandas UDF in PySpark, you first need to import the necessary libraries:

from pyspark.sql.functions import pandas_ud
from pyspark.sql.types import IntegerType
import pandas as pd
        

Next, define a Python function that takes a Pandas DataFrame as input and returns a Pandas DataFrame as output. This function should not reference any variables outside of its own scope.

def my_function(pdf: pd.DataFrame) -> pd.DataFrame
    # Your custom code here
    return pdf:
my_udf = pandas_udf(my_function, returnType=IntegerType())         

Then, you can use the pandas_udf decorator to create a PySpark UDF that applies your custom function to a DataFrame column:

This will apply my_udf to the value column of df, producing the following output:

from pyspark.sql.functions import co

df = spark.createDataFrame([(1,), (2,), (3,), (4,)], ["value"])
df.select(my_udf(col("value"))).show()

+-----
|value|
+-----+
|    1|
|    2|
|    3|
|    4|
+-----++