Partition By vs Bucket By: Which One Should You Choose?

Big data is a term that refers to the ever-increasing amounts of data being generated in today's world. As the amount of data grows, so does the need for efficient storage and processing of this data. In this blog, we will discuss two important procedures, partitioning and bucketing, which help address these challenges when working with big data in Apache Spark.

Partitioning in Spark

Partitioning is a way to split data into separate folders on disk based on one or multiple columns. This enables efficient parallelism and partition pruning in Spark. Partition pruning is a technique used to optimize queries by skipping reading parts of the data that are not required.

In Spark, partitioning is implemented by the .partitionBy() method of the DataFrameWriter class. To partition a dataset, you need to provide the method with one or multiple columns to partition by. The dataset is then written to disk split by the partitioning column, with each partition saved into a separate folder on disk. Each folder can maintain multiple files, with the amount of resulting files controlled by the setting spark.sql.shuffle.partitions.

Here is an example of how to partition a dataset in Spark:

from pyspark.sql import SparkSessio


# Create a SparkSession
spark = SparkSession.builder.appName("PartitioningExample").getOrCreate()


# Load a dataset
df = spark.read.format("csv").option("header", "true").load("path/to/dataset")


# Partition the dataset by the "date" column
df.write.partitionBy("date").format("parquet").save("path/to/partitioned/dataset")        

In the above example, we loaded a dataset and partitioned it by the "date" column using the .partitionBy() method. The resulting partitioned dataset is then saved as a parquet file in the specified directory.

Bucketing in Spark

Bucketing is a way to assign rows of a dataset to specific buckets and collocate them on disk. This enables efficient wide transformations in Spark, as the data is already collocated in the executors correctly. Wide transformations are operations that require shuffling data across partitions, which can be a costly operation.

In Spark, bucketing is implemented by the .bucketBy() method of the DataFrameWriter class. To bucket a dataset, you need to provide the method with the number of buckets you want to create and the column to bucket by. The bucket number for a given row is assigned by calculating a hash on the bucket column and performing modulo by the number of desired buckets operation on the resulting hash.

Here is an example of how to bucket a dataset in Spark:

from pyspark.sql import SparkSessio


# Create a SparkSession
spark = SparkSession.builder.appName("BucketingExample").getOrCreate()


# Load a dataset
df = spark.read.format("csv").option("header", "true").load("path/to/dataset")


# Bucket the dataset by the "id" column into 10 buckets
df.write.bucketBy(10, "id").sortBy("id").format("parquet").save("path/to/bucketed/dataset")
        

In the above example, we loaded a dataset and bucketed it by the "id" column into 10 buckets using the .bucketBy() method. The resulting bucketed dataset is then sorted by the "id" column and saved as a parquet file in the specified directory.

When to use partitioning and bucketing?

If you will often perform filtering on a given column and it is of low cardinality, partition on that column. If you will be performing complex operations like joins, groupBys, and windowing and the column is of high cardinality, consider bucketing on that column.

However, bucketing is complicated and requires careful consideration of nuances and caveats. For example, there are conditions that need to be met between two datasets in order for bucketing to have the desired effect. Additionally, bucketing can only be used when the data is saved as a table, as the metadata of the buckets needs to be saved somewhere, usually in a Hive metadata store.

Conclusion

When working with big data in Spark, it is important to consider how the data is stored both on disk and in memory. Partitioning and bucketing are two procedures that can help optimize the storage and processing of large datasets. Partitioning enables efficient parallelism and partition pruning, while bucketing enables efficient wide transformations. However, bucketing is a complicated procedure that requires careful consideration of the nuances and caveats involved.

In summary, partitioning and bucketing are important tools to have in your big data arsenal when working with Spark. By using these techniques, you can optimize the storage and processing of large datasets, making your data processing pipelines faster and more efficient.