Parquet Architecture and Internals

Apache Parquet is a game-changer in the world of big data processing and analytics. Unlike traditional row-based formats like CSV or JSON, Parquet stores data in a columnar format. This means each column's values are stored together, enabling super-efficient compression, encoding, and data processing.

Why Parquet?

?? Efficient Storage: Columnar format means better compression and reduced storage costs.

?? Faster Queries: Only read the columns you need, speeding up query performance.

??Optimized for OLAP: Supports both projection (selecting columns) and predicates (row selection criteria), perfect for complex analytical workflows.

?

• We’ll delve into the intricate details of how Parquet files are structured, how metadata and data pages are organized, and why concepts like dictionary encoding and predicate pushdown play a crucial role in optimizing data storage and query performance.

Structure of Parquet Files:

• Parquet files are organized in a columnar storage format, which means that instead of storing data in rows like traditional databases, Parquet stores data in columns. This columnar structure offers significant advantages in terms of compression and query performance.
• Columns and Row Groups: Data in a Parquet file is divided into columns, and groups of columns are organized into “row groups.” Each row group contains a section of the data, and columns within a row group are stored together to optimize compression and minimize I/O operations.

?

?

• Block (HDFS block):?This means a block in HDFS and the meaning is unchanged for describing this file format. The file format is designed to work well on top of HDFS.
• File: A HDFS file that must include the metadata for the file. It does not need to actually contain the data.
• Row group:?A logical horizontal partitioning of the data into rows. There is no physical structure that is guaranteed for a row group. A row group consists of a column chunk for each column in the dataset.
• Column chunk: A chunk of the data for a particular column. They live in a particular row group and are guaranteed to be contiguous in the file.
• Page: Column chunks are divided up into pages. A page is conceptually an indivisible unit (in terms of compression and encoding). There can be multiple page types which are interleaved in a column chunk.

Metadata and Data Pages:

1.???? Parquet files contain metadata that describes the structure of the data and allows for efficient retrieval. There are three main types of metadata: file metadata, column(chunk) metadata and page header metadata.

?

• File Metadata: High-level information about the Parquet file, including version, schema, row groups, and footer, enabling efficient file navigation and data retrieval.
• Column (Chunk) Metadata: Column-specific details within a row group, such as encoding, statistics, data type, and compression, optimizing data storage and query performance.
• Page-header Metadata: Information within each data page, like size, dictionary references, encoding, and value count, facilitating efficient decoding and processing of columnar data during queries

?

?

1. Dictionary Encoding:

• Parquet creates a dictionary of the distinct values in the column, and afterward replaces “real” values with index values from the dictionary.?

?

• This technique significantly reduces the storage footprint by storing repetitive data only once in the dictionary. It also improves query performance as dictionary-encoded columns can be efficiently compressed and decompressed during query execution.

?

?2. Predicate Pushdown:

Predicate pushdown is a query optimization technique that filters data at the data source before it’s read into memory. In the context of Parquet files, predicate pushdown involves pushing down filtering conditions to the Parquet reader level, allowing it to skip irrelevant data during the reading process.

Refernce:

https://parquet.apache.org/docs/concepts/

?

?