Polars vs pandas

Polars is a fast DataFrame library, similar to Pandas, but optimized for performance, especially when working with large datasets. Polars supports multi-threading and execution across multiple cores, making it an attractive choice for handling large datasets efficiently.

While both Polars, Pandas, and DuckDB are used interchangeably when working with large datasets, DuckDB is primarily designed for SQL-like operations.

Polars outperforms Pandas due to its Rust-based architecture. Unlike Java or other languages that rely on garbage collection, Rust does not use garbage collection, which helps avoid the performance pitfalls seen in other languages over the past decade.

Rust, developed by Mozilla in 2010, is designed for three core purposes in programming:

Performance

Safety

Memory management

Rust is used to develop advanced applications such as gaming engines, operating systems, and browsers, all of which require scalability. Rust shares similarities with C++, but it provides memory safety without relying on garbage collection. The language aims to deliver higher performance and better safety than C++.

import polars as pl

import duckdb

import pandas as pd

For the comparison, I used airline data stored in a CSV file (1996.csv) with 5,351,983 rows and a disk size of 540 MB. The comparison involved measuring execution time, CPU usage, and RAM consumption. The results clearly show a significant performance improvement with Polars, as seen in the accompanying image.

Airlines data- https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/HG7NV7

System configuration- 16GB RAM,5core ,Ubantu24.04,SSD HD

Feel free to contact me at cnsnoida@gmail.com.

Thanks for reading!