Iceberg’s Icy Ascent: How Apache Iceberg Became the Table Format of the Future

For years, the data engineering world wrestled with a critical question: which open table format would dominate the future? Would Delta Lake’s seamless Databricks integration prevail? Could Apache Hudi, with its streaming-first ethos, maintain an edge? Or would Apache Iceberg, quietly innovative, emerge victorious?

As of late 2024, the answer is clear: Apache Iceberg has won.

A Turning Point in 2024

Several key developments in 2024 sealed Iceberg’s position:

• Databricks acquired Tabular, founded by Iceberg’s creators, solidifying Iceberg’s stature in the data ecosystem.
• Snowflake unveiled Polaris, an Iceberg-based catalog, with support from major query engines like Starburst and Dremio.
• GitHub activity showed Iceberg gaining significant traction, underscoring its growing developer base and widespread adoption.

Together, these moves signaled a decisive industry shift toward Iceberg as the de facto open table format. But Iceberg’s story is far from over. The upcoming advancements in 2025 promise to cement its dominance and expand its utility across diverse data workflows.

What’s Next for Iceberg in 2025?

1. RBAC Catalog: Simplifying Permissions at Scale

Data lake permissions have long been a challenge, often cobbled together with bucket-level rules or engine-specific controls. These fragmented methods are inefficient and prone to security gaps.

Iceberg’s new OpenAPI specification (PR #10722) changes the game. By standardizing credential structures, Iceberg introduces built-in Role-Based Access Control (RBAC) capabilities at the catalog level.

• What it enables: Administrators can define fine-grained access policies that are independent of the underlying storage or query engine.
• Why it matters: This rivals enterprise-grade solutions like Databricks Unity Catalog but maintains Iceberg’s hallmark openness and flexibility.

2. Change Data Capture (CDC): Iceberg’s Streaming Evolution

Historically, Iceberg wasn’t considered ideal for streaming due to limited CDC capabilities. While versioned table snapshots supported some CDC use cases, high-frequency data changes and real-time analytics were less efficient.

Enter Iceberg Spec V3, featuring Row Lineage.

• What’s new: Row Lineage tracks individual row changes — updates, deletes, and inserts — enabling efficient CDC pipelines.
• Why it matters: Materialized view maintenance and real-time data synchronization become far more seamless. Once fully implemented, Iceberg will rival streaming-first platforms like Kafka and Hudi for real-time applications.

3. Materialized Views: Streamlining Derived Data

Derived datasets — aggregations, metrics, and other transformations — are critical for unlocking data value but have been cumbersome to manage with Iceberg.

A proposed materialized views feature (PR #11041) introduces built-in support for precomputed results stored as tables.

• What it enables: Faster query performance and automatic updates when the base table changes.
• Why it matters: It simplifies dependency tracking and reduces the overhead of managing derived data, opening up opportunities for systems like RisingWave to enhance the experience further.

Beyond Features: Iceberg’s Ecosystem Growth

As Iceberg’s capabilities evolve, so does its ecosystem. Highlights to watch in 2025 include:

• Support for nanosecond-precision timestamps: Critical for industries like finance and telecoms that demand high-precision data.
• Binary deletion vectors: Part of Spec V3, offering scalable and efficient deletion handling for regulatory compliance and GDPR requirements.
• Expanded engine compatibility: Iceberg already integrates with Kafka, PostgreSQL (via RisingWave), and query engines like Trino, Databricks, and Snowflake.

The One Missing Piece: Lightweight Compaction

Iceberg excels in many areas, but compaction remains a bottleneck, typically relying on resource-intensive Spark jobs.

This limits adoption for smaller teams and SQL/Python-centric users who need simpler, more resource-efficient options. Fortunately, the community recognizes this gap, and momentum is building for a lightweight, engine-agnostic compaction framework.

The Road Ahead

With innovations like RBAC catalogs, advanced streaming capabilities, materialized views, and new data type support, Apache Iceberg is on track to become the universal table format for modern data engineering.

2024 marked Iceberg’s victory in the format wars. 2025 will be about making it more accessible, versatile, and powerful for users of all sizes — from startups to global enterprises. Whether you’re managing historical data, building real-time pipelines, or exploring cutting-edge lakehouse designs, Iceberg offers something for everyone.