Apache Iceberg and Parquet now support 'Geo' formats! This is an exciting development for the geospatial sector, and it particularly highlights the continued innovation in cloud-native table formats, especially Iceberg. These open standards have been a game changer in the data ecosystem, breaking down the walled gardens created by major platform providers.
The data ecosystem has long suffered from a fundamental disconnect: while software engineering practices have matured with robust version control, dependency management, and collaboration tools, data management remains fragmented and platform-dependent. We'd never fork a codebase without proper version control, yet duplicating datasets across systems is standard practice. While dependency management is streamlined in software development, tracking lineage across datasets built through complex data pipelines remains a persistent challenge.
Major platform providers have capitalized on these gaps, creating walled gardens where advanced features like lineage tracking, access controls, and performance optimizations are tied exclusively to their infrastructure. The data marketplaces hosted by these platforms also favor consumers and the platform itself, with little innovation benefiting data owners. These owners lack the telemetry needed to implement alternative revenue models—there's no direct equivalent of API metering in the data world—beyond flat pricing.
Cloud-native formats are changing this paradigm.?By storing data in an open format, you can seamlessly use different compute engines—Spark, DuckDB, Flint, Trino, Dremio, or others—without loading data into proprietary storage. This approach eliminates vendor lock-in, removes data duplication, preserves a single source of truth, and enables selecting the ideal execution engine for each workload, delivering both high performance and genuine portability.
This transformation is particularly significant for small and medium-sized companies, allowing them to choose from a broader set of platform-agnostic tools. At Foursquare, we have innovated with organizations like Astronomer, Acryl Data, and Privacera to create an interoperable data ecosystem unconstrained by the technical limitations or business interests of major platform providers. We are hoping to bring some of these innovations to the broader geospatial sector very soon.
Special mention to Chris Holmes, Jia Yu and Jed Sundwall for all their efforts in creating an open geospatial ecosystem.
