Apache Sedona

The Apache Sedona community continues to grow!

You can use Apache Sedona to easily calculate the distance between two points in an Apache Spark DataFrame. Just use the Sedona ST_DistanceSphere function to compute the meters between two latitude/longitude points. See the following example that computes the distance between New York/Sao Paulo and Sao Paulo/Rio.

It's easy to use Delta Lake with Apache Sedona for geospatial data. Sedona adds?powerful spatial functions like range queries and spatial joins to Apache Spark. The project used to be called GeoSpark until it became an official Apache project in 2020. You can use Sedona to process geospatial data using familiar Spark dataframes and store your geospatial data in Delta tables. Delta Lake gives you advanced data skipping and clustering features compared to common geospatial formats like GeoParquet. Delta Lake's Liquid clustering stores spatially related data close together for faster querying. Using Sedona with Delta Lake requires some extra Spark configs. Drop a note below if you'd like a notebook with config code.

?? Happy Friday! In case you missed it, here’s the recording of the November Apache Sedona Office Hour. Thanks to everyone who joined us—we look forward to seeing you at our next office hour! https://lnkd.in/gW_6PsrK

Just a friendly reminder that our next Apache Sedona Office Hour is happening tomorrow! We'll be covering exciting new features in Sedona 1.7.0, including K-Nearest Neighbor Join, GeoStatistics, and more. ?? ? Register here: https://bit.ly/4hzBpOy

?? We’re thrilled to announce the release of k-Nearest Neighbors (kNN) Join in WherobotsDB and Apache Sedona 1.7.0! This powerful feature helps you efficiently locate the nearest entities in large datasets. Choose between: ?? Exact kNN Join – Ideal for precise applications like geocoding and real-time transit mapping. ?? Approximate kNN Join – Optimized for high-speed processing where accuracy can be slightly flexible, such as meeting production requirements and image search. Whether you’re building complex geospatial applications or need fast data processing on the fly, the kNN Join capability in WherobotsDB and Apache Sedona 1.7.0 opens up new possibilities for scalable, high-performance data workflows. Join us on this journey to make data processing more powerful and accessible—no matter the size or scope of your datasets. ???Explore how kNN Join can elevate your geospatial and big data workflows! https://bit.ly/4ebxZP6 ???Sign up to get started with Wherobots for free: https://bit.ly/3NUyiDb #GeospatialData #BigData #DataScience #ApacheSedona

Hey Apache Sedona community!??? Join us for the upcoming office hour as we’ll cover exciting new features in Sedona 1.7.0. ? New spatial join type: K Nearest Neighbor join ? New data readers: DataFrame-based Shapefile reader and DataFrame-based GeoPackage reader ? New module: GeoStatistics → DBSCAN clustering, outlier detection and Getis-ord hot spot analysis Join us on November 7th at 8:00AM PT. Register here: https://bit.ly/3UBmxFY

?? The updated Geospatial Data Stack for late 2024. Tons of updates, new categories, and more. Full newsletter coming - sign up in the comments! ?? Processing is now a new category for large scale data processing which includes major data warehouses and Apache Sedona and Wherobots. ??? Storage is now focused on cloud blob storage from Google Cloud Storage, Amazon Web Services (AWS) S3, Microsoft Azure Blob Storage, Cloudflare R2, and DigitalOcean Spaces. ?? Catalogs come along with storage to complete the data lake vision with Apache Iceberg and Delta Lake along with Earthmover and their Arraylake toolkit for scientific data. ?? OLAP and OTAP (analytical and transactional) processing are now distinct categories with DuckDB and MotherDuck leading the way for OLAP and PostGIS and Crunchy Data for OTAP. More and more these two are linking up as well! ??Transform now has it's own category for the T in ETL with BigGeo making their first appearance alongside GDAL, H3, Airbyte, and dbt Labs. ?? Similarly Orchestration is now a distinct category for pipeline management with Apache Airflow and Astronomer, Dagster Labs, Prefect, and Kestra. ?? Of course all your favorite Cloud-Native Geospatial Forum (CNG) file formats too! ?? Analytics continues to grow with CARTO and Fused along with first timers Aino and xyzt.ai. ??? GIS continues to grow with Felt, QGIS, Kepler GL, and Atlas ?? The Python ecosystem has never been stronger with classic tools such as Rasterio and Geopandas, lonboard from Development Seed, and geemap and leafmap by Qiusheng Wu ?? Finally the application development ecosystem is booming with PM Tiles from Protomaps, MapLibre, Deck GL, and new addition Geobase #gis #moderngis #geospatialdata #spatialsql #earthobservation #moderndatastack #spatialanalytics #gischat

So cool to see Apache Sedona number of downloads surpasses 2 million downloads per month. Early 2018 when we released the first stable version of Sedona, we thought maybe 5 or 10 people would download it and use it. Boy, we were wrong. Apache Sedona is getting more ubiquitous everyday and being used for a plethora of use cases that rely on building geospatial data processing and ETL pipelines! Everywhere I go, I bump into a Sedona user. And, this is just the beginning - stay tuned for what Wherobots is cooking ?? Apache Sedona GitHub repo: https://lnkd.in/gMfRdrZd