Easily configure shared memory size for CLI jobs with `--docker-shm-size`. Training PyTorch models on a GPU and need more memory? Ever run into "Error: No space left on device"? Coiled run supports `--docker-shm-size` so you can customize Docker shared memory size. More details on Coiled for CLI jobs in the docs: https://lnkd.in/gWH6Hzx7
关于我们
Python, but big. Churn through a ton of data, no cloud expertise needed.
- 网站
-
https://coiled.io
Coiled的外部链接
- 所属行业
- 软件开发
- 规模
- 11-50 人
- 总部
- New York，New York
- 类型
- 私人持股
- 创立
- 2020
地点
-
主要
获取路线
US，New York，New York，10018
Coiled员工
动态
-
Coiled转发了
The scale and impact of NOAA’s public data is widely underestimated. Since joining Coiled, I’ve had the chance to work with customers tackling large-scale geospatial challenges, seeing firsthand how deeply public-sector data underpins industries that many assume operate independently. In just the past year, I’ve seen NOAA data drive critical decisions in industries like: - Transportation: Airlines, shipping, and logistics depend on NOAA forecasts. - Utilities: Power companies assess fire risk and plan transmission routes. - Insurance & Risk Assessment: Climate models determine flood zones and shape home insurance rates. - Water & Agriculture: Farmers and municipalities track drought conditions and manage water resources. - Energy: Renewable and fossil fuel industries use remote sensing to optimize site selection. - Disaster Response & National Security: Emergency agencies depend on NOAA data for hurricane, wildfire, and extreme weather monitoring. - Environmental Conservation: Researchers track biodiversity loss, carbon storage, and ecosystem changes. These aren’t niche applications. NOAA’s data supports public safety, economic stability, and environmental resilience. While private companies contribute to weather and climate services, there is no viable replacement for NOAA’s scale, quality, and accessibility. The NOAA cuts this week aren’t just a government spending issue, they have real consequences for businesses and everyday people. And once this capacity is lost, there’s no quick fix.
Here’s my take on the NOAA: National Oceanic & Atmospheric Administration situation... The general public, and even many sophisticated tech leaders and investors, truly underestimate the reach and impact of NOAA data and services. As a data platform company serving customers in weather, climate, and geospatial domains, Earthmover gets a front-row seat to what sort of data companies actually use. Guess what? Every single one of our private-sector customers has the majority of their data coming from public-sector data providers like NOAA, NASA - National Aeronautics and Space Administration, European Centre for Medium-Range Weather Forecasts - ECMWF, and Copernicus. Why? Not only because it’s free. It’s also the highest quality, most reliably maintained, most consistently useful data about the planet that exists. The reckless moves being made to downsize NOAA risk jeopardizing our nation’s leadership in weather and environmental resilience. While I certainly believe that the private sector has an important role to play in the weather enterprise, the private sector cannot and should not replace NOAA’s critical role. If these cuts are not blocked or rolled back, it could take decades for us to recover from the damage being done.
-
?? Job setup option for Coiled Batch. You can now define a job setup script with `--host-setup-script` to configure your VM before your batch job starts. This makes it dead-simple to: ? Install dependencies ? Mount a cloud storage bucket ? Handle authentication? or any other pre-job configuration that your jobs need. More on Coiled Batch in the docs: https://lnkd.in/gDEe3kHd
-
First rule of distributed computing is don't use it if you don't have to. If your dataset fits on a single VM, you don't need a cluster. We noticed many Coiled "clusters" were actually just single VMs, so we added serverless functions for exactly this use case. Great for when you just want to spin up a VM, process some raster data stored on S3, and then turn things off when you're done, all from a Python session. Thanks for the shoutout Matt Forrest ?? .
?? Helping geospatial professionals grow using technology · Scaling geospatial at Wherobots
? Cloud-native geospatial is for small data (too). For years, geospatial workflows have been stuck in a false dilemma: Desktop GIS for small data, enterprise servers for big data. But cloud-native geospatial has started to change that paradigm. Here's why. ?? Cloud storage = simple, scalable, cost-effective Cloud storage (like AWS S3, Google Cloud Storage, Azure Blob) have changed the game by making geospatial storage limitless and cheap. No need to manage physical servers, upgrade storage, or worry about running out of space. Just store your data once—no need to duplicate it across systems. ??? Store petabytes of satellite imagery in Cloud-Optimized GeoTIFF (COG) ?? Manage time series climate data with Zarr or NetCDF ?? Store nationwide vector datasets in GeoParquet for instant access No database imports, no file transfers, no infrastructure headaches. Compute comes to the data—NOT the other way around Traditionally, you’d move your data to where the compute is—importing it into a database, spinning up a server, waiting on ETL jobs. Not anymore. Now, computation happens directly on cloud storage using modern serverless, parallel, and distributed computing tools, including: ? Wherobots – Scalable spatial analytics on cloud-native storage ? Apache Sedona – Distributed geospatial processing on Apache Spark ? DuckDB – Lightning-fast local SQL on cloud-stored Parquet & COG ? BigQuery GIS, Snowflake, Redshift – Data warehouse queries without moving data ? Dask & Coiled – Parallelized geospatial analytics in Python ? Trino & Presto – Distributed querying across cloud storage All of these allow geospatial analytics at any scale—without ever “importing” data into a traditional database. No more infrastructure maintenance. No database tuning, indexing, or optimization. No data pipeline maintenance. No provisioning or scaling compute resources. Instead, everything is on-demand, auto-scaling, and pay-as-you-go. You only use the compute power you need—when you need it. Oh, and you can replicate a lot of this locally too ??.
-
-
Coiled转发了
Coiled makes a dead-simple way to spin up serverless infrastructure for data pipeilines, without the usual infra and docker headaches. It's been an absolute treat to collab with Matthew Rocklin and Nat Tabris to bring Coiled's infra to Prefect.
Prefect and Coiled now integrate serverless push work pools. Prefect hosts logic about when to run flow, and Coiled runs those flows on cloud hardware easily and efficiently. We hope this combination opens doors for lots more people to get their work done. https://lnkd.in/giUXsB6y
-
Latest updates in our January newsletter: - ???2024 in Review: Key milestones and what’s ahead. - ?? Prefect + Coiled: Run Prefect Push Work Pools on the cloud without managing infrastructure. - ?? Parallel GeoTIFF Processing with GDAL: Reproject thousands of satellite images in ~5 minutes. - ?? Zarr 3: Faster IO and better chunk handling. - ?? Xarray Efficiency: Continued improvements to Dask-backed computations.
-
Coiled转发了
Coiled 2024 in Review https://lnkd.in/gmug5ina It’s that time of year where companies issue year-end summaries, looking back on the past twelve months, acclaiming success (or not), and forecasting incredible growth for the next year (or not). I thought I’d do something similar for Coiled. It’s been a significant year for us ...
-
Coiled转发了
Another huge improvement in the #Dask Array world! Calculating Quantiles over time is a common application for #Geospatial world. Historically, this was pretty slow because of a lot of GIL contention on the NumPy level (I am really looking forward to a free threaded Python world ??). Our new implementations is up to a hundred times faster than the old one and more importantly scales independently of the number of threads ?? I wrote a short blogpost explaining how we arrived at that solution including a before / after comparison. https://lnkd.in/gt5A_pVy We are still interested in feedback about what isn't working well for you with #Xarray and #Dask. Please reach out if you have anything that bugs you!
-
-
We're big fans of?rich?for a nice terminal experience, but have found sometimes folks log things that even?rich?can't handle. As of the latest?coiled=1.67.0?release,?coiled logs automatically falls back to non-rich?printing in these situations. More details in the release notes: https://lnkd.in/gj6Ruekq
-