The world’s first data collaboration project, Snowflake Summit reflections, and more

Last week, I was in Vegas for Snowflake Summit — at what feels like the first real data conference in years. And I have to admit, I was a little skeptical and a bit nervous. I asked myself, would practitioners actually show up at conferences after getting used to online events? Given the state of the economy, are data platforms really a priority for companies? How much value is there in in-person events anymore?

I couldn’t be happier to report that my skepticism was… well... squashed! I’ve been told that the Snowflake Summit drew over 10,000 attendees, about 80% of whom were data practitioners. Several conference sessions were fabulous, with attendees even having to be turned away from some sessions. The energy was intense. And most importantly, I had some amazing, deep, human conversations with data practitioners. After years of Zoom calls, it was a breath of fresh air.

The theme of Snowflake’s conference this year was “the world of data collaboration”. As the founder of a company whose mission statement is literally to help data teams collaborate better together, this clearly excited me. This week also marks three years of Atlan, and I’ve been a little nostalgic. When our team started thinking of the name of the company you today know as Atlan, we were looking for something that represented our vision for the world — a world where collaboration around data goes beyond the confines of a team or company and brings together organizations and communities around the world.

We took inspiration from the works of many amazing Humans of Data. One of them was Matthew Fontaine Maury, who in the 1800s figured out that data could change the way ships sailed the oceans. Our name, Atlan, is even a tribute to his first groundbreaking chart of the Atlantic: the first global data collaboration project when 32 countries came together to share shipping data from around the world.

The revolution that Maury sparked by dividing the Atlantic into 5-degree lat-long squares unified the world. They put aside their differences, shared data, and collaborated to make our oceans safer for everyone, and in the process made the world’s first operational data supply chain.

At Atlan, our vision is to create a world with a global data supply chain — where every data team collaborates to achieve collective goals and further human progress. We’re fortunate to be part of the revolution Maury sparked, and our name “Atlan” inspires us every day to keep moving this vision forward.

In this edition, we’ll deep-dive into the world’s first data collaboration project, and how Maury sparked a scientific revolution with data. ?

Happy reading!

???Spotlight:?How Maury Saved Millions and Unified the World with Data in the 1800s

We like to think that air travel today is long or even dangerous, but it’s nothing compared to sea travel a couple of centuries ago. Imagine getting on a massive boat with hundreds of passengers, and then waiting for 50 days. Or 100 days. Or even 150 days. Make it a return trip, and that’s most of the year gone.

And that’s just for boats that actually made it. If your unlucky boat was one of the?135 major shipwrecks?in the nineteenth century, you wouldn’t even reach your destination.

Suddenly in the mid-1800s, everything changed. One man, Matthew Fontaine Maury, figured out that data could change the way ships sailed the oceans. But this isn’t just his story. This is the story of Maury, the United States Navy, over a dozen other countries’ navies, and thousands of merchant ships working together. Oh yeah, and the Pope.

Their secret weapon? Data. Maury spearheaded a massive data collection project, with data in dozens of languages reported from ships around the world.

From ship routes to scientific investigations

In 1839, while Maury was traveling back to New York from a visit home to Tennessee, the carriage he was traveling in tipped over. Maury dislocated his knee and fractured his thigh bone.

His recovery was slow and painful. Though Maury was anxious to return to naval life, doctors decided that he was in no condition for life aboard a ship. Instead, in 1842, the Navy appointed him the Superintendent of the relatively new?Naval Observatory.

After a year of studying the stars and seas, Maury published a radical proposal. He argued that, contrary to what everyone thought, short journeys don’t just happen by luck. Instead, ships could stick to specific paths for each season that would help them?“blaze a way through the winds of the sea”. With data, he said he could find these ideal routes.

Maury didn’t have much scientific credibility at the time, so he asked the U.S. Navy for help. They sent memos to the captains of all naval ships to ask for their help, but none responded.

He then turned to the?Bureau of Ordnance and Hydrography. He tried to appeal to their sense of patriotism,?saying?how shameful the United States’ contribution to nautical science was. Every ship at the time, even American ships traveling up the east coast of the U.S., relied on British charts and almanacs to make their way home. But again, no response. Yet Maury was hardly discouraged.

Analyzing data from abandoned naval logbooks

Unable to get his hands on fresh data, he turned to thousands of logbooks stored in naval warehouses. It was mandatory for naval ships to complete these, but the Navy did nothing with them.

He and his team at the Naval Observatory started going through these books, cleaning the data, and turning it into insights about ocean routes. Five years later, in 1847, he was ready to publish his first results—the?“Wind and Current Chart of the North Atlantic”. Maury’s charts divided the northern Atlantic Ocean into a grid of 5-degree latitude/longitude squares. Each square included data on the directions of currents and the force and direction of winds, and there were separate charts for every month.

He compared his work in creating the charts (what we would today call data analysis and visualization) to a sculptor and his chisel. On each touch, the chisel seems to do very little, and yet by the end, the completed piece speaks for itself,?“eloquent with facts which the [sculptor] had never dreamed…”

Adding one piece of data to another often feels like a meaningless exercise. Yet the result was truly astounding. By comparing data for different areas across different months, Maury found which routes were fastest and which areas of the ocean should be avoided in each season.

Diverse types of charts and data

Thanks to Maury’s “corps of observers”, Maury was finally able to put away the old Navy logbooks and work with new data. By 1851, enough data had been collected to fill “two hundred large manuscript volumes, each averaging from two thousand to three thousand days’ observations”.

Maury and his team kept analyzing and publishing this data. From 1849 to 1853, they published?12 new sets of charts?covering the entire Atlantic and Pacific Oceans and Cape Horn.

These new charts went beyond just monthly winds and currents. They also included…

• Track Charts: ship traffic, plus general weather and wind readings
• Trade Wind Charts:?trade wind regions and nearby calm zones
• Pilot Charts: wind direction for 16 points on the compass for every square of ocean
• Thermal Charts: ocean surface temperature recordings
• Storm and Rain Charts: rain, fog, lightning, thunder, and storm frequencies
• Whale Charts: where different types of whales were being spotted and/or hunted

Maury had built a massive group of data collectors feeding him nearly real-time data, which he and his team processed, published, and sent right back to them. But he still wasn’t satisfied. He was already dreaming bigger.

With help from the U.S. Navy, Maury organized the Brussels Conference. It started on August 23, 1853, and was attended by representatives from Belgium, Denmark, France, Great Britain, the Netherlands, Norway, Portugal, Russia, Sweden, and the United States.

Seventeen days later, every country at the conference?agreed?that all their ships (both military and merchant) would use Maury’s logbooks to collect data. Prussia, Spain, Sardinia, the free city of Hamburg, the republic of Bremen, Chile, Austria, and Brazil later joined this agreement. Even the Pope joined in, creating honorary flags for papal ships that collected data with Maury’s logbooks.

Maury and his team soon found themselves processing heaps of incoming data, sent from around the world and written in different languages. They translated, analyzed, published, and distributed the data as quickly as they could, creating new charts for the Indian Ocean and revising the earlier Atlantic and Pacific charts.

These charts proved incredibly accurate. The accuracy and power of Maury’s charts galvanized the entire world. Captains were so excited to test them that they turned their planned voyages into global races.

After Maury’s charts, voyage times were down across the world. The travel time for the route that started everything — from the U.S. to Rio de Janeiro — decreased from 41 days to an average?30 days. The?Sea Serpent?even completed the trip in a record?18 days.

Before, a round-trip voyage from the U.S. to Australia took around 250 days, using the standard route around the Cape of Good Hope. Maury’s charts showed that ships could make the journey in far less time by taking a different route. Soon, the round-trip passage from the U.S. to Australia was down to under?200 days.

In the pre-globalization, pre-data world, Maury’s mission to make sea travel safer and faster united the whole world behind a data project that was big even by today’s standards.?Read the full story here.

??How WeWork Built a Culture of Trust, Transparency, and Governance with Atlan and Snowflake

There’s a lot to learn from how this 15-member data team with 1,500 data users rebuilt a culture of trust and transparency amidst many ups and downs.?Harel Shein?and I took over the stage at the Snowflake Summit to share the journey of how WeWork's amazing data team achieved this.?And our session was a FULL HOUSE at the event!

ICYMI, here’s a short video interview from the summit that Harel and I gave to share a glimpse into WeWork’s journey with Atlan, Atlan’s partnership with Snowflake, and why we’re excited about the future of metadata.

?????ICYMI our flash mob song: Activating Metadata on the tune of Levitating by Dua Lipa

We released Atlan’s own version of the song Levitating by Dua Lipa, Activating Metadata, at this year’s Snowflake Summit. And oh, the response has been SO GOOD. ?? Here are some of the lyrics:

If you wanna run away with me, I know a home for teams

Where we can co-lla-bo-rate

dbt, Looker, Tableau, Snowflake, and Airflow

All your tools, in one place

Analysts all right, scientists all night, collaboration feels so right

If you feel the need for a home for your data team

You met us at the perfect time…

Your data cloud, it’s activated

Yeah, yeah, yeah, yeah, yeah

Check out the full performance of the song here.

???From my reading list

• 5 Operational Challenges in Adopting Data Mesh Architecture?by Michael Ryan
• Creating the Data Experience?by David Jayatillake
• Data Mesh: Topologies and domain granularity?by Piethein Strengholt
• Data Contracts at GoCardless — 6 Months On?by Andrew Jones
• Why Data Product Management is Different — 3 Important Lessons Applied?by Sven Balnojan

I’ve also added some more resources to my data stack reading list. If you haven’t checked out the list yet, you can find and bookmark it?here.

???Next Stop: San Francisco

Last week, we were in San Diego, this week in Las Vegas, and next week in San Francisco to meet the amazing data community. The highlights? Great food, fun games, amazing conversations, and lots of appreciation for people working in data. ??

I’ll be in San Francisco later this month for Databricks Summit (and we have an upcoming Atlan community meetup). If you’re around,?ping me for an invite! I’d love to meet up.

If you’re new here, check out the?archive of this?newsletter?on Substack. I'll see you next week with some interesting stuff around the modern data stack.?

See you next week!

P.S. Liked reading this edition of the newsletter? I would love it if you could take a moment and share it with your friends on social.