2023 Product Roundup: AI, Data Mesh, and a New Age of Data Governance

100 new features. 20 new integrations. 2 massive product launches. A first of its kind customer conference. At Atlan, 2023 was the year of building the future, with data teams that are envisioning it. And we couldn’t be more excited to recap the year, so let’s get started.

The First AI Copilot for Data Teams

Generative AI changed the technology landscape in 2023 and every data team started thinking about the ways in which they could use AI to become more productive.

Armed with learnings from when we were a data team, we seized the opportunity to make the lives of humans of data better with Generative AI. In April 2023, we called for a company-wide AI hackathon, sourcing ideas for the future of data catalogs and data governance.?

Two months later, in June 2023, we launched Atlan AI. By partnering with Microsoft and using Azure OpenAI Service , Atlan was the first to bring AI superpowers to data catalogs, leading a wave of AI innovation for data governance.

Make your team love documentation with Atlan AI

In the old world, everyone neglected documentation, blocking businesses from trusting data and making faster, better decisions. Atlan AI brings an end to the pain by auto-generating descriptions for a wide range of data assets in Atlan. All data producers have to do is review, edit, and publish.?

After 6 months of testing with customers under a closed preview, we’ve seen Atlan AI’s description suggestions shine with an acceptance rate of 60%, giving humans of data more time to work on business-critical projects.

Explain lineage transformations with Atlan AI

Lineage simplifies transformations by taking complexity away, showing relationships between data assets. But when a data analyst wants to dive deeper into how and why a data asset was transformed, the process of parsing through complex SQL queries can take hours. But not anymore.

Atlan AI helps data analysts and engineers understand complex lineage transformations by explaining what’s going on in natural language.?

It’s not an exaggeration to say that data catalogs and data governance are changing forever, and for the better, because of AI.

The Future of Data Governance?

In 2023, we hosted a first of its kind community conference on modern data governance — Re:Govern . Nearly a thousand humans of data came together to hear visionary data leaders, from companies like Nasdaq, Fox, Autodesk, Elastic, and HelloFresh, talk about their strategies and playbooks for modern data governance.?

If there was only one key takeaway from Re:Govern , it would be this: the future of data governance will look very different from its past.?

Amy Raygada (Swiss Marketplace Group) and Mark Kidwell (Autodesk), shared their visionary approaches to building a data mesh. Takashi Ueki (Elastic) shed light on building automating trust through data contracts. And Mihir Modi (FOX) explained his vision for data products and AI in data governance.?

Data teams are leaving behind the old world of manual, siloed, one-size-fits-all approaches for a new world where automation & AI, collaboration, and flexibility are key to success.?

That’s why, in 2023, we started innovating towards a future where data mesh comes to life, AI is embedded in our workflows, and manual, traditional governance is automated. With that, Let’s recap 2023’s data governance updates:

Bring your data mesh to life with Atlan Mesh

Atlan Mesh is the first ever native data mesh experience in a data catalog. It’s an experience that caters to data consumers and improves their understanding of the data estate. Here’s how:

• Data products as first-class citizens: Treating data as a product requires a native solution, not workarounds. Data products can now be created & curated natively in Atlan, within a brand-new home for easy, context-rich discovery for business users.
• Dedicated spaces for federated domains: With Atlan Mesh, each domain gets its team’s own workspace and landing page to house curated data products and documentation for data consumers.
• Business lineage: Traditionally, lineage has been a technical tool for data producers to understand impact and find root causes. Atlan Mesh introduces business lineage for data consumers, who want to understand the provenance, not the technical architecture, behind how data products are created to guide usage decisions.?
• Data contracts: To support the creation of data products and proactively bridge the gap between data producers and consumers, we introduced a new vision for data contracts in Atlan.

We’ve already started rolling out Atlan Mesh to the first few customers, who will be a part of the Atlan Mesh Advisory Council, and we’re excited to build the future of federated, flexible data governance together.?

Manage compliance in one home with Tag Management

As the modern data stack continues to evolve, data teams need to ensure the right people have the right access to the right data. This involves identifying sensitive data and protecting it with the right access controls, while serving trusted data to data consumers.?

To solve this challenge, we launched Tag Management — a new way for your data team to manage compliance and security — and became one of the first Snowflake data governance partners to enable bi-directional tag sync between Snowflake and Atlan.?

Tag Management enables you to:

• Create tags natively or import tags from tools like Snowflake into Atlan
• Classify data assets with tags at scale using Atlan’s Playbooks
• Sync tag updates in Atlan back to data sources like Snowflake

With Atlan’s Tag Management for Snowflake, our team will have one central home to manage tags. Bi-directional tag sync will empower our data producers to tag assets where they work and enable our platform team to manage tags and permissions seamlessly.”

Understand impact and optimize costs with Popularity & Usage metrics

As a data leader, you’re always looking to get the most out of your data, while controlling costs. But you need visibility into who’s actually using what data in which tools.?

In 2023, we launched Popularity & Usage for 4 connectors — Snowflake, Databricks, Power BI, and Redshift — giving data teams the ability to:

• Discover the most or least used assets by sorting by popularity
• Understand popularity in lineage with popularity indicators & pop-ups
• See who’s using data with Top Users and Recent Users
• Optimize the data estate with Popular, Slow, and Expensive queries

With the launch of Popularity and Usage for Snowflake, Mistertemp, a leader in recruitment and temporary work based in France, deprecated 50% of unused Snowflake tables and over 60% of their Looker assets:

Everything downstream changed. We were able to see every existing connection in Fivetran. We could see what was actually used. We kept those, and for everything else, we would disconnect.”

The Era of Active Metadata

Active metadata has always been core to Atlan’s platform, and in 2023, we saw it becoming the center of data estates across industries and businesses too.

This year, we had 20 new data leaders, from businesses like Docker, Purple, and Datacamp,? join the Active Metadata Pioneers club — a visionary group that is pushing the boundaries of metadata forward by making active metadata a priority. And to close out the year, in November 2023, G2 launched its first Active Metadata Grid Report, driven by reviews from real users, with Atlan being the only leader in the category.?

With that, let’s recap 2023’s active metadata advancements:

Don’t go breaking my heart dashboards with Metadata CI/CD

Impact analysis is a tiring, time consuming, and disheartening process for data engineers. But without it, one small change could break thousands of dashboards. What if impact analysis could be proactive and preventative, instead of reactive and manual? Say hello to Metadata CI/CD.

With integrations for GitHub and GitLab, Metadata CI/CD automatically surfaces impacted assets right in the data producer workflow. This means data engineers don’t have to manually check impact and business users can trust their dashboards, which break less often.?

After its launch, Metadata CI/CD helped a data team on Atlan realize that the request for a column name change could impact more than 1,000 business-critical dashboards.

Atlan has been a great help. We no longer have to rely on these documents, and we’re able to do impact assessments at the click of a button.”

Bringing metadata to everyone’s favorite tool: spreadsheets

There’s one data tool that has stood the test of time: Excel.?

In 2023, we released and upgraded our integrations with Microsoft Excel and Google Sheets to enable new use cases:

• Accelerate documentation by enriching metadata at scale?

By importing data assets from Atlan into Excel or Sheets, you can now document descriptions, certificates, owners, tags, and announcements for your column assets using spreadsheet flexibility and sync the metadata updates to Atlan with a single click.

• Build trust and keep end users informed with impact analysis

You can now analyze impact faster by importing impact analysis into a spreadsheet and add announcements to keep end users informed.

Bring business and data together in Microsoft Teams?

To create true company-wide adoption, you need to meet your users where they work. That’s why Atlan now integrates with Microsoft Teams. You can now accelerate your data and business projects with better, cross-functional collaboration around data.?

Atlan’s integration with Microsoft Teams enables you to:

1. Share data assets & ask data questions in a Microsoft Teams channels without leaving Atlan
2. Link critical, context-rich Microsoft Teams threads to Atlan assets
3. Get notifications & alerts in selected Microsoft Teams channels

Push the boundaries of metadata with Webhooks and new Python and Java SDKs

Preparing your data estate and team for mission-critical data projects, like AI models, needs a platform approach to metadata. From event-driven metadata use cases like alerting to derived metadata use cases like Data as a Product scoring, data teams are making the future of metadata come to life with this year’s extensibility improvements.?

In 2023, we launched the Java and Python SDKs, enabling data teams to build custom active metadata use cases like:

• Governance reporting: Measure the success your governance initiatives by automating metadata enrichment reports.
• Custom connections: Connect Atlan to your enterprise homegrown systems to enable end-to-end discovery and lineage.
• Derived metadata: Create custom metadata such as a “Metadata Completeness Score” or “Data as a Product Score” by analyzing metadata enrichment.
• Metadata migration: Automatically migrate all your existing metadata from your legacy data catalog to Atlan.

Along with Python and Java SDKs, we also launched support for Webhooks in 2023 — opening up the world of event-driven metadata use cases.?

Webhooks allow you to monitor events happening in Atlan, receive notifications to a URL of your choice, and take action immediately. For example, you can create a webhook to send notifications to your email address or collaboration app, like Slack or Microsoft Teams, when a term is updated or an asset is tagged.?

The possibilities are truly endless and we’re excited to see the future of active metadata, built not by Atlan, but by data teams around the world.

A Collaboration to Deliver Trusted Data?

With every tool in the modern data stack becoming increasingly siloed, the humans of data are becoming siloed as well.?

Data consumers, who live in BI tools, don’t have visibility into the upstream pipeline world of data producers. So when things go wrong, they’re often the last to know. And data producers don’t know how their code changes are breaking downstream dashboards.?

This year, we released native, out-of-the-box connectors for 18 new tools, covering spaces like Data Quality, Data Observability, Data Orchestration, and Business Intelligence, to bring the world of data producers and consumers closer together. Let’s recap 2023’s key integrations.

All-New Partnerships with Data Quality Tools

Nearly 75% of the time, when things go wrong, business stakeholders are the first to identify data issues. It doesn’t have to be this way. Data quality is a fundamental signal into data trust, but it needs to meet business users where they work, along with the right metadata context.?

That’s why, in 2023, we launched two new out-of-the-box integrations with leaders in the Data Quality and Observability market: Monte Carlo and Soda.?

Monte Carlo x Atlan

With Monte Carlo and Atlan, businesses can gain an up-to-date understanding of their data health, build trust in data, and support innovative new ways to approach distributed data infrastructure. The native Monte Carlo integration gives you the ability to:

• Discover Monte Carlo incidents and monitors in Atlan with Monte Carlo-specific filters.
• Democratize Monte Carlo’s data quality signals wherever business users work with Atlan’s Chrome extension.
• Accelerate root cause and impact analysis through Monte Carlo incidents being surfaced in Atlan’s column-level lineage.

With Monte Carlo and Atlan, we can catch data incidents early on, and provide everyone with clear visibility into the current status of data accuracy. This has been critical for the executive team to have confidence we can deliver on our promise of reliable, trustworthy data."

Soda x Atlan

Atlan and Soda’s native integration provides data teams with an intuitive and comprehensive platform to find, trust, and use the right data. The native Soda integration gives you the ability to:?

• Discover Soda data quality metrics and results in Atlan.?
• Inform data users of data issues before they make decisions by highlighting Soda’s check results in Atlan’s Chrome extension.
• Accelerate root cause and impact analysis through Soda checks being surfaced in Atlan’s column-level lineage

BI Connectors for Business Adoption?

To drive adoption of your data catalog platform and data governance initiatives, you need to meet business users where they work: in BI tools. In 2023, we released native, out-of-the-box connectors for 6 new BI tools to help data teams drive company-wide adoption:

• Sigma
• Qlik Sense
• Amazon Quicksight
• MicroStrategy
• Thoughtspot
• Sisense

?These BI connectors enable businesses to:

• Build a verified, single source of truth for BI assets by enabling discovery and documentation of BI assets in Atlan.
• Enable proactive impact analysis of downstream dashboards and use cases by connecting BI assets to upstream warehouse, ELT, and source assets with cross-system lineage.
• Build trust in data by surfacing trust signals from data quality, observability, and orchestration tools.?

I’ve had at least two conversations where questions about downstream impact would have taken allocation of a lot of resources. Then actually getting the work done would have taken at least four to six weeks, but I managed to sit alongside another architect and solve that within 30 minutes, saying ‘If you’re changing the column name or adding an extra column, this is what it’s going to break or impact.”

Bringing ELT & Orchestration Context to the Business

Operational metadata can bring powerful context from pipeline processes for both business users and data teams. In 2023, we invested in building native connectors to tools that could help answer this question: when a pipeline fails, how do you inform a business user and help a data engineer find the root cause faster? That’s why we built our first event-driven integration with Airflow and OpenLineage.?

Airflow x OpenLineage x Atlan

In 2023, Airflow, OpenLineage, and Atlan partnered to build an ecosystem of trust by making real-time pipeline observability a reality. Here’s how:

• Get a comprehensive overview of your Airflow pipelines in Atlan by cataloging and documenting Airflow assets, such as DAGs and tasks.?
• Track and monitor your pipeline in one home with real-time operational metadata like task run details and statuses.?

Understand impact of your Airflow pipelines by visualizing how Airflow DAGs and tasks connect to your data assets with cross-system lineag

Checking out Atlan? Try our product tour .?

Thinking about data governance? Chat with us.