Data catalogs and data dictionaries are essential tools for data governance, as they help you document, organize, and discover your data assets. But how do you create and maintain them effectively? In this article, we'll share some best practices for data catalog and data dictionary management, based on industry standards and best practices.
Before you start creating your data catalog and data dictionary, you need to define your scope and purpose. What are the goals and objectives of your data governance initiative? Who are the intended users and stakeholders of your data catalog and data dictionary? What are the data sources, domains, and types that you want to include and exclude? How will you measure the quality, accuracy, and completeness of your data documentation? By answering these questions, you can establish a clear vision and scope for your data catalog and data dictionary project.
As we started building a data catalog at the State Department we were overwhelmed by the diversity and quantity of data assets the agency managed. To help avoid getting bogged down, we prioritized the creation of our data catalog based on our agency’s strategic goals. This helped us know what to catalog first and what could wait. It also meant what we cataloged first were generally things people cared about and needed.
I do not disagree with the initial comments but keep in mind that in a world where information and knowledge can be passed from one person to another in seconds, keep your scope and purpose flexible. Thus, the information you collect in the short term can have a more lasting impact in the long term. Especially with more collarboration between professional colleagues...After all that is why MS Teams is so popular!
Data dictionaries reflect the meta-data (e.g. tables, columns, data types, e.t.c.) of existing data, while a data catalog connects these elements with definitions of the meaning, and governance rules, defining responsibility of stakeholders w.r.t. the elements in the dictionaries. While data lineage, or horizontal lineage document how data as documented in the dictionaries flows, the vertical lineage or semantic lineage connects data from various sources with their meaning. Typically the elements of different dictionaries are all mapped to a canonical or target date model (TDM), whose elements are in turn mapped to the elements of the glossary.
One of the key challenges faced is how to prioritize the data assets that needed to be documented first. There are so many data sources and domains that it can overwhelming to decide where to start. To overcome this, using user stories to capture the needs and expectations of the data consumers, such as analysts, data scientists, and business users can identify where the value lies. By using agile data governance, value can be delivered incrementally. Receiving feedback from the data consumers along the way, helps improve the quality and usability of the data catalog. If you are interested in learning more about agile data governance for data catalog projects, please feel free to comment below or message me.
There are many tools and formats available for creating and maintaining data catalogs and data dictionaries, ranging from spreadsheets and word processors to specialized software and platforms. The choice of tools and formats depends on your budget, resources, preferences, and requirements. However, some general criteria to consider are: ease of use, collaboration, automation, integration, standardization, security, and scalability. You want to choose tools and formats that make your data catalog and data dictionary easy to create, update, share, and access, while ensuring consistency, quality, and security.
One of the challenges of data management is to avoid duplication and fragmentation of data assets. That's why a data catalog should not only be a repository of metadata, but also a "catalog of catalogs" that can direct users to other software and platforms with data dictionaries and similar functionalities. Instead of reinventing the wheel, a data catalog should be able to pull in data dictionaries and metadata assets that are stored across the enterprise in other software, such as data governance tools and documentation tools. This way, users can have a single point of access to all the relevant information about their data, regardless of where it is stored or managed.
A data catalog and a data dictionary should follow a consistent structure and style, so that they are easy to read, understand, and compare. A data catalog typically contains metadata about the data sources, such as name, description, location, owner, format, size, frequency, and lineage. A data dictionary typically contains metadata about the data elements, such as name, description, type, format, range, values, rules, and relationships. You should use clear and concise language, avoid jargon and acronyms, and follow common standards and conventions for naming and formatting your data catalog and data dictionary entries.
Data types should be as precise as possible: - Summable (extensive) variable vs non-summable (intensive) variable - Independent variable (dimension) vs dependent variable (measure) - String vs Category vs Relationship - Sortable vs non-sortable Also, the more metadata, the better: - Type of missing data (MCAR, MAR, MNAR) - Summary statistics (including percentiles if you can afford them) - Probability distribution - Sources and licenses
As computer applications become more progressively functional plan to expand the structure to include more details; think has any problem been happened because of more available information? Or have more problems arisen because of a lack of details?
Normalizing a business vocabulary may require reconciliation and mapping tasks. To support normalization, a crosswalk mapping entities serves as a translation layer for the catalog. Chosen labels and standardized metadata elements can be represented as the preferred label while other labels & metadata can be handled as alternative labels or aliases. By implementing crosswalks, system and data architecture changes are captured and documented. Data catalogs require methods to accurately handle data streams, data representation and normalization of vocabularies.
Creating a “Data Catalog Playbook" sets guidelines for consistent data documentation. It includes standards for naming, descriptions, and tags, ensuring a uniform style. Within this, providing category guidelines and templates will help structure information consistently. Procedures for version control maintain documentation accuracy over time. The playbook encourages collaborative practices, provides training resources, and establishes quality checks. Feedback mechanisms then allow users to contribute to continuous improvement, fostering a standardized and effective data catalog.
Creating a “Data Catalog Playbook" sets guidelines for consistent data documentation. It includes standards for naming, descriptions, and tags, ensuring a uniform style. Within this, providing category guidelines and templates will help structure information consistently. Procedures for version control maintain documentation accuracy over time. The playbook encourages collaborative practices, provides training resources, and establishes quality checks. Feedback mechanisms then allow users to contribute to continuous improvement, fostering a standardized and effective data catalog.
Creating and maintaining a data catalog and a data dictionary is not a one-time or one-person task. It requires ongoing collaboration and communication among your data community, which includes data owners, producers, consumers, analysts, stewards, and users. You should involve and engage your data community in the data catalog and data dictionary project, by soliciting feedback, input, and validation, assigning roles and responsibilities, providing training and support, and promoting awareness and adoption. You should also establish a data governance framework and policy that defines the roles, rules, processes, and standards for your data catalog and data dictionary management.
The two important factors, if not the most important factors, are (1) the colleague(s) imputing the information knows the information is correct or can at least confer with a colleague who can double check - especially when reviewing and reporting on numbers; and (2) ensure the person takes his or her time slowly and carefully. Involving/engaging your data community is great. But if the data is incorrect, the decisions from which the data is based upon will be wrong from the start.
Your data community is the lifeblood of your catalog, as they contribute, update, and use the data assets. To foster a culture of collaboration and trust, you should provide them with the tools and guidance they need to participate in the catalog creation and maintenance. This includes creating playbooks and training materials that empower data owners and stewards to organically contribute to the catalog, as well as providing feedback mechanisms and recognition for their efforts. By doing so, you will ensure that your catalog grows exponentially and reflects the needs and expectations of your data community.
A data catalog and a data dictionary are living documents that reflect the current state and quality of your data assets. Therefore, you need to review and update them regularly, to ensure that they are accurate, complete, and relevant. You should establish a schedule and frequency for reviewing and updating your data catalog and data dictionary, based on the changes and updates in your data sources and elements. You should also use tools and techniques that automate and streamline the review and update process, such as data profiling, data lineage, data quality, and data validation.
A good way to conduct data asset reviews is to establish a review cadence utilising a review council, such as a data governance community of practice. A review council is a group of stakeholders who are responsible for reviewing the data assets and providing feedback, approval, or rejection. The review council should include representatives from different functions and roles, such as data owners, data stewards, data analysts, data consumers, and data quality experts. By following these best practices, you can ensure that your data assets are well-managed, well-documented, and well-aligned with your business goals and needs. You can also foster a culture of data literacy and trust within your organization.
One critical point you should always consider when reviewing and updating information is to define change. Thus, you can automatically tell when the change is so significant in the data, if it is worth noting in a database for further study and analysis.
A data catalog and a data dictionary are not only useful for data governance, but also for data analysis, data science, and data-driven decision making. By creating and maintaining a data catalog and a data dictionary, you can leverage the benefits and value of your data assets, such as: improving data discovery, accessibility, and usability; enhancing data quality, trust, and compliance; enabling data collaboration, sharing, and reuse; facilitating data analysis, insight, and action; and driving data innovation, value, and impact.
Every important definition should be independently verified by and SME who did not help write it. While definitions are under development make sure they are not visible to users in the production version of the Business Glossary, as this drives confusion. A definition does not have to be perfect to be put into production, but it must be of a reasonable standard. After that, definitions can be improved indefinitely - they are not "once and done". Do not dismiss multiple different definitions for a given business term. They have a high probability of being correct. Instead, try to give each valid concept (definition) its own unique business term.
Storytelling is a powerful way to communicate the value and impact of our data assets to the leaders and stakeholders who need to understand and support our data governance efforts. One of the stories I like to tell is how the data catalog and data dictionary enable us to have a global view of our data assets, and learn from other regions and functions analytic capabilities. The data catalog and data dictionary are not just technical tools, but strategic assets that enable us to leverage the benefits and value of our data. They help us to enhance our data discovery, accessibility, usability, quality, trust, compliance, collaboration, sharing, reuse, analysis, insight, action, innovation, value, and impact.