How can you make metadata portable across platforms and tools?

To ensure metadata portability, employ standards like Dublin Core or AWS Glue Data Catalog's schemas. Design metadata repositories using tools like Apache Atlas or AWS Glue for centralized storage. Adopt metadata APIs for seamless integration between systems, enhancing interoperability. Leverage automation tools for metadata extraction, transformation, and loading processes, reducing manual effort and ensuring consistency. Implement metadata quality validation procedures to maintain accuracy, utilizing techniques like data profiling and cleansing. If on AWS; leverage Glue for metadata cataloging and Lake Formation for data lake management & Governance workloads, to ensures portability across diverse platforms and tools.

Employing standards for metadata is a first step, also, try to use tools that make that metadata portable across platforms and providers. Exporting your data to JSON, XML, and keeping it in cold storage can help leverage its use in different projects and help access. One standard not mentioned here is OWL (Web Ontology Language). These formats are designed to be machine-readable, also, it is important to communicate to stakeholders and developers using some more human-readable format, including ontology diagrams, schemas, and a mixture of machine generated and human generated documentation.

A standard should be selected based on what is best suited for the given data domain and the intended use of the metadata. For example, for audio content, one possible metadata standard that could be used is PBCore.

Metadata repositories are a great way to centralize metadata and make it available to stakeholders, users, and developers. They help keep the metadata updated and accesible. There are many ways to manage metadata but, even something like Airtable can help as long as there are APIs and procedures to extract, update and enforce the rules of the metadata.

Utilizing APIs is a go to for me personally in that it allows for the querying and retrieval of metadata in a standardized way, which is fully customizable to your business needs. APIs also provide a flexible mechanism for accessing metadata across different platforms and tools.

Instead of trying to solve or orchestrate metadata with "data catalog" i think this current year we should consider how to develop data marketplace on our data environment, especially who wants to implement data mesh.

Metadata management, sharing, orchestration and portability is an ongoing concern for architects, for data governance experts and can severely impact operation and planning, specially for an analytics area, where data quality and governance are of paramount importance. I've seen data scientists avoid using certain tables or bases because they know there is bad data and metadata does not reflect this, so management lagged behind and data quality efforts were directed elsewhere.

If you want to make metadata portable across platforms and tools, you need to understand what the overall goal is. If you want the metadata to be accessible through multiple tools, you have just created duplication, which means you've given yourself a task of keeping the metadata in sync. If the metadata is in a very restrictive tool and you want to make it more available to people, you have to give them read-only access in a more friendly environment and keep it up-to-date when there are changes in the "official" tool. If you're decommissioning a metadata tool and want to move it into another tool first, you need to plan your approach so you don't lose any metadata.

Documentation and adopt Best Practices: Documenting metadata schemas, standards, and best practices clearly and make this information accessible to all stakeholders. This ensures that everyone involved in data management understands how to maintain metadata portability.