Episode 6: Part 2 - Documenting Data Lineage

The workshop took longer than expected and we had to continue the following day.

Chike introduced me to the Monthly Operational Performance Report, an important dashboard used by senior management to monitor key metrics like transaction volume, branch efficiency, and customer satisfaction.

“Our job,” Chike said, “is to document the lineage of this report. We need to show how the data flows from its source systems, through processing steps, to its final presentation in the dashboard.”

Step 1: Identifying Data Sources

We began by reviewing the data sources feeding into the report:

• Customer Database: Contains customer profiles and branch assignments.
• Transaction Logs: Tracks daily financial transactions at each branch.
• Employee Records system: Holds branch performance metrics.

Each source was linked to metadata that described its structure, owner, and update frequency.

Step 2: Mapping Data Transformations

Next, we looked at how the data was processed before reaching the report:

• Data Cleansing: Duplicate records were removed, and missing values were handled.
• Aggregation: Transaction data was grouped by branch and region for easier analysis.
• Validation: Each dataset was cross-checked by the operations team for accuracy.

Chike demonstrated how to use the data catalog tool to map these steps.

“Each transformation adds context to the lineage,” he explained. “If there’s an issue in the report, this map helps us trace it back to the source and fix it.” He added these details to the data catalog, noting each step in the process and linking it to the relevant metadata.

Step 3: Documenting Metadata

With the lineage mapped, we documented metadata for each stage:

Descriptive Metadata

• Title: Monthly Operational Performance Report.
• Summary: A dashboard summarizing branch performance metrics for senior management.

Provenance Metadata

• Data Origin: Extracted from customer and transaction systems.
• Processing Steps: Aggregated and validated before being visualized in Power BI.

Technical Metadata

• Format: Source data stored in CSV files; final report in a Power BI dashboard.
• Frequency: Updated weekly.

Administrative Metadata

• Owner: Operations Team.
• Retention Policy: Stored for 12 months, then archived.

Step 4: Validating the Metadata

Once we’d mapped the lineage, Chike showed me how to validate the metadata:

• Accuracy: Cross-checking metadata against source system documentation.
• Completeness: Ensuring every data element had descriptive, structural, and administrative metadata.
• Consistency: Making sure naming conventions matched the bank’s standards.

“This is where metadata governance comes in,” Chike said. “Good metadata needs to be understandable and usable by everyone.”

Step 5: Validating the Lineage

After completing the documentation, we reviewed the lineage with key stakeholders:

• Operations Team: Confirmed the accuracy of the aggregation and validation steps.
• IT Team: Verified technical details, such as data formats and update schedules.
• Compliance Team: Ensured the metadata aligned with regulatory standards like NDPR.

Their feedback helped refine the lineage and fill in missing details.

As we wrapped up, Chike asked me what I’d learned from the task.

“Data lineage is like telling a story,” I said. “It shows where the data started, what happened to it, and how it ended up in the report. Without this, we’d be working blind.”

Chike nodded. “Exactly. Metadata—and lineage in particular—is what makes our data governance work. It gives us traceability, accountability, and trust.”

That evening, I felt a sense of accomplishment. This task has shown me metadata’s power to bring clarity and order to the rather complex world of data governance.