Detect Anomalies in Your Data and Empower Data Stewards with a Copilot Agent for Faster Remediation of Data Health Issues

Co-author: Shahruz Mannan, MSc. in Computer Science and Engineering, University of Washington, USA

Overview

Anomaly detection, powered by AI and integrated with Copilot to assist data stewards in remediation, is one of the most effective ways to identify and address data issues quickly. Leveraging advanced techniques such as machine learning, deep learning, and statistical modeling, anomaly detection enables organizations to proactively manage data quality problems before they escalate into significant challenges.

In the age of AI, the reliance on trustworthy data is continuously increasing. Data-driven business decisions are directly influenced by the quality and reliability of the data used as the foundation for these decisions. Effective data quality monitoring and foresight are critical for organizations, such as timely anomaly detection, ongoing remediation, and preventive actions help maintain the health of their data estate.

Faster identification of data issues means detecting problems as they occur, significantly reducing the time spent searching for errors manually. Early detection prevents downstream impacts by addressing problems early in the data supply chain, minimizing the costs associated with data correction and downstream business disruptions. By implementing AI-powered anomaly detection and leveraging tools like Copilot, organizations can empower data stewards to efficiently track, resolve, and prevent data quality issues. This approach not only enhances the trustworthiness of data but also ensures faster, more informed business decisions.

Types of Anomalies

Anomalies in data represent unexpected values or behaviors outside the regular and acceptable range. Anomaly detection identifies data points, patterns, or events that significantly deviate from expected norms. In the context of data quality and the health of a data estate, anomalies often indicate issues such as:

1. Outliers: Data values that fall outside the acceptable range.
2. Missing or Incomplete Data: Sudden gaps in datasets.
3. Inconsistencies: Conflicting or contradictory data entries.
4. Format Errors: Data that does not conform to predefined standards.
5. Trend Deviations: Sudden shifts in data trends or distributions, such as: Abnormal changes in the sum of numeric values and in average Abnormal changes in minimum and maximum value ranges and aggregate values.
6. Sudden Volume Increases: Data volumes exceeding regular trends.
7. Duplicate Records: Duplicate entries in business-critical datasets.
8. Late Data: Data that has not arrived or landed on time.
9. Point Anomalies: Notable deviations from the rest of the dataset.
10. Value Range Issues: Values falling outside predefined ranges.
11. Percentile Anomalies: Unusual percentages of anomalous values within datasets.
12. Relative Changes: Significant deviations in maximum and minimum values compared to historical data.

Anomaly Detection Techniques

There are numerous techniques for anomaly detection that data scientists employ to help enterprises address their data health issues. These techniques include, but are not limited to:

1. Flagging Data Outside a Given Range: Identifying data points that fall beyond predefined acceptable limits.
2. Profiling-Based Anomaly Detection: Using data profiling metrics, such as standard deviation, to detect anomalies.
3. Identifying Anomalies in Sequential Data: Detecting irregularities in time-series or sequential datasets.
4. Labeled Data Classification: Utilizing labeled datasets to classify and flag anomalies.
5. Machine Learning Techniques: Applying supervised, unsupervised, and deep learning algorithms to detect anomalies.
6. Predefined Rule-Based Detection: Using predefined rules for known anomalies, such as: (a) Flagging unusually low or high transactions. (b)Detecting unexpected data volumes. (c) Identifying unexpected PCI/PII (Payment Card Information/Personally Identifiable Information) data.

Creating Alerts to Drive Actions

Implementing alerting for anomalies is a critical step in ensuring that detected issues are addressed promptly and efficiently. Alerts notify relevant stakeholders when anomalies are detected, enabling immediate investigation and resolution. Effective alerts should be timely, informative, and prioritized based on severity. Key components of alerts include:

• Alert Conditions: The specific triggers for an alert, such as data threshold breaches, missing data, or detected outliers.
• Alert Severity: Categorization of alerts based on their impact, such as critical, high, medium, or low.
• Notification Channels: Methods for delivering alerts, such as email, Teams chat, or alert dashboards.
• Alert Metadata: Information included in the alert, such as the type of anomaly, affected dataset, and timestamp.

Alerts should be created for abnormal patterns, range deviations, unexpected volumes, and data inconsistencies. Data quality stewards can configure thresholds for alerting and actions to avoid unrealistic or excessive alerts. Thresholds should be set based on the criticality and understanding of the data as well as its importance to business use cases. Generic thresholds may not provide significant value to the organization. Prioritizing alerts, reducing false positives, integrating with data governance workflows, integrating with a unified data catalog, and providing actionable insights for the continuous improvement of data estate health.

To maximize the effectiveness of anomaly detection and alerting, organizations should focus on:

• Prioritizing alerts with high business impact to avoid alert fatigue.
• Using contextual or AI-driven alerts to minimize unnecessary notifications.
• Including all relevant details in alerts to guide stakeholders on the next steps.
• Ensuring that all anomaly actions are logged and auditable.
• Defining thresholds based on the criticality of the data, recognizing that not all data is equally important. Data minimalism should be considered before deploying anomaly detection features.
• Periodically reviewing and updating alert configurations to align with evolving business needs.

Integrating anomaly alerts with data governance workflows, unified data catalogs, and actionable insights can further enhance the continuous improvement of data estate health.

Copilot Agent for Assisting Data Stewards in Managing Anomalies and Alerts

A Copilot agent can significantly assist data stewards in tracking anomalies, managing alerts, automating actions, and optimizing thresholds to improve data quality. The Copilot agent provides a central interface that helps data stewards view and track the status of related actions and analyze historical trends of detected anomalies across datasets. This means a unified view for data stewards to see anomalies across their entire data estate, covering all data and governance domains in one place. The Copilot agent can also help data stewards find affected sample records, assess the severity of anomalies, and trace anomalies back to their origin using lineage data. Additionally, the Copilot agent can assist data stewards in adjusting or updating thresholds and provide suggestions for possible fixes to data quality issues based on historical trends, business rules, data sources, data lineage, and feedback.

Practical Example and Use Case of Anomaly Detection

Here are practical examples of anomaly detection applied to sales, finance, and customer data to improve data quality and enhance operational efficiency:

Sudden Dip or Spike in Regional Sales

A sharp drop or spike in sales for a specific region or product that deviates significantly from historical trends. There can be many reasons:

• Missing or incomplete sales records in the dataset.
• Data entry errors (e.g., an extra zero causing inflated sales figures).
• Incorrect mapping of transactions to regions or product categories.

Anomaly detection can monitor daily sales for each region, identify anomalies, and flag them. By integrating with a copilot agent, corrective actions can be recommended, such as inputting missing data using historical sales averages or fixing erroneous mappings. The impact of such data issues is significant. For large corporations, the sales team relies on accurate numbers for sales execution, inventory planning, and marketing strategies. Accurate and timely sales reporting enables better decision-making in these areas. Additionally, it helps identify lost revenue opportunities by detecting data gaps and improves forecasting accuracy by addressing data inconsistencies.

Unusual Transaction Amounts

Transactions with unusually high or low amounts, or repeated identical transaction values. There can be many reasons:

• Incorrect entries (e.g., transposing numbers: $1,900 instead of $19,000).
• Fraudulent activity, such as unauthorized transactions.
• Duplicated financial records due to errors in system synchronization.

Anomaly detection techniques can flag suspected fraudulent or duplicated transactions for review. Historical trends can be analyzed to recommend corrections, or a copilot agent can be trained to make approximate adjustments to ensure financial integrity, support compliance, and reduce manual reconciliation efforts. The key benefit of anomaly detection is that it helps finance organizations reduce financial risks by quickly identifying fraudulent transactions and enables them to maintain compliance and audit readiness with clean financial records.

Duplicate or Inconsistent Customer Records

Having multiple records for the same customer with slight variations in name or address (e.g., John Smith vs. Jon Smith) has direct impact on marketing campaign ROI, revenue allocation accuracy, sales quota setting, and street reporting. The root cause for this kind of data issue can be:

• Errors during data entry or integration from different systems.
• Inconsistent formatting of customer details across platforms.
• Incomplete profiles (e.g., missing email addresses or phone numbers).

There are various anomaly detection techniques that can be utilized to identify and merge duplicate records by analyzing similarities in names, addresses, and contact information. Rule-based checks can also be employed to flag incomplete customer profiles, such as those missing mandatory fields like contact details.

Providing a single customer view, improves CRM effectiveness, and ensures cleaner data for customer analytics and personalized marketing is critical for the organization. The use of anomaly detection and correction, combined with copilot agents, helps organizations enhance customer segmentation and personalization by ensuring data integrity. Additionally, it prevents inefficiencies in marketing campaigns caused by duplicate or incomplete data, ultimately improving operational effectiveness and campaign performance.

Benefits of Anomaly Detection

Anomaly detection offers significant benefits in managing data health, particularly for business-critical data. Key advantages include:

1. Faster Identification of Data Issues: Detecting anomalies as they occur minimizes the time spent searching for errors manually.
2. Preventing Downstream Impacts: Addressing issues early in the data supply chain reduces the risk of cascading errors that could disrupt business processes.
3. Cost Savings: Early detection and remediation lower the expenses associated with correcting data and resolving downstream business errors.
4. Structured and Efficient Remediation: Ensures that detected anomalies are resolved in a systematic manner, enhancing overall data quality.

For non-critical data, the value of investing in anomaly detection might be less apparent. However, for business-critical data, anomaly detection is vital. It ensures the integrity of key datasets, such as finance, sales, customer, and product data, which directly impact business decisions.

For companies, monitoring anomalies in business-critical data is essential. Examples include:

• Sudden revenue drops.
• Unexpected declines in sales opportunities.
• Abnormal product activation or consumption patterns.
• Significant increases in data volume.
• Large numbers of duplicate records.
• A sudden rise in customer opt-outs.

By leveraging anomaly detection, businesses can maintain data health, prevent costly errors, and make timely, informed decisions.

Summary

Anomaly detection powered by AI, integrated with Copilot to assist data stewards in remediation, is one of the most effective ways to identify and resolve data issues more quickly. By leveraging advanced techniques like machine learning, deep learning, and statistical modeling, anomaly detection helps organizations proactively address data quality problems before they escalate into larger issues. In the Age of AI, the dependency on trustworthy data is continuously increasing. Data-driven business decisions are directly influenced by the quality and trustworthiness of the data used for decision-making. Data quality monitoring and foresight are crucial for organizations, and timely anomaly detection, along with continuous remediation and preventive actions, will help maintain the health of their data estate. Faster identification of data issues means detecting problems immediately as they arise, reducing time spent manually searching for errors. This prevents downstream impacts by addressing issues early in the data supply chain. Effective anomaly detection is crucial for organizations seeking to fully leverage their data. By implementing proactive data quality management practices, businesses can quickly identify and address anomalies before they affect decision-making and operations. Resolving issues earlier in the process also minimizes the costs associated with data correction and downstream business errors.

References

Shafiq Mannan: Save Millions of dollars by Governing the Data that Matters for Your Business

Shafiq Mannan: How Data Quality Impacts the Achievement of Your OKR Targets?