What are some of the common data quality issues that affect your BI results and how do you resolve them?

- ?? Missing or incomplete data: Identify root causes such as human errors, system failures, data silos, or lack of validation. - ? Implement quality checks and rules: Ensure data is accurately captured, stored, and validated. - ?? Regular audits: Conduct regular data audits to identify and address any gaps or inconsistencies. - ?? Use ETL processes: Employ Extract, Transform, Load (ETL) processes to clean and standardize data before analysis.

"Missing or incomplete data can cause gaps, errors, or biases in your BI reports and dashboards, affecting your analysis and interpretation" If and when that happens, the error is in the "reports and dashboards" part where the ***missingness itself*** is not properly recognized and dealt with. The consequence of this is that there's a second resolution to this problem : revise the "reports and dashboards" part in such a way that the "missingness itself" is ***properly*** recognized and ***properly*** dealt with. "Techniques such as ... extrapolation" are a NATURAL and INEVITABLE cause of noise. Right to the extent where the actual signal gets lost in the noise.

When it comes to the key dimensions your users want to analyse or group by, for example: - In B:B we always like to analyze revenue, or service interactions, or leads by entity (customers, partners), we may want to analyze by: Industry vertical/sub-vertical, Company Products, or other useful grouping. This data is often found in the company profile. A useful technique here is to partner with a data identification, matching and enriching data provider to shore up missing data in the Company Profile subject area of your Customer (or partner) data. There are many services out there, and with the advent of AI tools that automatically and repeatedly scan the entire digital presence of the internet, the cost of such services is coming down.

There are some simple controls that help minimize missing data. Mandatory field controls with hard stops can force users to populate information, but can also lead to junk data being used simply to complete the forms if not planned with a holistic view of the data and its collection in mind. Validation services for common information like addresses can fill gaps and are useful for ensuring uniformity in format and accuracy validation. Depending on the nature of the information being collected, periodic assessment of need is a worthwhile effort to establish the current needs of the org and the actual benefits of the data vs. the costs of acquisition, which can go beyond fiscal.

To address the issue of missing or incomplete data, a comprehensive strategy is required: Identify Root Causes: The first step is to identify the root causes of data gaps and incompleteness. This involves investigating data collection processes, system configurations, data storage practices, and data validation mechanisms. Implement Data Quality Checks and Rules: Establish data quality checks and rules to ensure that data is captured, stored, and processed accurately. This includes defining data validation rules, identifying mandatory fields. Data Imputation, Interpolation, or Extrapolation: For missing values, techniques such as data imputation, interpolation, or extrapolation can be employed to estimate or fill in the missing values.

From my experience woking with finance industry, I have seen data duplication and inconsistency provide unique challanges. Duplication is areal problem with data quality, while inconsistency may be occasionally refrection of real industry trend. Duplication can be prevented by enforcing uniuq key constarint. and moving duplicate records to error staging table (which can be analyzed as a part of validation).

Business Intelligence deals mainly with aggregates on different dimensions. Duplicate records make aggregates incorrect and thus causes inefficient decisions. To avoid this, do the root cause analysis. It may be the integration jobs caused the duplicates while building the Data Warehouse of ODS; Or OLTP system may have the duplicate records; SQL Queries with improper joins may be the reason for duplicates; Depending on the scenario, a corrective action is required to eliminate duplicates before it causes damage to the dashboard.

Inheritance standards for data can drive consistency. The business can determine the most reliable 'source of truth' and adopt it as the primary source for the information and the IT developers can flow that data to all the locations in which it is needed. When there are multiple sources that have to be considered, mastering the data with survivability rules reflective of the most reliable sources can drive better data quality and outcomes. Regardless of method, when it is known there can be deltas in representation of the same facts across applications/systems, automated audit controls should be doing regular delta assessments across applications/systems and reporting the deltas found for correction.

If you formally recognize a single authority as the "authoritative source" for any single piece of information, then by definition you'll never run into "inconsistency", because you'll simply be discarding all the non-authoritative sources as contradicting the authoritative one.

Let’s understand how do we end up in this situation first. Reasons - Probable source misbehaviour pushing unexpected row/column Faulty load logic resulting in either missing or duplicate data DDL mismatch between insert job and DDL deployment job How to fix this - Check and Correct all these things stated above Implement unit test cases, data contracts or data governance and validation suites like Great Expectations packages.

The ***only*** way a ***machine*** can help is to detect inconsistencies. But for the machine to be able to do so, the "rules of consistency" must be inventorized, and subsequently declared to a machine that is able to detect the violations. Examples : keys, foreign keys and CHECK constraints in SQL DBMS's. Absent support by the machine, the only recourse left is to address the human procedures. And note that "inconsistent" is not synonymous to "invalid" or "inaccurate". I might inaccurately/invalidly claim that I am 50 yrs of age but that can only be found "inconsistent" by a machine if that machine also has knowledge of my birth date and the current calendar date.

Basic controls for this issue are validation logic, fixed lists, and automated auditing with exception reporting for corrections needed. Whether using programmed logic or a service, there are as many ways to validate information as there are types of information to be validated. Fixed lists 'idiot-proof' fields where manual entry is not the best approach due to a limited number of valid values. Automated audits against known good data sources are just a best practice.

The presence of invalid or inaccurate data can lead to several detrimental consequences, including: Misleading Insights and BI Results: Invalid or inaccurate data can distort patterns, trends, and anomalies in data visualizations and reports, leading to misinterpretations and flawed conclusions. This can hinder effective decision-making and undermine business strategies. Erroneous Calculations and Analyses: When data is invalid or inaccurate, calculations and analyses based on that data will also be flawed, producing misleading or incorrect results. Reduced Trust in Data-Driven Decisions: As the reliability of data is compromised by invalid or inaccurate values, the confidence in data-driven decision-making diminishes.

"Outdated or irrelevant data can cause obsolete, inaccurate, or irrelevant insights in your BI results, affecting your data timeliness and usefulness" Once again, if and when that happens, it's the BI machinery that is to blame. If you allow your BI machinery to have its output depend on "outdated or irrelevant data", then it's your BI inference processes that need revision. If data has become "outdated or irrelevant", they should be removed from the input sets of your BI machinery. Otherwise, you'll just get the "garbage in, garbage out" you deserve.

Setting timeliness standards by data type allow you to build refresh schedules for dynamic data that periodically check for updates to the information from available data sources. Depending on the nature of the data, it can be updated via automated query or periodic touchpoints with customers to update the information. The touchpoints, if handled well, can also serve to maintain positive engagement with customers over time and keep the relationship current.

In BI cases for the unstructured data; it depends how will work with it on BI tool like power BI or Qlik: extract key data to analyze or show it by specific mechanism like image displayer or key text cloud; the quality issues depend of how use it on BI Tools! For example; in the data extraction on image like voucher verification, you can extract the amount, operation number or sign in by consumer by the quality depend how recognize these element. To solve it, you can use several ML algorithms to recognize it and use the most popular value or calculate the average value.

Focusing the indexing of unstructured data on the facts relevant to the business can reduce the 'noise' of big data. It can be easy to 'boil the ocean' with big data due to excitement over all the things you can learn. Businesses have specific things that drive success, and those should be the focus of the analysis of their unstructured data. Figure out the questions you need answered and structure the indexing to find that information in the sea of data. After you have covered the basics, then worry about unseen insights of the past. Keep the lights on first, then see what neat bells and whistles you can find. Priority of function is important when it comes to big data analysis.

Data literacy matters. Many organizations have people who are so blindered in their silos that they don't understand the impacts of their actions on others, especially in the data space. Employees need to understand the holistic nature of data and the data lifecycle. Organizations that will succeed in the 21st Century will have employees who understand systems thinking and take ownership of their impacts on the data quality of the enterprise. Every employee who touches data should be a mindful data steward with that data. Chains are only as strong as their weakest link, and that holds especially true for data lifecycle chains.

Data should be supported by business rules as much as possible to support data consistency. When a metric or field lacks business rules, a situation can arise where two analysts arrive at differing results when trying to answer the same question. Business rules can help provide consistency to the data and clarity to analysts.

It affects a lot on the data model design consistency and quality when many-to-many relationships not designed well or not avoided at all