How do you test and refine your data analysis models and assumptions?

This concept map provides more detail on this approach: https://cmapspublic3.ihmc.us/rid=1M8NGW12Q-1PMR9P1-1G61/Design%20of%20Experiments.cmap And don't forget that we may want to take deductive, inductive or abductive reasoning approaches: https://cmapspublic3.ihmc.us/rid=1M8NGW12N-WNY424-1G2S/Reasoning.cmap

But beware of falling into the trap that correlation implies causation. Application of qualitative analysis techniques ( eg surveys, focus groups etc) may yield more information about causation than using statistical analysis alone.

Testing and refining data analysis models and assumptions is like baking a cake - you gotta get it just right or it'll be an epic fail. 1. You need to know what the heck you're aiming for! It's important to define what success means for your project. 2. You gotta make sure you're not making any crazy assumptions. 3. Don't put all your eggs in one basket - use multiple data sets to make sure your analysis holds up. 4. Cross-validation is your new best friend. Split your data into two parts and use one half to train your model and the other half to test it. 5. Teamwork makes the dream work! So, there you have it - some tips to help you bake your data analysis cake to perfection.

One important thing to note is the bias variance trade-off. When using models to try to predict a relationship, we run into a couple of problems: either we cannot accurately predict the true value (bias), or the variance is too wide (it is too focused on training data and not as generalizable on new/tested data). When conducting data analysis, we usually have to pick a reasonable amount of both, instead of totally eliminating both, hence the trade-off. This is related to the idea of overfitting (when a model is so accurate that it picks up random noise along the way) and underfitting (the model can’t accurately pick up the relationship). We want to make sure our model is appropriately capturing the relationship of interest without problems.

This concept map shows a range of approaches that can be taken to judge the fitness for purpose of both qualitative and quantitative analysis approaches: https://cmapspublic3.ihmc.us/rid=1M8NGW12P-T36QT2-1G3S/TM%26M%20Fitness%20for%20purpose.cmap Again, don't ignore the fact that many analysis problems include a human/ social behavioural aspect. Don't just rely on statistical approaches.

Understanding your customer is key. Early on, ask your customer what questions she wants addressing, in what timescale and at what level of accuracy. Try to understand why the customer is asking these questions and whether there are other stakeholder or 'political'perspectives that need to be taken into account.

The C-Suite frequently asked me to present financial, operational, and performance reviews. There are several types of data consumers. My COO boss liked raw data as a spreadsheet to pivot. He disliked data visualizations, considering them imprecise. He also preferred long-winded presentations exploring the data in depth. The busy CEO chose simple visualizations. He didn't respond to the raw data. He preferred a Powerpoint presentation, three pages or less. As mentioned above, the key is knowing your customer. Data visualization tools allow analysts to move quickly between graphs and charts with a simple drill-down. They often suggest terrible layouts and colors. Be sure to consider design elements and formats before presenting.

It is important to keep biases in mind when gathering and analyzing data. Learning the different types of bias and keeping them in mind throughout the process can help to prevent data bias from affecting your assumptions. For example, medical researchers often use double-blind studies to help eliminate bias.

In addition to choosing appropriate data analysis methods and techniques that suit the data type, size, and quality, you need to consider factors such as the level of measurement and scale of variables, the assumptions underlying each method, and the potential for bias or confounding variables. For instance, if your regression analysis shows a significant positive relationship between customer satisfaction and loyalty, you need to consider whether this relationship is causal or correlational, and whether there may be other factors that influence loyalty, such as price, quality, or competition.