How do you choose the right confidence level for your margin of error calculation?

In my experience, this is the sort of thing that can never be communicated enough. The number itself is very important, but so too is the methodology by which you find it. Keep careful, updated documentation detailing how you reached your conclusions. This won't only help if you are ever under audit, but is also beneficial for business continuity. For most data sets, I think that you would first need to know how often data was gathered erroneously, or how often it may need to be corrected after ETL, vis-a-vis the aggregate amount collected. That is a problem which haunts data engineers more than analysts, but it has repercussions for both. From there, you can collaborate with the audience about what an acceptable level may be.

Choosing the right confidence level for your margin of error calculation means balancing accuracy with practicality. A confidence level, like 90%, 95%, or 99%, shows the probability that the true value falls within the margin of error. Higher confidence levels give you more certainty but also increase the margin of error, making your insights less precise. For example, if you’re deciding on a major investment, you’d want a 99% confidence level to minimize risk. But for a quick exploratory analysis, 90% might be enough. Always consider the importance of the decision and the potential impact of being wrong to choose the right confidence level.

Risk is everywhere, and like opportunity costs it can never be avoided. The trick is in knowing how to manage risks in an acceptable fashion. Risk can also be a nebulous topic which defies easy quantification; as with the context listed above, you would need to articulate where there risks are (data gathering, how much corrective action is needed against an aggregate population, etc.) before the confidence level really starts to make sense. In my experience, that is an ever-evolving topic of conversation - situations change, and so will the confidence level in a study's efficacy.

In my experience, continuous review most often takes shape with scheduled or ad hoc audits for data gathering or transformation procedures. I believe that documentation can never be too extensive, but it would also help to have more direct (less technical) versions available for non-technical audiences. That later part doesn't have to detail systems or coding, but a summary of steps taken when getting from raw data to a final product. Screen shots may be helpful for procedural instructions, but may be a little less so here.

Analyze historical data to determine how outliers influence the confidence level on your decision making process. If there is a presence of a fat tailed distribution (greater kurtosis), then higher confidence level may be more appropriate.

Consider the implications of the confidence level on your decision-making process. Higher confidence levels might require more resources and time for data collection and analysis, but they provide more robust and reliable insights, which are critical for strategic decisions.