How do you compare survey results across different groups?

Steps for defining groups in research: Clear Definition: Start by clearly defining the groups you want to compare. These groups should be based on specific criteria, such as demographics, preferences, behaviors, or attitudes. Rationale: Have a clear rationale for why you're choosing these particular groups. What research question or hypothesis are you trying to address with these group comparisons? Mutually Exclusive: Ensure that the groups you define are mutually exclusive. This means that each respondent should belong to only one group. There should be no overlap between the groups.

I think it is good practice to keep groups relatively parsimonious. Age, for example, doesn't need to be broken down in 5 year blocks unless there is a specific research purpose to do so, so a larger "bin" might be best. Or keep it continuous and plonk it into a regression model. It's also a good idea to keep in mind the sample size within the smaller groups. If conducting an analysis using race as a group, for example, certain racial groups might be dramatically underrepresented in the data so they might need to be relegated to an "other" category. If the subgroup size is too small, one cannot draw meaningful conclusions from any comparison.

I've seen countless colleagues stumble over this seemingly simple step. Remember, the key is to understand your data's personality and dress it for success. Don't force a square peg in a round hole by using the wrong method. Think beyond the basics. Are you missing out on hidden relationships by relying on mere percentages?

Steps for choosing a comparison method in research: Type of Data: Consider the type of data you have. Is it categorical or numerical? Different data types may require different comparison methods. Research Objectives: Define your research objectives and what you aim to learn or discover through the comparisons. Your choice of comparison method should align with these objectives. Level of Detail: Consider the level of detail you want to achieve in your comparisons. Some methods provide a broad overview, while others offer more detailed insights. Data Visualization: Think about how you'll present your comparisons. Will you use tables, charts, graphs, or other visual aids to effectively communicate the differences between groups?

Steps for checking your sample size and distribution in research comparisons in a more concise format: Sample Size: Ensure each group has at least 30 respondents for statistical validity. Representativeness: Your sample should mirror the population you're studying. Distribution: Aim for an even distribution of respondents across groups. Proportionality: Ensure group proportions align with real-world criteria. Statistical Power: Adequate sample sizes are crucial for meaningful comparisons and results you can trust.

This is really the most important element to successfully comparing results. Knowing the confidence levels based on the various error rates like sample error and non-response error. Also understanding if each result was weighted and what for.

In addition to the significance test, it might also be helpful to conduct an analysis of effect size. When communicating statistical significance to non-statisticians or individuals unfamiliar with the lingo, "significant effect" can often be interpreted to be larger than it is in the "real world". In fact, in many cases, "statistically significance" means a very tiny effect, so one should look for additional ways to contextualize a statistically significant result and communicate it in a way that makes the most sense to the audience.

Here is where you tell the story of your results. Be sure you have a solid understanding of what that story is. Once you do, practice telling it without charts and visuals. If you can tell the story simply and concisely without visuals, then when you present with visuals, it will be more impactful.