Here’s how you can measure interaction effects when running multiple tests:
?? Check the chance of possible interactions with Sample Ratio Mismatch (SRM) Checker from Lukas Vermeer.
?? Segment the data in the same way you would segment traffic sources or device categories and check the outcomes for cross-sectionality.
?? Adjust for multiple tests with ?idák correction.
?? If the number of tested interactions is in the hundreds or more, consider using the Benjamini-Hochberg-Yekutieli False Discovery Rate adjustment—Geogi Georgiev.
Do you now immediately need to learn about the corrections and adjustments?
Not necessarily.
"Generally, these are rigorous ways to address the interaction effects. But also just knowing these effects do hurt detectability level (and if you and your team are ok with this tradeoff), then you take the hit on confidence and move forward”—Ben Labay, Managing Director of Speero.
Congrats, in my experience, only the very best panels are good at In-Home Use Testing.