The Cognitive Bias of Oversimplification

Oversimplifying probabilities to make them easier to think about is a well-known bias in research circles. In looking over past research built on conclusions made from prior studies that focus solely upon p values, I am struck by how simple the observations become. Often, we overlook effect sizes when p is significant because we found a significant relationship. But how meaningful is that relationship when the effect size is small? In many studies of complex psychological constructs like juvenile zero empathy and fire setting behavior, we might tend to use one or two variables in our statistical modeling. This oversimplifies the issue in my mind. 

For more information on cognitive biases see: https://commons.wikimedia.org/wiki/File:Cognitive_Bias_Codex_-_180%2B_biases,_designed_by_John_Manoogian_III_(jm3).jpg