Applying Evidence To Decision-Making
“That may work in theory, but will it work in practice?”
“That may sometimes work in practice, but not always…what is the theory behind it?”
Theories can never be proven... only supported by current evidence. Intuition is a powerful tool people can use to address issues, but is prone to error. Even scientifically valid evidence can be inappropriate to apply in making decisions, since it might have been created in a different contxt.
Effective decision-making is facilitated by using the highest quality of evidence that is relevant to the issue. There are four principal sources of evidence (CEBMa.org):
Increasingly executive management is asking that people support their recommendations with all the high-quality evidence that is relevant to the issues. Doing that can ensure those recommendations are considered carefully and that decisions are appropriately informed by the knowledge behind them.?
The literature is often the primary source of practitioner knowledge, although the literature often contains unsubstantiated claims. Articles and books contain personal opinions and flawed research. There is also an embedded bias, since only successes tend to end up in print. Few people are eager to publish their failures, even though it can be a source of valuable intelligence for others. Reports on the success of a new program in a respected organization can be tempting to emulate, but for something to work in the same way across organizations the contexts must be very similar.
The academic literature is where most scientifically sound studies are published. Yet the way findings are reported may make it difficult for an untrained practitioner to understand and to apply. Academics are subject to strict guidelines, to ensure studies are valid, and as a result the papers include results that are produced using quantitative analysis methods that can require graduate level training to understand fully. But even the academic literature is subject to bias. Editors of journals rarely accept studies that did not support the hypotheses established prior to conducting the research. As with the practitioner literature this deprives the user of knowledge that might have been useful.?
Academic journals consist primarily of single studies, which can limit the strength of their findings. There is a way to combine multiple studies to increase sample size and strengthen the likelihood that the results are not unique to a single context. A study that does this is called a meta-analysis and will generally provide stronger evidence. Yet another type of analysis is called a systematic review, which contains relevant qualitative evidence as well.?
The knowledge gaps that are common between academics/researchers and practitioners can be bridged by an intermediary that interprets research studies in a manner that makes the conclusions accessible. World at Work (worldatwork.org) has partnered with the Center for Evidence-Based Management (CEBMa.org) to facilitate practitioner access to high quality research findings. A series of articles has been published by WAW (worldatwork.org) on topics useful to practitioners, such as what research tells us about financial rewards, which was the topic of the first article.??
What makes evidence relevant?
One of the limitations of applying research findings is that they must have been produced in a context that is similar to the context they will be applied in. What happened in a research study in a lab may not provide useful guidance if the field context differs. A commonly cited lab study concluded people would throw tennis balls at targets longer if they were not paid than if they were. Even though the study had a sound design it was only “internally valid,” which means the same results could only be expected if the conditions were the same. Yet for a research study to be “externally valid” it would have to meet the tests to determine it would be generalizable to other contexts. The conditions of that lab study (small sample of people involved in a pleasurable activity for a short period of time for small stakes) bore little similarity to a context where people performed unpleasant work for long periods to support themselves and their family. That study has little relevance in dramatically different contexts.
领英推荐
“What works is what fits… the current context” is a principle that should guide the selection of strategies and practices. The “good fit” requirement also applies to benchmarking practices with other organizations. What worked in Google may not work the same way in a financial institution. What worked in a financial institution may not work the same way in another financial institution. What worked in Google two years ago might not work the same way today in Google. Although applying rigorous relevance tests lessens the amount of research that would contribute to better decision-making attempting to incorporate irrelevant evidence is ill-advised.
The “fit to context” principle also applies to professional expertise. Experienced practitioners may have dealt with a wide variety of situations, producing knowledge that can guide decisions. Yet, as with benchmarking and applying research, that experience may not be a good fit to the current issue. The pandemic has dramatically increased the number of people working remotely for the first time. Unless a manager has had prior experience with managing remote work existing knowledge may not be of much help. Dealing with complaints such as “why do others get to continue to work remotely, and we have to once again face punishing commutes?” are difficult. If evidence suggests some types of work will be less effective when done remotely the organization may be justified in establishing different rules for different occupations. Yet skeptical employees may need compelling reasons and supporting the belief that location impacts productivity may be tricky.?
What Makes Evidence Usable?
In order to use evidence, it must be understood. Understanding how it was created and how it can be applied may require knowledge that a practitioner does not have. Data scientists are being used to do workforce analytics, to discover what the relationships are between strategies and outcomes. For example, selecting the right candidate to hire ideally involves the use of a model that results in the best decisions. Knowing what characteristics best predict success is the key. Statistical techniques used by data scientists can uncover correlations between factors such as education or experience and performance on the job. It may not be necessary for the practitioner who is applying the results to know how a technique like regression analysis works, but it is important to know what the results establish and what applying the results entails.?
Practitioners ideally work with data scientists to first identify the issues, and then to participate in applying parameters to how an analysis is done. It is important to distinguish between a correlation between two things and determining that one thing causes another. For example, there may be a correlation between current pay rates and gender, but since gender should have nothing to do with pay rates it is incumbent on the practitioner to rule out the use of that factor in developing a model for administering pay. The correlation can only establish that the relationship between pay and gender exists. That fact can alert the practitioner to do further analysis to ensure there is not a causal relationship.?
Conclusion
Using relevant high-quality evidence to inform decisions is a sound practice. It can support decisions that otherwise may face resistance based on personal judgment alone. Inertia exists when strategies and practices have been in place for long periods and persuading others that what has worked in the past is not likely to work in a very different future. The pandemic has eroded the credibility of past data used in workforce analytics, since the context has been dramatically altered for many organizations.
The use of one-to-one interviews as the primary selection tool is common, even though research has demonstrated it has little or no validity for predicting success. The fact that an invalid practice is being relied upon so widely is evidence that relevant and compelling evidence is being ignored. This is an example of how decisions could benefit from the appropriate use of evidence to inform them.
About the Author:?Robert Greene, PhD, is CEO at Reward $ystems, Inc., a Consulting Principal at Pontifex and a faculty member for DePaul University in their MSHR and MBA programs. Greene?speaks and teaches globally?on human resource management. His consulting practice is focused on helping organizations succeed through people. Greene has written 4 books and hundreds of articles about human resource management throughout his career.