How do you know if the intervention was successful? On developing interventions and measures.

1             Introduction

Interventions are an essential part of special education. Often the goal of interventions in the school context is to improve the achievements of the students within a specific curriculum-related subject. However, several interventions also aim to affect the cognitive processes of the whole group or certain individual learners. It may be that when implementing interventions based on e.g., mathematics or reading abilities, planning the intervention is more straightforward than when the intervention is not based on any school-taught topic. In the latter case, the conventions of social sciences and psychology, such as constructing and operationalizing ideas (Cozby & Bates, 2015, p. 74) and measuring latent variables are useful.

Interventions aim to a particular outcome, that should be defined before the development process begins: What is it that one would like to achieve with the intervention in question? In what areas will there be changes? What kind of changes? It may, e.g., be that the teacher would like her students to develop their emotional competencies or that the teacher would like to implement some tool used in positive pedagogy to increase the level of self-efficacy and happiness of her student's. And, as in all areas of science, the inter- ventions should be based on existing knowledge and theories.

In addition to developing a new intervention or using an existing one, measuring is vital in implementing the planned procedure. Without pre- and post-tests, there is no way of knowing whether the intervention executed was efficient. Also, for the results to be valid one must use measures that are reliable and valid. 

In this essay, I will briefly describe the process of developing an intervention and discuss the development process of a reliable and valid measure intended to gauge a latent variable. I will write briefly on the desired psychometric properties of such measure and reporting the properties. In addition, I will take a stand on the replication crisis that among many other fields of science also affects the field of education. Finally, I will discuss the current state of teaching research methods to future scientists.

2             Developing and using measures in interventions

The development processes of interventions and measures for assessing the effectiveness of such interventions are dynamic and can be conducted in parallel. In this way the processes facilitate one another, and it may result to the construct of interest becoming clearer. 

Validity of the development process in general, and reliability and validity of the used measure are the cornerstones of both science and evidence-based education practices, and should therefore be taken seriously. It is wise to utilize the conventions of psychology when creating these kinds of tools. 

Briefly on developing interventions 

When developing interventions that aim to alter people’s behavior, the basis of the pro- cess should be some of the theories of behavior change (Barker et al., 2016, p. 90), otherwise it may be that the results of the intervention may be difficult to replicate. For this, it may be worth a while to familiarize with the COM-B behavior model and behavior change wheel (BCW), that together make it easier for the teacher or the researcher to begin with the development process (Barker et al., 2016, p. 90). COM-B and BCW used together to make it more probable that the developed intervention is of adequate quality (Barker et al., 2016, p. 90). Also, when using a structured way of developing an interven- tion, creating a measure for assessing it may be simpler. 

In COM-B, the changes in behavior are seen to require three elements: capability, moti- vation, and opportunity, that together affect the behavior in question (Barker et al., 2016, p. 91). The development process consists of four phases: in the first phase, the problem should be defined in behavioral language; in the second phase, a target behavior that could address the problem in question should be selected; in the third phase, a clear description of who, what, where, when, how often and with whom, should be formulated; and in the fourth phase one should utilize the COM-B to analyzing whether the target population has the qualities required (Barker et al., 2016, p. 91-92). 

If the development of an intervention is conducted in a structured way and is based on a validated theory, it is more probable that the intervention results can be interpreted reliably (Barker et al., 2016, p. 90).

Developing reliable and valid measures 

It is essential that a measure that takes a stand on individual development or group differ- ences, or a one that is used in guiding policy decisions or investments is reliable and valid. Therefore, the development process of such measure should be transparent and of high quality. Also, the replication crisis that broke out around 2010, has had a major effect not only on the quality of research work in general (Zwaan et al., 2018, p. 2-3) but also on developing measures that are a vital part of most research in any field of science. 

As mentioned in the introduction, developing measures that assess a certain latent variable (a construct that cannot be measured directly as would be the height and weight of a person) is not as straightforward as developing ones for curriculum-based areas of interest. As Flake et al. (2017, p. 370) state it, the first phase of studying a latent variable is to identify the construct in question; second, to define it, and third, to formulate a theory concerning the construct. In their article, Flake et al. (2017, p. 371) divide the process of studying such construct to three separate phases: substantive, structural, and external. The outburst of the replication crisis has shown to both the science community and the public that often the first two phases of this process are neglected or conducted in an inadequate way (Flake et al., 2017, p. 371). In the development process both the qualitative, substantive phase and the quantitative, structural phase are equally important, and only after they have been executed with due care is it possible to proceed to the last phase (Flake et al., 2017, p. 371). 

3             Is there a problem in teaching research methods?

It may be that the low-quality state of researching latent variables is due to the way the research methods are being taught. Are the qualitative and quantitative processes and the methods that should be used described to the students adequately and correctly during their studies? Do the students get an opportunity to witness and practice how a top-quality research is conducted? It seems that currently the answer is no, which, according to Cas- sidy et al. (2019, p. 233-234) is a problem that begins from the introductory textbooks of psychology and continues when contemporary teachers pass their faulty conceptions on to their students. Maybe then it is no wonder that the replication crisis took place. It would be of the essence to teach proper research methods to students in all fields of science, not only the ones studying psychology.

An example of the current state of faulty ideas concerning the quantitative methods is the fact that many of the current researchers only rely on Cronbach’s alpha coefficient (Flake et al., 2017, p. 375) when reporting the psychometric properties of their study and the measure that was used. It is obvious that in many of these cases the researchers do not realize that the alpha is not suitable (Sijtsma, 2009, p. 107), maybe because they do not understand what the alpha is. It is no doubt the best-known and most used internal consistency coefficient – and it has also been used in e.g. trying to demonstrate the unidimensionality of the construct (Sijtsma, 2009, p. 115). 

When trying to measure a latent variable, it would be particularly important to conduct an exploratory and thereafter a confirmatory factor analysis to gain knowledge of the construct validity of the measure and also to report, among other psychometrics, the test-retest reliability and, if applicable, the interrater reliability. It would also be better to report internal consistency using e.g., McDonald’s Omega if the prerequisites for using the alpha are not met (Flake et al., 2017, p. 375), or to use one of the lambdas (Sijtsma, 2009, p. 112). 

I conclude that the issue concerning the quantitative methods is something that can be solved by developing and implementing adequate teaching methods and not to rely on unscientific but popular heuristics (such as ‘a result is significant if the p-value is lower than 0.05’ or that ‘the internal consistency is at adequate level if it exceeds 0.7’) in the teaching. The students should be guided to use the quantitative tools so that they understand the mathematics based on which the method in question was developed. It is also of the essence that every researcher understands that defining an adequate significance level and other reliable enough indicators and thereafter interpreting the results depends on the context.

4             Conclusions

In developing interventions, creating measures, and conducting research, high-quality standards should be met. For an intervention to be efficient and accurately targeted, it should be constructed using a structured, theory-based method. And for a measure to be reliable and valid, it too should be developed using the best possible conventions, emphasizing construct and other types of validity and reliability. In developing the measure and in interpreting the intervention results, one should concentrate on appropriate psychometric properties and the context of the project.

For the replication crisis to have a positive influence on the quality of science, the teach- ing of research methods should become high-class and evidence- and theory-based. The students should not be taught any heuristics concerning quantitative methods but instead they should learn how to correctly utilize the tools available, with an understanding of their applicability.

Bibliography

Barker, F., Atkins, L., & de Lusignan, S. (2016). Applying the COM-B behavior model and behavior change wheel to develop an intervention to improve hearing-aid use in adult auditory rehabilitation. International journal of audiology, 55(sup3), S90- S98.

Cassidy, S. A., Dimova, R., Giguère, B., Spence, J. R. & Stanley, D. J. (2019). Failing grade: 89% of introduction-to-psychology textbooks that define or explain statis- tical significance do so incorrectly. Advances in Methods and Practices in Psy- chological Science.

Cozby, P.C. & Bates, S.C. (2015). Methods in Behavioral Research. New York: McGraw-Hill Education.

Flake, J. K., Pek, J., & Hehman, E. (2017). Construct validation in social and personality research: Current practice and recommendations. Social Psychological and Person- ality Science, 8(4), 370-378.

Sijtsma, K. (2009). On the use, the misuse, and the very limited usefulness of Cronbach’s alpha. Psychometrika, 74(1), 107.

Zwaan, R. A., Etz, A., Lucas, R. E., & Donnellan, M. B. (2018). Making replication mainstream. Behavioral and Brain Sciences, 41.