Being and Becoming Responsible Consumers of Educational Research

*This is an excerpt from my book, ‘Understanding the Science of Reading: Context Matters’.? It will be published by Guildford sometime in 2025.

?

Podcast Version of this chapter excerpt:

Starting Assumptions

Those who have never taught don’t realize that teaching is hard and demanding work.? According to EducationWeek, teachers work about 54 hours a week (Hardison, 2022).? That’s over 10 hours a day if you worked just five days a week.? Besides the actual job of teaching, teachers must plan effective learning experiences, respond to individual students, evaluate activities and assignments, record students’ progress, communicate with parents, fill out forms and paperwork, communicate with colleagues, address social and emotional issues within the classroom, and try to live a life outside the classroom.?

The point is, that there’s not a lot of time or cognitive space left to read and analyze research articles.? And if the choice is to read a research article or to plan for tomorrow’s lesson, most are going to do the latter.? And even if there was time and space, most teachers are not very good at reading, interpreting, and evaluating research articles.? This is not to be demeaning.? But unless you have experience, understanding, and time, you most likely won’t know how to do this very effectively.? I am not very bright.? That’s a given.? The only reason I know how to do this is that I’ve been doing it for over 30 years.? It’s part of my job.?

And even if teachers had time and cognitive space, where would they get research studies to read and evaluate?? They could subscribe to a professional journal, such as Reading Research Quarterly, which is written for academics?? They could also get a practitioner-oriented journal such as The Reading Teacher.? But this still leaves them with a very narrow scope of research.? They would need to have access to a database of journal articles like the one most professors have at their university.? If state legislatures and school districts were serious about professional development and “research-based practice”, they ensure that teachers had access to this kind of database.? Every school district would be mandated to have a subscription.? This could be part of an effective and economic tool for professional development.? This would also provide teachers with the tools to evaluate claims of “research-based strategies” and “evidence-based practices”.?

But that won’t happen, because disempowered teachers are easier to control and manipulate.? The message to teachers is clear: “Why do the thinking when others will do it for you?? You just go back to your little classrooms and love kids.? Get your thank-you notes and hugs and do what we tell you to do.”

“If you knew things, you’d be able to make your own research-based decisions, and we simply can’t have that.”

“If teachers knew things, there’d be little need for expensive programs, profit-based professional development, and consumable workbooks.? That would upset the educational economy.? And we simply can’t have that!”?

Reading About Research

There’s a difference between (a) reading research related to reading instruction and (b) reading what others have said about research related to reading instruction.? It’s important to know the difference.? When you read research articles, you get to evaluate the methodology and interpret that data.? When you read what somebody else has written about research, you must trust that their evaluation is fair, and their interpretation of the data is accurate.?? You are reliant on the relative clarity of their lens.

So, far too often you’re left with people like me whose job it is to continually read and evaluate research.? But this chapter is written so that you will be able to do this.? It’s written to make me obsolete.

The Process

To critically read research you can’t expect to read it through like an article in People magazine.? It takes a bit of time, especially at first.? Articles that are poorly written, under-described, over-described, or nonsensical take more time.? This section will show you how to read and evaluate quantitative research.?

Using the Gold Standard

According to the SoR research standard described in an earlier chapter, only controlled experimental or quasi-experimental research that has been conducted in classroom settings can be used to determine causal relationships.? Hence, this is said to be the only kind of research that should be used as the basis for making decisions related to reading instruction.? While I disagree strongly with this premise, these are the rules to their game.? Hence, I will limit this chapter to reading and evaluating only this kind of research with two caveats: First, real science doesn’t exclude research or data because it’s inconvenient (Pressley, Duke, & Boling, 2004)).? Second, there are a variety of other types of research methods including qualitative and descriptive research.? When these are included along with controlled experimental research, a much clearer and more accurate picture of reality is created.?

Basic SoR definition = Using strategies and practices that have shown to be effective using controlled experimental or quasi-experimental research that has been conducted in classroom settings.

The section below provides a general guideline for reading and analyzing research.?

Abstract

Start with the abstract.? This provides a basic overview of the study including the research questions and results.? It helps you to make sense of the information you will be reading.? But be wary of abstracts that are difficult to understand.? Nebulous or poorly worded abstracts can be used to cover up bad research.? And if the abstract doesn’t make sense, it’s often because it’s nonsense.? Research has “proven” that nonsense does not make sense (Johnson, 2025).?

Review of the Literature

The literature review is found between the Abstract and the Methods section.? Once you understand the basic elements of the research being conducted, the review of the literature often makes much more sense.? Thus, I recommend you go right to the research questions and read this section last.?

Review of the literature.? This provides background information and research related to the study being conducted.? It also embeds the research question within a theoretical context.? There are two things to be aware of when reading the literature review: First, while the review of the literature is used to provide a theoretical grounding for the current study, it is not part of the study.? You shouldn’t pull an idea from the literature review and cite the author of the current research.? That gives the impression that the idea being pulled is supported by the current research study when that’s not the case.? This is often done.?

Second, literature reviews should not obscure the basic elements of the study.? Be wary of extensive literature reviews.? They’re sometimes used to justify bad research or to divert attention from the basic elements of the study.? A good literature review will lead you directly to the research questions.

Research question or questions.? There can be a single research question, however, in my experience, there is usually more than one.? Thus, the plural form is used here.? The research questions are where the real analysis begins.? These are usually found right before the methods section.? The research questions should be easy to identify and understand.? Be wary of research studies in which the research questions are nebulous or hard to identify.?

Research is not simply collecting a bunch of data to see what you come up with.? Research is not collecting data and finding research questions to explain your data.? Research of any kind is planned, purposeful, and based on clear research questions.? Everything that follows the research questions, (and nothing else), should be used in support of answering these questions.?

Methods and Procedures

The next four elements are part of the methods section.? The standard for a well-written research article is repeatability.? Thus, this section should be written in a way that would enable the reader to repeat the experiment.? This means that concise and precise academic language is used.? Watch out for the old literature-review-in-the-methods-section trick.?? This is where mini-literature reviews are used to describe or justify the elements found here.? While it may be necessary to sometimes justify elements here, these written justifications should be minimal.

Subjects or participants.? Subjects or participants are the people involved in this study.? These are the ones who are observed or given some sort of instructional treatment.? Questions to consider: Were there enough of them to generalize to larger populations?? Did the population that was being studied represent the populations to which generalizations were made?? What other variables might impact the participants?? And importantly, if one group is being compared to another group, are the groups equal or equivalent before the study begins?

Treatment, criteria, or conditions.? For quantitative studies, this is the independent variable.? This is the approach, treatment, or element that the researcher was trying to isolate to make it independent of all other variables.? It’s called controlled experimental research because the researcher is trying to control all the variables except the one being observed.? If this is done, and if there is a change or difference, and if that change or difference is statistically significant, we can say with some confidence that the independent variable had a causal effect on the dependent variable.? That is, we can say that with this population, this one thing seemed to cause the other thing to occur.

Measures or instruments.? The measure or instrument describes what was used to collect the data.?? The data collected is the dependent variable.? The data is dependent on the independent variable.? Included here is how, when, and how often data were collected.

Design and analysis.? This is the statistical analysis used to assess the data collected.? While it is optimal to have a strong statistical background, it is not necessary.? We should be able to rely on the peer-reviewers to ensure that there are no statistical anomalies or funny business going on.

Results?

This is the data that was collected.? It’s often presented in a table.? Don’t let the limitations of statistics keep you away from this.? All you need to know is the significance of the differences (p > .05), or the strength of the correlation (r = .67).? Below is a very basic understanding.

Statistical significance.? Statistically significant differences between groups mean the differences are greater than could occur by chance.? For example, if one group scored 240 on NAEP tests and another group scored 238, this difference is relevant only if this difference is statistically significant.? That is, the difference is greater than could be attributed to chance.? If this difference is p > .05, this means that we could be certain of the probability that one thing caused another thing to occur or that the difference was greater than could occur by chance in 95% of the cases.? Only in 5% of the cases would we be uncertain.? This is usually the basic level used for statistical significance.? If the difference is statistically significant a p >.1, this is even better.? We could be certain of the probability in 99% of the cases.? Only in 1% would we be uncertain.? Statistics are the mathematical algorithms used to determine these things.

Correlation.? Correlation describes the strength of a relationship between two things.? They are co-related.? These range between -1 and +1.? -1 is a perfect negative correlation, meaning that when one score goes up, the other goes down.? +1 is a perfect positive correlation, meaning that when one score goes up, the other score also goes up.??

1.0 to 0.8: Very strong positive correlation.

0.8 to 0.6: Strong positive correlation.

0.6 to 0.4: Moderate positive correlation.

0.4 to 0.2: Weak positive correlation.

0.2 to 0.0: Very weak positive correlation.

0.0: No correlation.

0.0 to -0.2: Very weak negative correlation.

Correlation does not infer causation.? We can’t say one thing caused the other to occur.? But we can infer the strength of the relationship.? However, we don’t know which thing causes the thing other to occur, and we don’t know if there were other variables present that hadn’t been considered.

Conclusions

Conclusions are logical inferences as to what the data might mean.? Five things to consider: First, conclusions should only be made based on the data collected.? Doing research does not give you license to make comments on things unrelated to the data collected.? If a conclusion is not based on the data collected, it is not a research-based conclusion.? It’s just somebody saying things.? Be wary of research that reports results and then goes off in all sorts of directions making speculations.?

Second, be wary of data unrelated to the research question and conclusions that go beyond the research question.? These may be interesting things, and they may be the basis of future research, but from a research perspective, the only thing you can say with any amount of certainty is, “It may be that …”

Third, research in the social and behavioral sciences does not prove things.? The science of reading is not a settled science.? In reading research published in academic journals you will rarely (if ever) see the phrase, “This proves that …”? It may suggest, it may indicate, it may lead one to conclude, and it may even demonstrate, but it never proves.?

Fourth, research is a snapshot in time.? It’s always described in the past tense because it describedsa specific instance with a specific population, at a certain time, when certain conditions were present.? Longitudinal research may provide multiple snapshots over multiple times.? This provides a more accurate view of the thing being studied, but each of these snapshots is stuck in time.

And fifth, conclusions are interpretations of the data.? Data can be interpreted differently based on one’s paradigm or theoretical framework.? For example, in the Ehri et. al study (2007), students who practiced reading using texts that they could read with 97% word reading accuracy scored higher on measures of comprehension.? If you adhered to a bottom-up theory of reading, you’d see that data and say, “Aha!? Word read accuracy leads to better comprehension.? Children need to be able to sound out words.” ?If you adhered to a meaning-based theory of reading, you’d see the data and say, “Aha!? It is important that children practice reading with text they have selected and are at their independent level or below. Children need to be able to create meaning with text.”? Same data, different interpretations.

Limitations

There is no such thing as the perfect study.? There’s no such thing as a totally objective researcher.? There is subjectivity in the questions we ask, the types of data we collect, what we use to collect the data, and how the data are interpreted.? A good research article will end with a short description describing the limitations of the study.

References

Ehri, L.C., Dreyer, L.G., Flugman, B., & Gross, A. (2007). Reading rescue: An effective tutoring intervention model for language-minority students who are struggling readers in first grade.? American Education Research Journal, 44, 414-449.

Hardison, H., (2022),? How teachers spend their time: A breakdown.? EducationWeek, April 19, 2022.

Johnson, A. (2012). A short guide to action research (4th ed.). Boston, MA: Pearson.

Johnson, A. (2025). Don’t be silly, this is a ridiculous statement.? Ridiculous Statements Quarterly, 37, 41-72.

Leedy, P.D., & Ormrod, J.E. (2010).? Practical research: Planning and design (9th ed.). Boston: Pearson

Pressely, M., Duke, N., & Boling, E. (2004). The educational science and scientifically based instruction we need: Lessons from reading research and policymaking. Harvard Educational Review, 74, 30-62

?