Don’t be Duped! Using Data to Soup up EdTech Decision-Making

by Kimberly McClain, EdD

Let’s be honest - buying educational technology without data is like grocery shopping while hungry. You end up with a cart full of random stuff, some of which might work, but most of which will gather dust (looking at you, 3D printer). That’s where data-driven decision-making comes in - our not-so-secret weapon against impulse buying and shiny object syndrome.?

Collect the Right Data: Not Just a Vibe Check

Before we even think about signing a purchase order for the latest and greatest EdTech tool, we need to gather data. And sadly, “It looked cool at a conference” does not count as data.?

We’re talking about real, measurable data like:

• Student engagement metrics - are students actually using the tech, or does it become another icon on their device?
• Training opportunities - Does the technology come with built-in training, so teachers are more likely to adopt the technology??
• Teacher adoption rates - Are educators incorporating it into their lessons, or is it collecting virtual cobwebs?
• Learning outcomes - Does it improve comprehension, retention, or critical thinking, or does it just look impressive on a grant application?

Analyze Like a Pro (Or at Least Like Someone Who Read a Few Articles on Analytics)

Data isn’t like a horoscope. Instead of glancing at it and hoping for the best we have to analyze it - statistical trends, comparative studies, and even some good old-fashioned teacher and student feedback.?

Hershkovitz et al. (2024) emphasize that data alone isn’t enough; instructors need clear, user-friendly dashboards with information compiled from EdTech vendors, teachers, students and other sources instead of overwhelming charts to make data actionable. These tools provide both visuals and numbers to guide decisions. For instance, if an app shows students use it for only five minutes before switching to games, it may not be effective.

Compare Before You Commit (No Swiping Right on the First Option)

Remember, just because one district swears by a tool doesn’t mean it’s the right fit for your schools. Take a page from counterfactual analysis (Fernández-Loría et al., 2022), which essentially asks, “What if we had chosen differently?” (Fernández-Loría et al., 2022, Explaining Data-Driven Decisions Made by AI Systems: The Counterfactual Approach). Before you make the final call, consider:

• Are there tools that do the same thing but better, or for less money?
• Have we checked user reviews from actual educators, not just EdTech marketing teams?
• Will this integrate with our existing tech, or are we signing up for a compatibility nightmare??

Pilot Programs – Because Nobody Wants Buyer’s Remorse

Would you buy a car without a test drive? Would you commit to a lifetime supply of mystery food without a taste test? The same applies to EdTech. Before rolling out new technology across an entire district, run a pilot program.?

• Select a diverse group of teachers and students to test the tech.
• Gather real-time feedback on usability and impact.
• Tweak or ditch it based on actual classroom experiences.

Using this data to drive decisions is key. According to a study on data-driven decisions in higher education (Usher, et al, 2023), instructors who had access to real-time analytics could better adjust their strategies, ensuring that tools were actually beneficial instead of just trendy.?

Review, Adjust, and Repeat: Because Tech Moves Faster Than a Middle School Gossip Chain

The beauty (and curse) of technology is that it changes constantly. What worked in 2023 might be obsolete by 2025. That’s why we need ongoing evaluations of our EdTech investments. Learning analytics tools can help by tracking student progress and engagement, allowing us to see if our shiny new purchase is still pulling its weight.

Final Thoughts: Making Smart Choices and Avoiding EdTech FOMO

At the end of the day, data-driven decision-making saves time, money, and the frustration of watching expensive tools go unused. It helps us choose technology that actually improves learning outcomes instead of just making our schools look high-tech.

So, next time a vendor tries to sell us the "next big thing in education," we’ll smile, pull out our data, and make a data-driven decision.