What I learned from testing Products with Young Users? (Part 2/2)

My previous post covered a few lessons learnt on the field when testing EdTech products with young users. In this post, I hope to cover a few lessons learnt during the setup that precedes these field tests. As a PM doing user testing you'd find yourself spending time on setting up test environments. Setting up test environments and being explicit about the hypotheses and goals is critical for a good test.

a) Clarify your outcomes. Is it Findings or Results?

Humans are intrinsically wired to solve for a problem. This means that we try to solve before we understand. Ethnographic observation in order to empathize and understand is not a skill that a lot of people possess in the form of an instinctive skill - it is to be learnt and practiced. But any good product owner will tell you that - understanding user behavior as well as acting on findings from that understanding are both equally important. This becomes an even more important distinction to make when it comes to educational testing. That is, when you are in the field:

Are you observing how the students are using your product?

OR

Are you making sure that the students achieve results as they use your software?

At one of the products I was working on this became fuzzy. The Product team was clear about the outcomes. However, the other teams who were also occasionally in the classroom were very judgemental of our users - they were there for results. No one understood that it simply didn’t matter to our observations if the child was being a dunce or a rowdy in the class and that our job was to observe the class in its natural uncontrolled form. The team eventually got so sidetracked that a thread was created to discuss disciplining strategies for these classrooms.

Lesson Learnt: The mistake I made was not clarifying our intentions to our internal stakeholders. Being part of PM team doesn't just mean liaising with other teams for feature prioritizations. Sometimes you might have to bring them on board with your tests and ask them to be your observers too. Be explicit when you are testing that you are in that room to observe and that if a good result came out of the ‘experiments’ we did - well, that’s just a bonus.

b) Set the right expectations with your stakeholders at school.

Just like the ‘findings vs results’ understanding is clear internally, it has to be made explicit with the stakeholders as well. Tell them that while you and your team will try and do everything you can to improve the learning outcomes — there is simply no guarantee.As this is still an experiment, you are still learning along the way. Once you’ve made that clear and they still have their doors open for you, thank them profusely!

Most of the times the schools/principals will be okay with this honesty. They understand that no one can guarantee results (and that includes their teachers) as long as you do not have a negative effect on the marks of their children consistently, they will let you continue.

Lesson: Do not over sell your proposition if you are there for testing - be it with schools or parents. Free + testing is a concept that Indian parents and schools understand.

c) A/B experimentation can be tricky in a 'perfect environment' like schools

First off, the schools did not like the word ‘experiment.’ It just makes them jittery. So, of course one needs to instil confidence by telling the school the problem they are addressing, their approach, the people they have on board trying to solve the problem etc.

Secondly, the researcher in you will be tempted to take advantage of the fact that every standard has two randomly separated groups who are in the same environment[for the non-researchers among us — we mean a/b treatment]. However, be prepared for the school to say ‘No. You may not.’ As someone on my team said, it’s their way of saying “We realize our students are the guinea pigs but we’d like you to treat all our guinea pigs the same.” If you think about what the school authorities are saying from the student point of view — it is extremely fair. They should be able to produce the same result for all their students. At the end of the day, they are responsible for all of their students equally.

Good news is, that this shouldn’t be a deal breaker. You can work around this with a few permutations, combinations and a little bit of creativity.

d) Be vary of the passionate ‘i want to do it all’s on your team.

Education is a very emotional topic. We all have ideas on how the system needs to change. But we all may not share the same passion when it comes down to rolling up our sleeves and fixing it. Those of us who do decide to roll our sleeves are extremely passionate about it. Passion can sometimes be blinding.

In my opinion, it’s important to stay level-headed about the Theory of Change they embrace. There may not be bad ideas but there is such a thing as too many ideas. For example, with the schools that we were working, there was very little we could do to help very weak students who had been passed on through standards and reached Grade 10 while they really were at Grade 3 level. In addition, we had only about 5 hours each week to work with these students. It was important to understand our limitations. However, the passionate folks on the team felt that we could do a lot more. As the person in charge, I was responsible for the tasks they took on and I had to be the one to draw the line. I heard them grunt when I asked them to not overcrowd their plate. I was okay with being the bad person who told them that they can’t do it all and that there was only so much we could do for these kids. I realized they didn’t appreciate this side of me but I figured I could live with that.

Lesson: The only way to pick which idea to work on is to look at numbers. Data holds the answer to the question "how are our resources best utilized."

e) Teachers can be a blessing or a bane. Principal is your best bet.

That is, teachers are either going to be your ‘besty’ or ‘a pain in the neck.’ I've seen all kinds - teachers who are thankful for a tech intervention - these are also the ones who will make sure you get coffee (yay!) and teachers who either hate you because your software threatens them or are plain relived because they can spend that hour in the staff room.

The only reason we were able to stick around and get things done in any of these cases was because we built a great relationship with the principal. What does it mean to build a relationship, you ask? Listen to them, keep them updated and ask them if there is a particular problem they are trying to solve and see if you can solve it for them. Certainly, don’t go out of your way but asking by itself helps. Bonus points if you can at least fix it to an extent.

Of course, apart from all of the above there are the a few obvious ones such as setting up systems or analytics that collect data in the backend - data such as time taken to finish a task, to take a test, times a video was replayed/skipped backwards+forwards and other data that matter for your product. Your OMTM may need you to collect a bunch of different behavioral and usage data so keep that in mind too.

Like I mentioned in part 1, this is a high level overview. Feel free to reach out in case of questions.