Guess how much I love you? Thoughts on student engagement

Sam McBratney (1) introduced us to the challenge of measuring something which is hard to get your hands around, and the same can be said of student engagement.

Student engagement has become a mainstay of university structures over the last few decades, to the level that many student services departments have renamed themselves just to draw attention to it. Supporting that transition, significant literature has been created to promote various student engagement strategies and simultaneously introduced numerous metrics to try and measure it. Most of these metrics are based around tracking VLE/LMS/other digital footprint usage. There is an inherent challenge in this approach as ‘engagement’ is difficult to define - and hence, difficult to measure. It’s not to say there aren’t things to try measuring - the problem is that there are too many, and it’s hard to know which are actually real, and which are just vanity metrics!

Before we posit a different way of thinking about this, let’s take a moment to see how the world sees student engagement today.

Most recognise that it represents students taking their educational experience and internalising it. It’s more than simply whether or not they show up to class - it’s whether they raise their hands in the class, and whether they choose to participate in online discussion forums or sign up for extra curricular activities.

Newmann (2), one of the first to publish on this topic, provided us with not only some proposed definitions but also a framework to think about engagement. Namely, engagement can be thought of in a few different ways:

1. Behavioural - whether the student takes the right physical actions (e.g. doing homework, coming to class, and participating in extra-curricular)
2. Cognitive - whether they engage their minds in a deep way (e.g. critically thinking about why they are doing a certain task, and looking for better ways of doing it)
3. Emotional - whether it looks like their ‘heart’ is in it (e.g. does the student show signs of a sense of belonging)

Particularly when thinking about the latter of the three, it becomes easy to see that definitions can get a bit blurry. What does it really mean to assess if a student is “emotionally invested” in their learning experience? Could you guess how much your students love you?

Instead of getting bogged down in that, perhaps it is worthwhile to remind ourselves why this is important. A study at Central Queensland University (3) found a correlation between the number of student views on the course home page and students final grade. A similar study at California State University, Chico (4) identified that the more time students spent on learning tasks within the LMS, along with a high number of visits to the course home page, was associated with higher student grades.

In both cases, higher level of engagement led to higher grades for the students.

A 2015 Gallup study (5) has shown that  only 50% students consider themselves engaged.

This is obviously a lot of money left on the table.

The money becomes much more real when we consider that in the UK, student drop-out rates have increased for the third successive year (6). Universities are increasingly using learning analytics (7) to identify levels of student engagement, and using the insights to build student intervention strategies.

Higher levels of engagement leads to greater student persistence.

Clearly, student engagement is important to both academic success, and retention, and measuring it seems like it should be a key task. However, current research (8) confirms what we can guess: this is a complex concept, and even the three national student surveys (National Survey of Student Engagement, National Student Survey and UK Engagement Survey) only can collect limited and discrete perspectives of engagement.

Alternatively, if we look at the raw data, it’s easy to get lost in the numerous metrics and flashy dashboards. In this climate, finding a universal metric that covers everyone’s use cases (9) is clearly not straightforward.

We posit that perhaps this is a broader customer engagement problem which others have posed solutions to, which the education sector as a whole could perhaps draw from.

Customer loyalty strategies have evolved over time with these same challenges, and brand advocacy has been identified as a significant indicator of engagement (10). This is likely because it encapsulates not just satisfaction but perhaps also that indefinable emotional investment that Newmann first directed us to look towards. If your students are advocates, maybe they really do love you?

We’re not claiming this is the newest or most unique piece of rocket science around, but perhaps this is a sign that can be used to separate signal from noise. How can an institution measure student advocacy as a sign of engagement? Social media sentiment analysis might be a key step given its reach and the fact that it is where students can be found in their natural habitat.

It should be acknowledged that although it’s easy to use social channels in this way, the enforcement of GDPR and growing awareness of privacy rights has contributed to declining social activity (11) and many more users are making their social profiles private (12). The end result is that sample sizes may not be statistically relevant enough to inform decision making.

So if we’re saying that advocacy is the direction to look, but social media may be an incomplete picture, what should you do? Two-thirds of the Fortune 1000 have adopted the Net Promoter Score approach (13). The principle here is deceptively simple: if a customer is willing to recommend a brand to someone else, it means something. If your student is willing to recommend your institution to their friends and families, using whatever channel they prefer, it means something.

NPS isn’t without its share of criticism (14), but there needs to be an acceptance that nothing is perfect, and a metric which can aggregate many other emotions into a single number is a powerful ally. Particularly when these emotions can’t really be defined effectively individually in their component parts.

So what’s next? Ask the question of your students. Think carefully about when you’ll ask them, and then ask them - you’ll find out how engaged they are. Ideally, you’ll also be able to  guess how much your students love you. Hopefully, it is all the way to the moon. And back.

Thanks for reading this article. It was originally posted on the Squiz website. If you liked it, you may appreciate some of my other posts:

• How to set up real metrics for measuring your tech project's success
• 5 ways to get more out of your CRM investment
• How to ensure your tech project doesn't lose its fizz

Please feel free to use the comments box to say you love it, hate it, or have better ideas. Or if you just want to share your own experiences that'd be great.

Citations

1. McBratney, S. (1994) Guess How Much I Love You. Walker Books.
2. Newmann, F. (1992) Student Engagement and Achievement in American Secondary Schools. Teachers College Press. pp. 2–3.
3. Beer, Clark and Jones. (2010) Indicators of engagement. https://www.ascilite.org/conferences/sydney10/procs/Beer-full.pdf
4. Whitmer, J., Fernandes, K., & Allen, W. R.. (2012). Analytics in Progress: Technology Use, Student Characteristics, and Student Achievement. EDUCAUSE Review Online, July. https://er.educause.edu/articles/2012/8/analytics-in-progress-technology-use-student-characteristics-and-student-achievement
5. Gallup Student Poll 2015 Results. https://news.gallup.com/reports/189926/student-poll-2015-results.aspx
6. https://www.theguardian.com/education/2018/mar/08/university-drop-out-rates-uk-rise-third-year
7. https://www.jisc.ac.uk/blog/how-universities-can-use-learning-analytics-to-boost-fair-access-and-retention-11-apr-2017
8. Emma Charlotte Maskell, Lorna Collins, (2017) "Measuring student engagement in UK higher education: do surveys deliver?", Journal of Applied Research in Higher Education, Vol. 9 Issue: 2, pp.226-241, https://doi.org/10.1108/JARHE-11-2015-0082
9. https://xkcd.com/927/
10. https://influitive.com/blog/customer-engagement-loyalty-revenue/
11. https://www.bloomberg.com/news/articles/2018-07-27/facebook-says-eu-privacy-law-caused-user-drop-europe-disagrees
12. https://www.youtube.com/watch?v=E6DcWMaX1HU
13. Jennifer Kaplan. "The Inventor of Customer Satisfaction Surveys Is Sick of Them, Too". https://www.bloomberg.com/news/articles/2016-05-04/tasty-taco-helpful-hygienist-are-all-those-surveys-of-any-use
14. Satisfaction as a Predictor of Future Performance: A Replication. Jenny van Doorn , Peter S.H. Leeflang, Marleen Tijs International Journal of Research in Marketing (Impact Factor: 1.71). 12/2013 https://www.rug.nl/research/portal/publications/satisfaction-as-a-predictor-of-future-performance(d0a09fc5-48d4-417b-9b2a-f35e13e0c71e)/export.html