Cutting mind-body control to unlock a new level of immersion

Imagine playing a game with yourself and not knowing what move comes next. Or being a spectator and player at the same time. Futuristic? Yes. Impossible? Not anymore.

Lend your body to technology

We’re not talking about clicking a mouse, swinging a Wii remote or running in place with a VR headset. Although you’re combining physical movement with tech, everything still happens on a screen – immersion’s arch nemesis.

What we’re referring to is letting consoles ‘play’ you, a novel approach by our exertion gaming PhD student Rakesh Patibanda that better integrates the human body with computers.

This innovative project has the potential to greatly benefit areas beyond games, from rehabilitation to assistive technologies – even astronauts in space! But more on that later.

To demonstrate Rakesh’s ingenuity in action, we’re putting you in as player one for two scenarios (based on true events).?

#1: Putting the ‘auto’ in ‘auto-paizo games’

It’s a sunny day and you’re strolling down the street. A friendly man calls out to you. ‘Hey! Did you know you can play games like rock, paper, scissors – with yourself?’.?

Odd, but curious. How can one play it alone? You always know your next move.?

As you walk over to the marquee, you see a bunch of equipment on a table.?

‘Hi there’ the person says. ‘I’m a researcher from Monash University’s Faculty of IT, and we’re testing a new way to immerse in games and play.’?

‘We hook this EMS system, microcontroller and electrodes to your left hand, and this glove and phone to the right. Through the app on the phone, choose a game and calibrate with the EMS system.’

‘In this case, let’s play Elements. It’s similar to rock, paper, scissors, except it’s with water, air and fire. Here look at these cards as a reference.’

‘The electrodes, which are placed over certain muscles, produce small electrical currents that move your hand into a gesture reflecting one of the elements.’?

‘You’ll hear a sound before the system takes over. And don’t worry, it’s perfectly safe. The currents are small so you can overpower them at any time. Also, this field study has full ethics approval from the university. So, want to try?’?

You read over the instructions, agree to join and let the researcher fit the equipment onto your hands. Afterwards you open the app and click “Elements” and “Free Play”.

‘Ready? Now, think of a symbol to make with your right hand. Then curl your left hand inwards to start the countdown’.?

You decide to gesture ‘Air’ with your right hand and then curl your left hand inwards.?

After a few seconds you hear a beep followed by a tingling sensation. Suddenly your left hand’s pointer finger and thumb shoot up.

‘Looks like the EMS hand won! It chose Fire which beats your Air. How did you find the experience??

Here’s how real users found it

“It was a weird experience losing to myself if the EMS hand won.’

“It does make you take 15 minutes out of your day to think about your body, and that’s an interesting concept.”

“I thought of these games as almost like, mindfulness or meditation activities, where I had only to use one part of my body actively and let the computer play the other part.’

In the study, the half-second delay between the currents passing through the body and the EMS hand showing the gesture also helped create a ‘meta-guessing game’ where the player anticipates the gesture after feeling the currents.

Have to see it to believe it? Way ahead of you ??

#2: Whose game is it anyway?

‘Want to take it a step further?’ the researcher asks. Why not, you’re here already.

You nod, and they pull out a different set of equipment with two EMS systems connected to one microcontroller.

‘This time, you’re going to experience being a spectator and player at the same time. You’re going to watch both your hands play against each other, and you won’t be controlling either of them. Just interpret and enjoy’.?

He removes the glove and phone from your right hand and sets the new system up. You stand there like something from behind the scenes of Avatar.

So we’re going to play one of our specially-designed games similar to Elements called ‘Godai’. Here are cards to tell you what the different gestures mean.

‘Now, you’re going to hit play on this phone and then all you do is breathe. After 20 seconds, a countdown will show on the screen. When it reaches 0, a sound will play indicating when these EMS systems will take over your hands. It also has a voice control, so you can tell it to ‘stop’, ‘pause’ or ‘resume’ at any time.’

You breathe, wait and sure enough the countdown starts followed by a beep.?

Your right gestures an open palm, symbolising ‘Water’, and your left also gestures an open palm except with your wrist bent backwards, symbolising ‘Wood’.

‘So, who wins?’ the researcher asks.

[Look at the symbols to try determine which of your hands win. The correct answer will be revealed at the end of this article.]

The proof is in the pudding – see the research in action ??

Beyond gaming: Benefits for the real world

Rakesh’s approach gives us a glimpse into an exciting direction for more meaningful, engaging and immersive gaming experiences. But the potential of his work extends far beyond the arena of play.?

Lending bodily control to technology can augment motor rehabilitation, enabling therapists and medical practitioners to help patients with certain movements and activities – even from afar.?

Adding another layer of potential, it can also be used for assistive technologies to help people living with disabilities become more independent.?

Other real-world applications include enriching the teaching of sign language to better communicate with those who are deaf or hard of hearing, improving fine motor skills and even providing physical assistance to astronauts in microgravity environments.

Ditching the dystopia with trust

Movies like Tron, Minority Report and The Matrix often depict technology as an entity of total domination that only wants to control human life. Although great for cinema suspense, this isn’t (and should never be) the case.

Trust is a must – and it comes from ethics, safety and comfort. The scenarios above, and the real studies behind them, show ways to build these elements:

• Ethical design: Providing detailed information about the experience, obtaining informed consent and implementing protocols to prevent the overuse of systems, such as kill switches and voice control.
• Benefits-focused: The guiding light in any innovation should be how it will benefit real people, such as giving them better experiences or encouraging bodily reflections and mindfulness.
• Comfort and safety: In both studies, users were aware of when the EMS systems would take control because of the sound. In the second one, breathing was also encouraged to relax muscles and reduce any uncomfortable sensations.?
• Respect for autonomy: Most importantly of all, at every stage of each study users could take back control by easily overpowering the currents.

Taking the global stage

Rakesh’s research will be presented at the prestigious CHI PLAY ‘23, a global conference open to professionals and researchers across all areas of games, play and human-computer interaction.?

This premier event will run from 10 to 13 October in Stratford, Canada.

The winning hand

If you interpreted your right hand, gesturing ‘Wood’, winning over your left, which was showing ‘Water’, then you were CORRECT!