Part 2 - Machine Learning AI in a Hack 'n' Slash game - Neuro Evolution and Conclusions

In my last article, I talked about my University project - Machine Learning based AI applied in a Hack 'n' Slash game. I detailed the general Game Development work that went into the game, as well as the implementation of Q-Learning.

I finished working on the project and dissertation for University about a week ago. I will talk a bit about what went well and what could be improved.

Final video of the Game -

You can download the game here -

Progress I have made since my last update -

1) Implemented Neuro-Evolution using UnityNEAT (Thanks to David Fogel for recommending Neuro Evolution)

2) Selected the Q-Learning and NEAT AI that I wanted to include in the game prototype

3) Worked on some Game Design and basic Level Design to make a more interesting game

4) Asked some people to test the finished game

Here's a short video of UnityNEAT in action, training my army of knights -

NEAT is a really interesting and powerful technique. It stands for "NeuroEvolution of Augmenting Topologies". It is a type of EANN (Evolutionary Artificial Neural Network), which searches for the best architecture of an Artificial Neural Network (ANN) to solve a problem. Generally, the architecture of a Neural Network is central to how effective it is, and choosing the parameters for it requires a lot of expertise and trial and error. NEAT attempts to overcome this problem by starting with a population of simple Neural Networks, and evolving their structures and weights using a Genetic Algorithm. This method of increasing the complexity of the ANN emulates how nature works. NEAT was developed by Ken Stanley in 2002. You can find out more about it here - https://www.cs.ucf.edu/~kstanley/neat.html

Implementing NEAT

In this section, I go into detail about how I implemented NEAT. It is a lot of text, so feel free to skip over to the last section for a summary of the project.

Because of the limited time I had left before my submission was due, I was very relieved and excited to find a port of SharpNEAT (A C# implementation of NEAT by Colin Green https://sharpneat.sourceforge.net/) for Unity, called UnityNEAT, by Daniel Jallov (https://github.com/lordjesus/UnityNEAT). Ideally, I would have liked to have a more in-depth understanding of Neural Networks, Genetic Algorithms, and Neuro Evolution, possibly by programming at least some of them on my own. I intend to spend time on this in the future.

But what was important to me was using NEAT in the context of a game, and producing a fun and interesting game play experience. I concentrated on learning how to interface UnityNEAT with my project and AI code, This involved re-writing and re-structuring the code for my AI character, and altering some of the UnityNEAT code to evolve my AI character to fight well.

I felt SO accomplished and happy once UnityNEAT was working with my project. The real challenge after this was making it work "well". Implementing NEAT to quickly solve a problem requires expertise and experience. Since I lacked both of these at the time, figuring out how to make it work well involved a lot of trial and error, and many frustrating moments when no progress seemed to be made. It was generally easier to find information about the original NEAT rather than SharpNEAT or UnityNEAT, through Stanley's website for NEAT, and the original Journal Paper and Dissertation about NEAT. A lot of the information about NEAT didn't apply to SharpNEAT. I'm sure it wouldn't have been as difficult if I had an in-depth understanding of NEAT, but I found it to be quite complex.

The fitness function was a very important part of this. I started with a simple and to the point fitness function - (fit=myHits - enemyHits). The justification for this was that as long as the AI had more hits on the enemy, it was performing well. My hope was that the evolution process would produce a mix of behaviours, and a set of behaviours such as evading hits and attacking the enemy when within range would naturally produce a high fitness. This should be the way the AI evolves. However, since the most straightforward solution leads to the highest initial gains in fitness, what I found was that the "best solution" found by NEAT was an AI that would stand in a spot and spam a single attack. This was neither interesting, nor useful, as the AI would deal about as much damage as it would receive. Even after a 1000 generations of evolving a population of 50, NEAT would still be stuck on the same solution with the same fitness.

After a lot of painful trial and error with the fitness, which involved tests with logical errors on my part - (fit = dodge + myHits) would just result in the AI walking all around evading every attack and dealing no hits as this easily maximized the fitness - what finally worked was a higher population size of 100 and a fitness function that took into account factors that would be likely to lead to a good AI. This included implementing "diminishing returns" for these factors, so that a mix of high values for the factors would have a higher fitness than having just a single factor with a high value. This encouraged a balanced play-style for the AI.

fit = (myHits + myDodge + MyBlocks - 0.25 *(miss + OpponentBlocks))

I have since received some more advice and ideas for improving NEAT's performance from Colin Green (Thanks Colin!), including using a Log function for the fitness. I intend to try this out soon.

I was able to get some AI using NEAT that had useful and interesting behaviours. One good result took 350 generations of evolving a population of 100. The other took 85 generations of the same population size. The way the AI is tested for fitness in Unity is with somewhat random actions by the competing AI, so it is non-deterministic, and using NEAT multiple times with no changes to the parameters produces different results.

The most interesting NEAT AI is now at the end of the game, as the final boss that the player must defeat. The AI starts the level standing still with its back towards the Player. This tends to give the Player a false sense of security while approaching the AI. The moment the Player comes within range, the AI suddenly turns around and charges at the Player with an attack. It has some ability to evade attacks and circle around the Player as well. Sadly, it is a bit unreliable when compared to the Q-Learning AI, as it tends to repeat its initial behaviour, which makes it predictable. And by the end of the game, the Player is quite used to the mechanics and the AI's quirks and flaws, so many Players had no trouble at all defeating this AI.

Results and Conclusion

My project included building a final prototype of the game, in which the Player faces increasingly challenging AI as they fight their way to the King (The Boss Enemy). This included some more Game Design, Level Design, and a Tutorial. The data isn't too indicative of how well I did this, as it was tested by some students from my university and some friends.

My main goals were to provide a fun and interesting game play experience, and a smooth increase in difficulty, using Machine Learning AI.

The first opponent was a rule-based AI, followed by a weak NEAT AI, then increasingly harder Q-Learning AI. The hardest Q-Learning AI faced the Player three times with real-time learning enabled. The final boss enemy, or the King, was the NEAT AI that had a deceptive behaviour, mentioned at the end of the NEAT section above.

(Grey is Rule-Based, Red is Q-Learning, Blue is NEAT, Red evolving is Q-Learning in real-time)

I found that all players genuinely enjoyed the game, and were intrigued by and interested in how the AI were behaving. All but 1 preferred the different Machine Learning AI to the standard Rule-Based one. The Q-Learning AI was the most preferred. Most players felt that the game provided a smooth and immersive difficulty curve. The Q-Learning AI performed consistently better than the others. The NEAT and real-time Q-Learning AI were a bit unreliable, but at times performed well and were appreciated by the Players.

Real-time learning with Q-Learning

The real-time Q-Learning AI had one of 3 results when tested by Players -

1) The Player's skill would improve so much while fighting it three times that they wouldn't notice much improvement in the AI.

2) The AI would get hit a lot by the Player mid-attack and get discouraged from attacking. It would become more defensive and evasive.

3) When the AI got a lot of successful hits on the player, it would become more aggressive and use a lot of attacks. This was judged by the Players as an increase in challenge.

Even though the AI didn't always learn to get better at defeating the Player, it was appreciated by the Players for its change in strategy over time. Players tend to perceive a changing AI as an improving and smart one, and become more cautious and strategic in their approach.

NEAT AI

The NEAT AI was the most interesting one for me. Both the NEAT AI used in the game have a larger variety of behaviours that they take in different situations. They can be hard to defeat but sometimes exploitable.

Conclusion

Both Q-Learning and NEAT produced a variety of AI with different behaviours. The Q-Learning AI is a bit more straightforward in its actions, while the NEAT AI behaves in a clearly different manner. Q-Learning is more reliable, but requires the support of a rule-base to work well. Neural Networks have more potential for completely controlling the AI without the help of a rule-base. NEAT evolves ANNs well with a bit of trial and error. Both can produce interesting and fun behaviours, and unique solutions to problems, within the context of games. Machine Learning techniques can be used to create AI of a variety of difficulty levels.

The fact that the gameplay is slightly different every time is both its strength and its weakness. Players appreciate the novel and changing experience of the different AI enemies. But since the challenge of the more interesting AI isn't reliable, it cannot be guaranteed to work well for every Player. These trade-offs can be balanced in the Game Design, with a large variety of enemy AI that behave differently and keep the Players on their toes, while covering up for the fact that some of them might not perform well.

Overall, I believe I have achieved to an extent my goal of a Hack 'n' Slash game with Machine Learning AI. The game has been appreciated by most so far as challenging, fun and immersive.

Future work -

1) Better AI! Q-Learning and NEAT AI could be improved with further experimentation, including training them using "coevolutionary complexification" (Mentioned in Stanley's dissertation), or just training against smarter AI rather than a basic Rule-Based one.

2) More polish - Animations, glitches, smoother feeling to the controls

3) Changes to the combat system - Some players wanted attack cancelling and faster gameplay

I am happy to share technical information and sources of information, if anyone is interested.

Any feedback and suggestions for the project are appreciated. Thanks!