Planes, Games and Performance Trains

There is much to learn from Massively Multiplayer Online Games (MMOG) and the amazing experience they deliver to vast numbers of users, despite limitations in connectivity and client hardware.

This post is about my recent encounters with one such game and the various enlightening 'trains of thought' that followed, especially relating to performance and user experience (UX).  There are many larger games than the one I started playing, but it serves as a good example of UX optimization. My thinking was not only prompted by the great game play experience I enjoyed, but because it is possible to join the testing team by earning 12 stars.  Nice to see 'testing' as something to play for!

We didn't go away for the recent summer school holidays, but we did play lots of online games at home.  I've not really played online games - but this holiday was different.  My children introduced me to War Thunder, one of many games they enjoy.  The name did not grab me, but the game play did.  I like technical detail, 'ease of use', performance, flying and military history, so it is not really surprising that I enjoyed the game.

What really was surprising is that War Thunder has only 4 server locations, globally, and my experience was fantastic - even when playing on  server with 350mS network ping time!  The following 12 key observations about the game and its performance are relevant to any IT application that seeks to deliver a great user experience.

1. Focus on Interactions over Transactions

In the world of corporate transaction systems, response time and end user experience is generally centered on the individual transactions.  However, in games, UX is oriented around each user interaction.  This is a significant development in the evolution of User Experience, as transactions tend to focus on the back-end server, but interactions focus on the user's experience.  Very profitable online games deliver such rewarding and exciting experiences that users will pay significant real-world currency to unlock greater levels of satisfaction within the game.

For much of my performance testing career, the focus has generally been on "not loosing" a user due to slow response time or poor transactional performance, but game makers constantly come up with new ways of eliciting new money from engaged users who want even more of what the game has to offer.

2. Do all the 'slow stuff' out of the 'main game'.

The download for War Thunder is 13GB.  That is a lot of data.  It is mostly maps, which means that during a game, the only communications that are necessary between the players and the server are the individual actions, such as accelerating or shooting.   I monitored the communications while playing and was surprised to see only a few packets per second being transferred.  I even pulled the network cable mid game, and was able to continue on for many seconds before getting a 'disconnected' error.  With the network cable pulled, I still saw increasing detail as I approached a group of trees, because every tree in the thousands of square kilometers of game map is in the stored locally.

When starting a game, it appears that all of the game configuration, such as vehicle parameters, target locations, moving target tracks, crew details and other details are determined and shared with each of the clients participating in the game.  This means that once the game commences, the only communication is the vital real-time messaging required to keep every clients view of the game world updated.

It is amazing to me that such an immersive, data rich experience can be enjoyed by some many participants at the same time, with each player directly interacting with their own copy of the game universe - as if it was purely local.  Many business applications could benefit from the philosophy of such an approach.

3. Use a Super Lightweight Protocol

Many corporate applications routinely transfer huge volumes of data between server and client, increasing data requirements for users and adding latency to their experience.  However, many online games are effectively a type of real-time application, and delays caused by transmission of large blobs of data would undermine the game experience.  The fact that available bandwidth is increasing year on year does not mean that web based apps should needlessly send the same large CSS and JS files to each browser session.

I still find it surprising when performance testing identifies that compressible resources are not compressed, when content that could be cached on a client is not cached, and when unnecessary content is downloaded in such a way to degrade the end user experience significantly, especially when the end user does not have the benefit of a low latency/ high speed connection.

4. Security

Access to many online games, such as games on the Steam platform, are now controlled by two-factor authentication, so that inventories of virtual game 'treasures' can not be easily stolen.   I'm sure that children, who understand that two factor authentication is necessary to protect their virtual treasures, would be surprised to know that some adults don't use two-factor authentication to protect their real money!

5. Give users visibility of Performance Impacting issues

It is surprising how much value can be derived by displaying a few key metrics in a corner of the application window.  For example, if I were to start an aggressive (incremental) backup on my work PC while several of my children are playing online games in our home, I would suddenly hear cries from all of them at the same time, with phrases like "what happened to the ping", "what just happened", "Dad!! What are you doing"....  They have learned that a sudden change in their individual ping times suggests a local problem rather than a problem with their actual game (on a remote server), and just about all of the games they play include ping time to their server on the main game display.  Likewise, a sudden decrease in fps causes players to focus on their individual client (PC) to see what is going on.  Seeing the metrics helps users understand the environment that the application depends on.

Games such as War Thunder include 3 key metrics.  Ping Time, Frame Rate and Packet Loss in the bottom left corner of the screen.  Armed with such metrics, game players can get a sense of problems that are degrading their game play experience - without simply blaming the gaming provider or the 'Internet'.

For game players that do not care about the ping time, frame rate or packet loss, they can easily ignore those numbers that are hidden away in a corner of the screen. 

This is also relevant to some modern business applications, where an indicator of local machine performance, as well as local network performance, would be provide valuable information to individual users who care.

6. Play Optimistic / Work Optimistic

Online game play can reveal inner attitudes.  When I started playing War Thunder, I was quite defensive.  When I faced a robust AAA system, I approached very cautiously.  However, as my game play evolved, I became more confident and more optimistic about achieving the objective.  When flying into a AAA flak field, one can defensively try and dodge the flak, or one can realize that it has a small statistical chance of hitting.  By taking a risk based approach, and targeting the primary threats first, one becomes free to attack any problem with confidence, and be much more successful in the process.

Historically, many testing teams have been 'defect' and 'problem' focused, rather than 'success' focused.  While this is understandable, I believe that a better approach is to be 'optimistic' - if that is possible.  This means developing a testing strategy and approach that 'optimistically' expects to get a pass - if and only if the System Under Test performs well enough to earn one.  It also means being able to recognize the situations where it is folly to be optimistic.

7. Design the app to handle sudden changes in latency

While optimism is great, reality often gets in the way.  For many online applications, network latency is to UX what Kryptonite is to Superman. The most impressive aspect of games such as War Thunder is that sudden changes in network Latency, and even Packet Loss - have surprisingly little impact on the game play experience.  War Thunder, for example, sends messages to the server as UDP packets (rather than over TCP).  This means that communications or server side infrastructure issues can result in dropped packets. (Severely delayed packets get counted as 'dropped'.) Each time you alter the flight parameters of your aircraft in War Thunder, a message is sent to the server in a UDP packet. Each message contains enough information so that a missed packet would not cause a severe issue as the following message would have enough information in it to make up for the content of the lost message.

The reason this works for War Thunder is that, like in many similar games, the entire game universe is instantiated in memory on each and every client instance that is participating in that game.  When I dive and attack a ground unit, each of the interactions that I have with my 'aircraft' is handled, without any loss or latency, within my local copy of the game space.  I get a 'real time' experience. However, each of the interactions are also communicated to the server, so that each other participant can be informed of my actions, and if they are looking in my direction, then they will see my aircraft attacking the ground unit.  

If one of my bullets is in a UDP packet that is dropped, then I may not register a hit, but in real life, not every bullet is perfect - so the occasional dropped packet does not take the game too far from reality.  However, if it is a hit, and the packet is not dropped, then all other players will get messages that update their game instances with the fact that I just destroyed a ground unit.  I watched a game replay (session video recording rather than in-game recording)) and observed that early in the ground strike, Packet Loss suddenly rose from zero to 5% and lasted for a couple of seconds.  I did not notice anything bad during the actual game play, but only noticed it as I was focusing on packet loss in the replay for this article. My ground strike maneuver (dropping 2 bombs and firing 6 rockets against a group of tanks) took 3 seconds, but I can't say that the packet loss made any difference to the result or the game play experience - and I still took out 5 of the tanks.

To further see how this worked, I played a game, and in the middle of an attack sequence, I disconnected my Ethernet cable, and to my surprise, I was able to complete the attack, as if I was connected to the game, but the hits did not register.  (It took several seconds for the game to give me a "disconnected" error.)

8. Provide a variety of user experience levels, based on client hardware

Millions of players play War Thunder each month, with a diverse variety of hardware.  If a player wants a fantastic gaming experience, then they may well have high end graphics cards and high performance displays and fast CPUs, but many other users will have much lower specification equipment.  The game play is primarily dependent on the Frames Per Second (FPS) that the local instance can generate with the users hardware.  The more pixels in each frame, and the more detail in each frame, the more work that needs to be done to render each 'frame', so selecting 'easier' rending options reduces the CPU and Graphics requirements to generate an acceptable frame rate.

If one has high end hardware, then one can select 'movie mode' graphics, and if one has low end hardware, then one can select 'low graphics'.  This means that the 'game play' may be more immersive and visually stunning for a user with high end gear, but the low end hardware player can still compete and win against the high end player - even if a little squinting at the lower quality graphics is required.  There is a vast set of options to customize, prioritize and optimize War Thunder game experience to a particular PC.

As an aside - I don't have a 144 fps screen, and I didn't think using one would make any difference (based on my flawed logic that movies and TV look fine at 24 fps - so why would I need 144 fps) , but after trying one out, I was amazed at the smooth screen panning, and my eyes resolved detail while the screen was actually panning.  This benefit of high fps screens would give users are significant advantage in competitive game play - but is a very expensive step up in hardware costs.

9. Train your crew

In an Air Battle, your ground based crew is just as critical to both survival and success as your flight crew.  For example, War Thunder recognizes this by altering component reliability (e.g. the mean time between failures for guns) based on flight and ground based crew training.  I tend to focus on training crew as a higher priority than upgrading aircraft components, because a well trained crew can do more with a mediocre aircraft than a poorly trained crew can do with an aircraft with all the 'bells and whistles'.

This also applies to test teams, but it is not limited to formal test training.  I have found that user conferences and similar testing events have been the best opportunity for understanding what is possible and challenging me to maintain testing 'best practice'.

Most testers see value in some 'unstructured' testing, and for me, conference attendance is like 'unstructured' training.

9. Communicate and collaborate

16 players doing their own thing will achieve less team objectives than they would if they communicate in real-time, as the game progresses. Most MMOGs allow for real-time chat and many teams also participate in audio conferencing, so that they can communicate and collaborate their way to victory.  

Games such as War Thunder promote collaboration, by significantly rewarding teams that win the individual game objective. By ignoring the collaboration tools, the objectives are much less likely to me achieved.

Based on my observations of my children enjoying MMOG game play, it seems like animated (and loud) conference call participation is the "icing on the cake" from an enjoyment perspective of MMOG game play.

10. Use client side CPU where possible

Games such as War Thunder have three types of targets. Stationary targets, such as enemy bases, moving "AI" targets, such as vehicles and ships, and vehicles controller by other players.

The AI targets require little CPU resource, and are fairly easy to hit.  However, the most gratifying targets to hit are the ones controlled by other players.  The CPU requirements for directing those targets would be massive if they could play with the same skill as real players - which is why the game works so well. All of the really heavy processing is offloaded to other game players and the PCs of every game player.  

While the game is free to play, it effectively leverages each player's PC and the brains of each player, to generate a game of complexity that would otherwise require huge hardware expense.  This is the "magic sauce" of the scaleability of MMOGs and the high level idea should be considered for any business applications where scale is important.

11. Think about both strategy and tactics

 I would call the game play within a session the 'tactics', and I would call the decision on technology trees (to be able to select your desired combat vehicles) strategy.  War Thunder game play helps participants consider both short term (tactics) and longer term (strategy) and learn from mistakes.

The same type of constraints are relevant to IT generally, as well as the testing of IT systems.  Projects are like game sessions, and the tools and methodologies we use are like paths in the technology trees.

12. Motivate & Monetize

Game play is motivated in the player by the gaming experience, and the game development is funded by the game's monetization strategy.   It is interesting that many MMOGs with a "Free Play" offering can generate vast amounts of revenue.

While designers of corporate web applications are trying to minimize the number of users that 'leave' their site, in the hope that they will 'stay and purchase' a physical item, MMOG designers are not only figuring out ways to keep their clients active on their site, but they are also succeeding in generating revenue from those clients - often in exchange for virtual items that don't have any real intrinsic value in the real-world.  

Key Observation from Online Gaming Experience

The most profound observation from my recent game play experience is that the standard for online end user experience is being lifted by popular well designed games, that engage with users and deliver an immersive experience, despite differences in client side hardware and network connectivity.

Paul McLean is the owner of RPM Solutions with 16 years of deep technical experience with all things Load Testing. He applies creative approaches to identify and eliminate defects that lead to performance and stability problems, through a regime of methodical testing that is in line customer with timeframe and cost constraints.

Here are some of his other LinkedIn articles:

• Long running load tests with sin are better
• Performance Engineering in Agile Projects
• Performance Testing Mashups over Hybrid Networks with HP LoadRunner & NV
• The Quality Magnifying Power of Redundancy
• Lean Load Testing in Agile Projects
• How expensive is a non-expert consultant
• IT Quality and the Irresistible Force Paradox
• Comparing Performance Testing Specialists with Baristas 
• Testing with More Cloud – for Less
• Lean and Mean Application Load Testing
• Load Testing in the Fog of WAR
• Load Testing: A Walk on the Plank