Introduction to Games QA | Study Notes #1

When I decided to go deeper into QA and learn about this crucial role beyond the classic "they play a lot and find bugs", I started looking for courses online.

These Notes only cover the first module of the course and work as an introduction to QA in games. As Test Cases get introduced I will follow this up with a Study Note on test management software (Test Rail) before the next module. Hope this helps!

Introduction to QA

What is it?

There is no game industry standard for QA in games as every game is different and every company has its own process, but a tester’s job typically implies playing different builds of the game over and over looking for bugs or trying to force them by stressing the software.

• Quality Assurance = Testing
• Process of verifying if the product meets required specifications
• The output of QA is bug (defects in the code)

When is QA done?

QA could and should be done in every phase of software development, It depends on how the team works, it also depends on what kind of QA you’re doing and on what kind of game (a live game needs QA even after release).

If you’re a functional QA tester you could start testing features as early as the first phases of production or as late as when you first have a complete build of the game.

How is QA done?

• By testing against something (a game/test plan that tells you how the game is supposed to be)
• With a test oracle, a set of information that helps you decide if the outcome was successful or not
• By following predetermined procedures (test cases)
• By repeating your steps several times to test reproduction rate

What are you going to discover?

• Errors in the code
• Flaws in software design
• Missing features/elements
• Legal and cultural issues
• Behaviour of software under ideal circumstances
• Behaviour of software under stress
• Quality of software performance (lag, framerate, loadings etc.)
• Production quality (how polished is the game)
• How does the software benchmark against competition (ex. loading times)

Who does the QA?

• QA team (testers, analysts, specialists)
• Internal QA team
• External QA team
• Development team
• Players will do it naturally as they play

What is NOT QA?

• Criticism of the development team
• Personal opinions on the game
• An enemy of the team or the product
• A substitute for unity testing done by the developers
• Self serving, you shouldn’t report irrelevant bug or bugs created in very unrealistic scenario just to create more bugs

Future of Testing

• Testing is evolving with agile methodologies which require to be more flexible and reactive to changes in the code with a faster turnaround, this also resulted in a rise of automation in testing and in the use of a lighter documentation
• Testing is also evolving towards being more in service of the customer, you should test the game thinking about how a potential customer would play it more so than how the designer planned it

Games QA basics

What do we test in games

• Evaluation of game rules
• Purchasing
• Physics testing
• UI and UX
• 3rd party integrations
• Sound and music
• First Time User Experience
• Social
• Realism
• Performance
• Multiplayer
• Text and translations (LQA)
• OS/Platform compatibility
• Fun factor
• Compliance/legal
• Visuals
• Balancing
• Playthrough
• Game Logic
• Game level/world
• Achievements
• Consistency
• AI

Another important aspect to test is hardware

• You need to run tests on a variety of hardware sets
• For mobile you need a good coverage of devices and of operating system’s versions (this also applies to web browser versions)
• For PC you can’t test all configurations so you make estimates based on the data of your players, your target market and the type of game you are making?
• Console games are tested on dev kits ordered from the manufacturers (Sony, Microsoft, Nintendo)

What is the most important thing to test?

• Depends when you’re testing
• You want to find the most complex and serious bugs as early as possible as they could take lots of time and effort to fix
• Depends on the game, based on the target, how would the player interact with the game
• Could depend on the platform or the business model of the game (premium or F2P)
• Some features, like multiplayer, require more time and testers
• Platform requirements could be more of a priority if you’re testing for consoles compared to PC

Testing set up

You should test the game in an environment that copies the production environment as close as possible with some unique points:

• You will work in a testing environment where you can access all areas of the game and (if the game requires it) simulate purchases
• You will have a “cheat” system to allow you testing different moments of the game without doing full playthroughs

Testing playthrough

• These happen several times for QA cycle on premium games as they’re closed ecosystems
• Impossible for most f2p games as they don’t end
• Focusing only on playthrough and balancing could be a bad investment of resources and make QA miss important bugs, but not doing it at all might make you miss performance or transition issues late in the game
• There are specific methodologies for testing open world games or games with branching paths and a deep metagame

When is testing done?

• As long as the game is live it’s not done
• Even if not all bugs can be fixed they should be addressed so that we can decide what to do with them
• QA often acts as a gatekeeper to submissions, generally QA is involved in the submission process
• Live games operate differently, such as dealing with bugs that appear live while testing the new features in development

Know your player

• QA’s role is to represent the player, therefore you need to understand them
• Who plays your game and how?
• Casual, mid-core, hard-core, button masher
• builder, killer, socializer, team player, solo star, collector, explorer…
• Use these user personas to help you understand the player and put yourself in their perspective

Tips for effective game QA

• Good communication with the rest of the team
• Prioritization skills
• Speak up on time about constraints so that your managers can act accordingly
• Keep up with trends and technologies
• Attend QA conferences, read blogs, find a mentor
• Teach others on QA, They will be able to help you if needed and understand your work and limitations

Types of QA

Unit tests/white box testing

• Testing done by the developers themselves
• It’s done while looking at the code
• It’s generally good practice to test the code you compiled on your device and on the target device, just by doing this we could decrease the bug count significantly by catching the obvious ones

Functional testing

• Verifying the functionality of the game
• Testing features, interactions, performances etc.
• It’s what we usually think of when talking about QA, it also takes the longest time
• Often testers will specialize in checking a specific area of the game functionality

Regression testing

• Verifying that bugs that are supposedly fixed are really fixed
• Checking that a bug fix didn’t create some new bugs
• The test should be run under the same preconditions as when it was found

Stress testing

• Putting the game under extra stress
• Button mashing
• Fast tapping
• Running multiple games/apps at the same time
• Doing multiple actions quickly or in extreme quantities
• Often automated
• Done to find bugs but also to see how the software behaves and to find out performance boundaries

Compliance testing

• Testing done to verify that the game is going to pass platform requirements
• Extremely demanding for console, very lenient for PCs
• Important for mobile, especially if you hope to be featured on the stores
• Compliance requirements are often changed by the platform and sometimes are not straightforward
• It is considered demanding testing, bigger companies may have a dedicated team or it may be done by the publisher

Basic acceptance testing

• Small set of test cases to verify the major functions of the game
• the game loads, runs and the core gameplay is working
• Usually done to verify if a build is ready for more in depth testing
• If we are in a rush it could be used as a minimum set of requirements the game needs to pass

Localization testing

• Testing games in various languages
• Usually done by a native or native level speaker
• Checks for wrong translations, misspellings and consistency issues, but also for UI problems, breakdown of the text, missing translations, logical strings

Disruption testing

• Usually for mobile testing, you test the game while receiving a call, message or a push notification that could interrupt the game

Negative testing

• Using different testing paths from the “ideal”
• If allowed, the player will most likely not follow what the designer intended
• Important for mobile, especially if you hope to be featured on the stores
• Useful to find problems that are not obvious but could cause a lot of problems once the game is out

Test cases

What is a test case ?

• A test case is a set of instructions for the tester to know how to perform testing and what is the desired outcome.
• The process of developing test cases can also help in finding problems in the requirements or in the design.

When are test cases useful?

• When the QA is outsourced or when we are onboarding a new team member
• When they are repeated frequently
• To measure testing coverage
• To help testers understand possible permutation
• To guide you when working on tight deadlines and might work with new people
• To split the work more efficiently between testers
• For legacy as your current tester might leave or move to a different position

When are test cases NOT useful?

• When it takes longer to maintain them than to execute them
• When they are too restrictive and don’t allow you to follow the lead
• When they are used as the only measure of testers’ efficiency
• When they only exist because management wants them but nobody is using them

Other testing tools

Test scenarios

• They’re one line of information about what to test (“test the shop”)
• It’s about thinking and discussing details
• It saves a lot of time when team members can understand the details from it
• It’s a time saver, stimulates ideas and thinking, it’s easy to maintain and it doesn’t create testing fatigue (when a tester has run a test case multiple times and it starts becoming ineffective)
• It makes people think rather than follow commands
• The disadvantage is that it requires more effort from the team and it could be misunderstood

User stories

• They are high level instructions on how software is meant to be used (“As a player i want to do X”)
• They can be a source to write the test case or they could be used on their own
• They’re easier to maintain and more hands-on
• They give more freedom to the tester to “follow the lead”
• They require the tester to be more experienced and confident

Pairwise testing

• Intensive testing session for 1 to 3 hours
• Two tester, generally an experienced one to act as the lead and the other to act as the follower
• The two people test the same thing together at the same time by continuously exchanging ideas
• They have a high level test plan that they follow while bouncing ideas with each other and testing with different approaches?

What does a good test case have?

• A unique identifier
• A feature/area of the game to which it relates to
• Clear instructions
• A testing oracle (the document we are testing against -> what do we expect from the game)
• A status (pass, fail, incomplete)
• A link to the bug found so everyone know to which test case they are related

Platforms and OSs

Platform and OS relevance

• Every platform has its own set of rules for developers
• To submit games, the dev has to register to the platform (and usually pay a fee)
• Every platform has different submission rules, schedules and guidelines that have to be respected to get the game published (Compliance testing checks for this)
• Every platform runs on an OS that changes all the time and may affect how your game runs
• Often the game has to run on different versions of the OS even though they may be very different from each other and may require a different approach to testing

Platform differences

• Consoles have very strict compliance definitions (TCR’s, TRC’s). After the game is submitted to the platform it will be carefully tested by their internal team against their policies. Submission takes a couple of weeks.
• If the build doesn’t pass it will be rejected and the dev will have to resubmit, multiple submissions could result in a financial penalty
• On Steam (and PC in general) there are no specific test sets and the submission time is not specified.
• On mobile it depends on the platform. iOS and Android have their own set of regulations but the latter will generally test more thoroughly

Android

• Main store for Android phones
• Cheap to register (60€ per year)
• Games are submitted to the Google Play portal in an APK format with additional info about the game (screenshots, description, integrations etc.)
• You need to specify the minimum android OS version otherwise You are supporting all of them
• You have an option to blacklist devices and countries
• There are so many combinations of devices and OSs versions that you have to choose based on
• The most popular devices in your main market
• By grouping similar devices
• If time is limited pick at least flagship models for the most popular brands (also important to get featured on the store and to get early adopters to play your game as they’re usually the best spenders)
• Always pick at least one of the lowest supporting devices
• Test on latest devices
• Always test on different versions of Android (don’t update all of your phones)
• Submission time is usually only a couple of hours

AppStore

• Main store for Apple phone
• Submission requires an .ipa file along with a description, screenshots and additional information about the game
• If the submitted file is more than 100mb it will be only downloadable with wi-fi
• Generally the submission period is 48 hours
• The games are tested by Apple and the regulations are relatively strict
• YOu can’t blacklist devices but you can limit support to a specific version of the OS

[Experiment] Test Case by ChatGPT

While taking this course I was also experimenting with ChatGPT (before it was closed in Italy) and I tought it would be interesting to ask it to create a test case for compliance with the App Store, the results were pretty good and could be used as a basis to create an official document or to insert the test cases in a dedicated QA software.

It's relevant to notice that being able to source the official guidelines from the AppStore website fits with the AI strenghts.

Test Case Document for App Store Guidelines on In-App Purchases using Sonic Dash as an Example

App: Sonic Dash App Version: Latest Version Platform: iOS

Test Case:

1. In-App Purchase Test

Objective: To ensure that Sonic Dash adheres to the App Store guidelines on In-App Purchases.

Steps:

1. Launch Sonic Dash and navigate to the in-game store.
2. Browse through the available items and select an item to purchase.
3. Initiate the purchase and provide payment information.
4. Verify that the purchase completes successfully and that the item is added to the player's inventory.
5. Check the receipt of the in-app purchase in the App Store.
6. Check the App Store listing to ensure that the in-app purchases are accurately described and priced according to the App Store guidelines.

Expected Result: The in-app purchases in Sonic Dash should adhere to the App Store guidelines and accurately describe the items and their pricing.

1. Restore Purchases Test

Objective: To ensure that Sonic Dash provides an option for users to restore their purchases.

Steps:

1. Launch Sonic Dash and navigate to the in-game store.
2. Select "Restore Purchases" option.
3. Verify that all previously purchased items are restored to the player's inventory.

Expected Result: Sonic Dash should provide an option for users to restore their purchases, and all previously purchased items should be restored to the player's inventory.

1. Parental Controls Test

Objective: To ensure that Sonic Dash adheres to the App Store guidelines on parental controls.

Steps:

1. Enable parental controls on the device.
2. Launch Sonic Dash and navigate to the in-game store.
3. Attempt to make a purchase that exceeds the parental control settings.
4. Verify that the purchase is blocked and that a message is displayed informing the user that the purchase cannot be made due to parental controls.

Expected Result: Sonic Dash should adhere to the App Store guidelines on parental controls and block purchases that exceed the parental control settings.

1. Purchase Verification Test

Objective: To ensure that Sonic Dash provides secure and reliable purchase verification.

Steps:

1. Initiate a purchase in Sonic Dash.
2. Check the App Store receipt to ensure that the purchase is valid.
3. Verify that the purchased item is added to the player's inventory.
4. Attempt to make a purchase with an invalid payment method.
5. Verify that the purchase is declined and that a message is displayed informing the user that the payment method is invalid.

Expected Result: Sonic Dash should provide secure and reliable purchase verification, and purchases made with invalid payment methods should be declined.

1. Subscription Test

Objective: To ensure that Sonic Dash adheres to the App Store guidelines on subscriptions.

Steps:

1. Navigate to the subscriptions section of the App Store listing for Sonic Dash.
2. Verify that the subscriptions are accurately described and priced according to the App Store guidelines.
3. Initiate a subscription purchase in Sonic Dash.
4. Verify that the subscription is added to the player's account and that the player is charged the correct amount according to the App Store guidelines.
5. Check the App Store receipt to ensure that the subscription purchase is valid.

Expected Result: Sonic Dash should adhere to the App Store guidelines on subscriptions, accurately describe the subscriptions and their pricing, and provide valid subscription purchases.