In this meaty article I go over the bread and butter of testers, going from bug reporting, talking about how to do it and how to insert them in the development pipeline, to all the different testing methodologies that a good tester needs to know in order to apply them in the appropriate situation!
(Also a final addendum about the role of the QA tester in live ops)
Bug reporting flow
- It’s the main way of communicating between QA, coders and producers
- If it’s not optimized it might take too much time on administration
- If rules are not set, bug can take a life of their own, while being left in an unresolved status forever
- Potentially big problems can fall through the cracks
- The process is too heavy (too many stakeholders or steps)
- Process is defined very loosely and leaves room for interpretation
- Process is too stuck up, but the development process changes all the time
- No ownership of the process, everybody follows the process but nobody is responsible for it
How to setup good bug flow
- Analyze current flow (what is working, what isn’t)
- Decide what to keep and what to change
- Sit together with all stakeholders
- Appoint flow owner (usually a QA Lead or a producer) and make sure everybody is familiar with it
- Document the flow in some manner
- Setup the bug database in a way that fits your flow
Open (bug reported) -> In progress (assigned to dev) -> Done (Fixed by dev)-> Verified (someone checked that it’s actually fixed) -> Closed
- Reopened (after verification it wasn’t actually fixed, or a former invalid bug was reopened)
- Invalid (Maybe it’s not a bug, or it was tested in an old build)
- Not a bug (sometimes QA could open a ticket about a functionality that it’s more a suggestion than a bug, or there was a misunderstanding a bout a feature)
- Duplicate (We knew this already from another tester)
- Won’t Fix (the bug is small and we don’t have time to fix it, we can leave it)
- Incomplete (the tester didn’t report it properly with screenshots or with the build number)
- Works for me (the person that needs to fix the bug doesn’t find it when trying to fix it)
- Who can do what? (Most importantly changing a bug's status or making decisions about them)
- If it’s too limited the producer or QA Lead spend a lot of time just changing statuses for others
- If it’s too loose there is the risk of people who like to decide on everything by making executive decisions on bugs
- Roles are best decided tight to transition status
- Bug flow can often be the most problematic part of QA efforts
- From time to time it’s good to clear out bug repository by going through it with stakeholders to decide which are old/irrelevant/hanging
- All bugs should be addressed in some way (not necessarily fixed)
- Bug should flow through the system and not get stuck
- If you don’t know what to do with the bug, ask your supervisor
Bug flow implementation guidelines
- Make sure the entire team understands what each status means and that the bug life cycle is documented
- Make sure each individual understands their responsibility to each bug
- Make sure enough detail is entered when a status is changed
- Don’t entertain bug fixing requests outside of the proper bug flow
Bug reporting?
- It’s the main communication between QA, devs and producer
- Legacy: some issues could re-occur later while re-using parts of the code
- If incorrect they waste valuable time
- Mess up analytics about the maturity of the software
What does bug report include
- Good headline: it’s good practice to have a standard for naming bugs
- Version tested: very important, on which version are we testing
- Steps to reproduce and any known preconditions: describe in detail how you got the bug
- Screenshot, video etc.
- Repro rate: very important for crashes and freezes, how often do they happen
- Priority and severity
Mandatory vs optional fields
- ?FIelds in the bug repository could be mandatory or optional
- Reccomended practice is to decide together with all stakeholders which information is must have
- It’s easy to say that everything is mandatory but that can lead to delays or miscommunication
- If you can, link bugs to sprint or test cases
- Severity: What is the impact of the bug
- Priority: How quickly it needs to be fixed
- High priority is also usually high severity, the opposite is not necessarily true
- As a tester you are more likely to determine severity as priority is decided by producers who have a wider perspective of the whole project
How to determine severity
- What is the impact on the player?
- Does it have legal or financial consequences for the company
- Does it affect money/income in some way
- Is this a root cause for other high severity bugs
- Does it require more people and time to fix
- Will it impact submission to the platforms
- Will it affect company reputation or the game ranking
- A bug can be high severity even if it takes a second to fix and the opposite
- No need to have too many levels
- Usually 3 to 4 is enough
- Low-Medium-High-Urgent
- Too much info is better than too little
- Don’t presume the reader will know what you did
- When fixing complex bugs the coder will need as much info as possible, anything could matter
- Use straightforward language, it’s a technical report
- You want it as mandatory for bugs involving crash/freeze/kicking out of the game
- You don’t care about a 50/50 reproduction rate of a misplaced pixel, spend your testing time on more important bugs
- 1/1 doesn’t work as a repro rate, but it could happen on minor bugs
- Minimum of 5 attempts, better to go up to 10 if the problem seems fairly random, it will help the coder and in understanding severity
- Decide which versions are going under test, there could be multiple per day so you and the producer need to decide what to test
- Some could be used just to test a specific fix
- Always make sure you are testing the right version
- Make sure all versions are in the bug database
- Especially important on verifying previous bug fixes, it has to be done on the right version
- Common response from coders or designers
- If QA reports it, it’s probably a bug or it looks like one
- If a tester thinks so, a player could think that too, it should at least be better explained
- Decide on how to assign bugs before you start the testing cycle
- They can be assigned directly to the dev or go through a producer/ QA Lead
- Direct distribution makes it faster but not all team members might be comfortable with bugs or with their status changes
- Going through one person can create a bottleneck
- Be as objective as possible
- Focus on quality of bugs, not quantity
- If you have nothing more to test, change approach or stop testing
- Irrelevant bugs can cause tension in the team (unrealistic scenarios, splitting bugs in multiple issues)
- ?A bug is an actual problem in the software
- It’s fine to put forward a suggestion but don’t report them as bugs, it will only make devs angry
- Check with the producer how to make suggestions
- Don’t sit on bugs, report timely
- Decide who will report bugs, it’s good if the team helps with testing but make sure they know how to do it
- If you use external testing, check their reports, especially critical bugs
QA Methodology and Techniques
- Definition of methodology is not unified: a set of methods to execute tests
- Knowledge of QA techniques will help with selecting the right approach to testing and optimize your efforts
- It should enable you to work faster
- Risk analysis is done to determine what to test next
- Testing is prioritized in terms of probability that some feature will fail and the cost of said failure
- The greater the probability of an expensive failure the more important it is to test
- When we study a feature we should ask how it can fail
- Hands on approach with minimum planning and maximum test execution
- You create a test charter with a declaration of scope for a short time boxed test?
- You get test ideas to write down and use in later test cycles
- You learn about the product and about how it fails
- Fast attack on the game
- Exploratory testing usually done by experienced testers
- A tester spends a day testing an area that is usually ignored deeply
- If significant problems are found that are might get more resources later, changing the initial plan
- It has four attributes
- Must be realistic, something the player might do
- The test should be complex, involving several features
- It should be easy to tell if the game passed or failed, easy to argue
- If it has this four attributes it will be persuasive and yield bug fixes
- It takes time to create a good scenario test
- It’s not practical or possible to cover all possible permutations in the software
- In this technique you divide set of test conditions into partitions that can be considered the same (meaning you only test one value for each partition)
- Used a lot for data testing
- Partitions are generally “Valid” or “Invalid”
- While testing soft currency value the partition 0 to 1000 is valid, the partition of negative numbers or numbers higher than 1000 is invalid
- In this you test the boundaries between partitions
- Statistically bugs show up more in borderline areas
- You should test values at valid and invalid boundaries
- Also called range checking
- Based on the previous example you will test what happens if you spend 3 of your soft currency while you have 1, will it go into negative?
- Decision tables are used to model complicated logic
- Conditions are usually expressed as true or false
- Each column in the table corresponds to a rule in the business logic that describes the unique combination of circumstances that will result in an action
- The advantage is that you will detect combinations that would otherwise not been found and tested or developed
- Requirements become clearer and you realize that some of them are illogical (especially in early design phases)
Strategy for dealing with new code
- Start with mainstream level test, easy to pass test that will be taken as serious problems if failed
- Test broadly rather than deeply initially to understand what needs more focus
- Once a feature survives easy test think about more powerful ones
- Pick boundary conditions, otherwise there will be too many good tests
- Do some exploratory testing with new tests every week
Testing Approach
- What to test and when
- Obviously everything needs to be tested, but you can probably do it only one or a few times
- The best QA is always embedded and continuous, it’s cheapest and the most efficient
- For QA to work like this you need a good QA Lead who knows what to test and understands the different phases of production
- QA is a big part in all phases of development
- In Agile, QA needs to embrace the constant change and plan differently
- Continuous critical thinking
- Embedded QA requires more team work
QA role in each milestone
- Pre-Production - User stories, scalability, game design flaws, architecture (document inspection, decision tables)
- Alpha - Gameplay, fun factor, stability (selected devices, emulator)
- Beta - UI, game flow, UX, stability, FTUE, monetization, coverage (wide range of devices)
- Release Candidate - Balancing, localization, polish, shop, coverage, compliance (on supported devices)
Approach in live production
- Regression rounds - importance of communicating with coders, regression tests wear out with time
- You should ask, did you change anything significant with the code? Does the new feature affect older ones/performance? Would you like us to test something specific?
- If not with the coder, talk with your producer
- Integration round - using your experience and understanding of the game figure out how new features fit in the game ecosystem and how they could break older ones
- As QA you not only check if it works, you represent the player and should check how new features feel and if they are fun (and if older features are still fun)
- Some features look good standalone but don’t fit with an existing game
Black box vs white box testing
- Black box - Testing with no access or knowledge of the code
- White box - code testing (code review, technical review)
- Coders usually do white box testing, QA does black box
- Even if QA doesn’t do white box testing it’s good to know how it works and to ask for test results or a summary to help with your testing focus
- How do we plan to cover the product to develop an adequate assessment of quality?
- Good testing strategy is
- Product specific
- Risk focused
- Diversified
- Practical
Heuristic Test Strategy Model
- It’s a set of patterns for designing a test strategy, the purpose of the model is to remind testers of what to think about when they are creating tests
- Project environment - resources, constraints and other elements in the project that may enable or hobble testing, sometimes you challenge constraints and sometimes you accept them
- Product elements - things that you intend to test. Software is complex, take care to cover all that matters, not just what is easy to see
- Quality criteria - The rules, values and sources that allow you as a tester to determine if the software has problems. These are multidimensional and often hidden or self-contradictory
- Perceived Quality - The result of testing. You can never know the actual quality of a software product, but through the application of a variety of tests, you can make an informed assessment of it
- 5 dimensions of testing that you focus on while testing
- Testers - who does the testing, for example user testing is focused on testing by members of your target market
- Coverage - what gets tested. For example in localization testing, you are testing the implementation of different language
- Potential Problems - Why you are testing, what risks are you testing for. For example, testing for loading time duration using wi-fi connection
- Activities - How you test. For example exploratory testing
- Evaluation - How to tell whether the test passed or failed?
- Testing tasks are often assigned on one dimension, but you do the work on all five dimensions
- If you need to do feature testing, you know what to test, but you still need to decide who does it, what types of bugs are we looking for, how to test each aspect of the feature and do we know if it passed or failed
- By keeping all five dimensions in mind as you test, you might make better choices of combinations
QA and live ops
- In the mobile F2P world, QA never really ends, as they get frequently updated with new content
- Live ops are all development and marketing actions taken while the game is live
- QA practices need to adapt to fit live ops
QA in development vs in live ops
- Check everything - Check what is changed
- Enough time (maybe) - Limited time
- Test cases - No test cases
- Increasing towards the end of dev - Ongoing effort
- Full regression - Partial regression
- Can work alone - Must be part of the team
- Thorough - Quick
- You don0t know about player behavior - YOu know exact players patterns
- There are different type of updates: submissions and back end pushes
- Both need to be tested
- Yet sometimes the back end pushes skip QA (ex. game balancing), this can create problems
- Make sure you have a reliable environment for testing which replicates live
- Make it obligatory to go through QA
- Understand roll-back procedure (speed, ease, process)
- Work together with the development team, not against them
- Understand common risks of your game
- Make sure you understand them completely and how they affect existing ones
- Will they affect performance, loading time, build size?
- Will they affect game balancing and monetization?
- Will it make old features obsolete, confusing or not fun?
- How does it affect FTUE? Is it easy to understand for new players?
- Live environment is dynamic and it’s not beneficial to have to tight change control
- On the other hand, without a change control system you end up with releases that have not been properly tested (last minute bug fixes)
- As you get closer to submission slowly increase change control
- progressively more conservative set of freeze rules (UI freeze, partial function freeze, full function freeze, partial code freeze, full code freeze, showstoppers only)
- After deploy in live, do at least a basic live testing, you don’t want to hear it from the players
- When you find live bugs, assess how widespread it is
- Raise it immediately with the team, the producer will decide what to do with it (roll back, patch or hot fix)
- Address any severe live bug immediately, don’t sit on it, false alarm is better than no alarm
- Try to understand why the bug happens and what we can learn
When to cut off support for a device/platform
- if you notice that many happen on a particular device/os version, sometimes it’s better to cut it
- How to decide?
- Percentage of active players
- Percentage of buyers
- Specific market relevance
- What does it mean to cut off support? (no more online features/no more updates/ game completely removed)
Working with customer support
- CS is not QA, but they usually have to work closely together
- In live ops you end up spending a lot of time testing stuff brought up by CS instead of feature testing
- How to avoid it? when CS reports problems check:
- How many players are affected
- What device/platform is affected
- How quickly it escalated
- Include CS in dev early so they can understand new features and prepare FAQ on time
- Don’t be a hostage to angry players, if an issue affects only a few players that are very loud about it you have to prioritize other things
- Analytics is your best friend
- Generally they are keen to share their findings and reports
- Learn how data can help you on test focus and prioritization
- Very useful in determining your test set, understanding player behavior, considering cutting off devices
Who should work in live ops QA
- Working in live ops it’s fast paced and can be stressful
- Good live ops QA will be quick to learn and adapt
- Think outside the box
- Willing and able to work closely with the rest of the team, not a good match for lone wolves
- Good in balancing between shifting priorities in a short period of time
- Your external partners should have the same qualities
