Accelerating Prezzee Mobile Apps Testing with Maestro Automation: A Journey from Zero to 60% Coverage, Mono Repo Consolidation, and CI/CD Integration

Introduction:

Prezzee Mobile team recognises the potential for exploring advanced use cases and expanding their automated testing efforts to improve test coverage, and deliver high-quality mobile apps.

On 9th September 2022, Mobile team gathered for an exciting experiment during ?? FireFriday ??. We were eager to explore Maestro, a new automation tool developed by mobile.dev. The goal of the day was to test the tool's feasibility and determine its potential for automating mobile apps.

As a conclusion of the FireFriday, Mobile team found Maestro has revolutionised UI automation, making it simple, efficient, and enjoyable for the team. Its YAML syntax simplifies readability, eliminating the need for additional setup or extensive coding. This promising tool motivated us to invest more time in getting familiar with the tool.

The Starting Point: As Prezzee moving towards Quality Assistance Model and Shifting testing towards left Mobile team realised that relying solely on manual testing was becoming a bottleneck in development cycle. We faced challenges such as repetitive task, increased time-to-market and insufficient test coverage. Recognising the need for a change, the team set out to explore the world of automated testing.

Selecting the Right Tool: After extensive research and evaluation, the team chose Maestro automation as their go-to tool. It offers a range of benefits, including simple setup, declarative yet powerful syntax, built-in tolerance to flakiness, built-in tolerance to delays and fast iteration.

Learning the Ropes: To effectively leverage Maestro automation, Mobile team has devoted time and effort to self-learning and skill development. This investment has allowed us to enhance the proficiency in utilising Maestro's capabilities and stay up-to-date with the latest tools in automation. By continuously expanding the knowledge and skills, Mobile team can maximise the benefits of Maestro and optimise automation efforts.

Building the Test Suite: Mobile team started creating the first set of automated test cases for critical flows. When considering critical flows for test automation, it is essential to prioritise buying flows as they directly impact business revenue and customer satisfaction. Buying flows encompass the entire process of a user purchasing a product, from product selection to payment completion. Automating these critical flows offers significant benefits. QA identified critical workflows and scenarios via E2E Traceability Matrix which establishes a way to ensure that there are correct checkpoints on the test coverage for an application, prioritising areas where automation would provide maximum value.

Building the Automation for New Features: Mobile team includes test automation as an integral part of every new epic. Automation tickets are created and associated with each new feature, forming an essential component of the definition of done.

Continuous Improvement/Maintenance: Continuous improvement of the Maestro script is crucial due to potential challenges, such as the possibility of encountering incompatibility issues with newly released versions. Additionally, Maestro regularly introduces enhancements and new features that directly affect automation scripts, necessitating ongoing maintenance and updates.

Integrating Maestro into CI/CD pipeline: Integrating Maestro into a CI/CD pipeline involves setting up the Maestro cloud environment, installing and configuring the Maestro CLI on your CI/CD server, and defining scripts based on their priority. Critical scripts are designated to run daily on Master branch, ensuring that essential functions are validated regularly, while high-priority scripts run weekly to cover important but less critical features. By creating distinct Maestro workflows for each priority category and automating their execution with CI/CD jobs, we can ensure that critical processes are checked daily and high-priority ones weekly, optimizing both efficiency and reliability in your development workflow.

Consolidate iOS and Android scripts into a single script using a mono repo: To enhance efficiency and streamline development processes, we have started consolidating iOS and Android scripts into a single script using a mono repo approach. This allows for easier code sharing, consistent maintenance across iOS and Android platforms.

Celebrating Success: After eight months of dedicated efforts, the mobile app team achieved a significant milestone: a 60% automation rate on regression testing coverage. This accomplishment has been pivotal, as it significantly reduced manual testing efforts, leading to faster releases, improved productivity, and increased confidence in the app's quality. Buoyed by this success, the team has continued to focus on Maestro automation. We have set up the Maestro Cloud and integrated it into the CI/CD pipeline, further enhancing our testing processes. Additionally, the team consolidated iOS and Android scripts into a single script using a mono repo approach, streamlining development and ensuring consistent maintenance across both platforms. This success has inspired us to aim for even higher levels of automation in the future, setting the stage for continuous improvement and innovation.

Learnings:

• Invest in learning time: The best way to keep the whole team engage on a new tool is by letting the team explore the tool and come up with their own learnings and excitements
• Start small: Begin automation efforts by focusing on critical aspects of the mobile app before gradually expanding. Be prepared for code refactoring and potential discrepancies between iOS and Android during the automation process.
• Tagging: Running the whole automation suite might be expensive and take too much time to complete. Discuss with your team and come up with a priority matrix in order to run the critical flows as early as possible. Maestro allows you to tag each script allowing flexibility when running those scripts as part your CI/CD.

Conclusion:

The journey from zero to 60% automated regression coverage within eight months highlights the power of Maestro automation and the dedication of Mobile team. With Maestro's simplicity, we have integrated it into our CI/CD pipeline and are consolidating our scripts into a mono repo. We are excited about the future of our automated testing journey. We look forward to leveraging Maestro's features to optimize our testing processes, improve app quality, and stay ahead in the competitive mobile app market.

Example Script: change-region.yaml

appId: ${APP_ID}
---
# Load platform variables     
- runScript:
    when:
      platform: iOS
    file: ios/init.js

- runScript:
    when:
      platform: Android
    file: android/init.js

# Tap `change region selector` button
- tapOn:
    id: ${output.changeRegion.selectCountryView}

# Assert Change region bottom sheet modal is displayed
- assertVisible:
    text: Select your region
      
# Assert `change region selector` button is longer displayed
- assertNotVisible:
        id: ${output.changeRegion.selectCountryView}

# Select the targeted country from bottom sheet modal
- tapOn: ${CHANGE_REGION_COUNTRY}