Automated E2E testing of Web Apps on AWS using Amazon CloudWatch Synthetics
Quality is crucial to any software product success, because:
- it builds trust with your customers,
- results in fewer customer complaints,
- fuels recommendations.
No matter how good your manual testing QA team is, as your web application grows, the chances that your team overlooks more and more defects grow as well.
Defects, when found later, become more and more expensive to fix and can also negatively impact your users trust.
So if you think seriously about your product quality, you shouldn't forget about automated End-to-End testing. It will help you discover defects early and before your customers do.
If you are interested in running automated E2E tests from the AWS cloud, please meet Amazon CloudWatch Synthetics.
What is Amazon CloudWatch Synthetics?
Amazon CloudWatch Synthetics is a fairly new service that was introduced to the public just one and half year ago in November 2019. It extends Amazon CloudWatch by allowing you to setup monitoring canaries.
What are canaries?
Canaries are configurable scripts that regularly monitor if your web application is functioning and performing correctly in production-like scenarios - E2E workflows that you define.
Canaries can be also used for checking for unauthorized changes from phishing, code injection and cross-site scripting, the latency and availability of your endpoints, store load times data, capture screenshots of your UI.
How do canaries work?
Canaries offer programmatic access to a headless Google Chrome Browser via Puppeteer or Selenium Webdriver.
Selenium Webdriver is an open-source collection of APIs which is used for testing web applications to verify if they work as expected or not. It mainly supports browsers like Firefox, Chrome, Safari and Internet Explorer - it also permits you to execute cross-browser testing.
Puppeteer is a Node.js library which provides a high-level API to control headless Chrome or Chromium over the so called DevTools Protocol. It provides to you automated testing environment allowing you to run your tests directly in the latest version of Chrome browser using latest JavaScript and browser features.
Puppeteer is generally better choice when you have to perform unit level testing for any web application and a fast & flexible solution is required.
Selenium will be the better choice for the cases where we have a web and mobile application and cross-platform support is our number one requirement.
How can I create a canary?
Canaries can be written in either Python or Node.js programming languages.
The good news is that even if you do not have any prior knowledge of Node.js or Python scripting, you can still create and setup your own canary.
This is possible thanks to the 5 preinstalled blueprints that are offering you the helping hand with the setup some of the basic automated tests:
- heartbeat monitoring - runs a check if a webpage residing on single URL can load in browser,
- API canary - helps you monitor your APIs,
- Broken link checker - performs crawl to detect if any of the links on given URL is broken,
- Canary recorder - captures your activity in Chrome browser and records it as a workflow for you so you don't have to write the testing script,
- GUI workflow builder - allows you to define testing steps manually with just few clicks and generates testing script out of this for you.
Note: These blueprints are great for QA automation test beginners who are starting to learn how test automation works. The scope of what blueprints can help you test is rather limited. For more complex testing scenarios, learning either Node.js or Python becomes necessary.
Example - Canary that validates login procedure
In our example, to make things simple, we are going to create a canary using the "GUI workflow builder" blueprint. By doing it this way, "GUI workflow builder" will generate programming code of automated test on our behalf.
The canary will try to visit SauceDemo.com website and login into it using valid user credentials (Username and Password).
Steps to setup the canary:
- Navigate to the "Create canary" screen
- Choose "GUI workflow builder" from the "Blueprints" box
- Enter into the "Canary builder" box - Name: login_ok
- Enter into the "Canary builder" box - Application or endpoint URL: https://www.saucedemo.com/
- Enter following actions into the "Workflow builder":
Action: Input text Selector: input[id='user-name'] Text: standard_user /click on the "Add action" button"/ Action: Input text Selector: input[id='password'] Text: secret_sauce /click on the "Add action" button"/ Action: Click with navigation Selector: input[id='login-button'] /click on the "Add action" button"/ Action: Verify selector Selector: div[id='shopping_cart_container']
6. Scroll down to the "Schedule", choose "Run once"
7. Scroll down to the bottom and click on the "Create canary" button
8. Wait a minute and reload the screen in browser
If you have done everything right and our demo website is running fine, the canary will be executed and you should see "Passed" in the "Last run" column and 100% in the "Success %" column of the Canaries table.
Are there any statistics of canary runs available?
Amazon CloudWatch Synthetics offers detailed statistics about canary runs. In order to explore them, you need click on the respective canary in the table located on the Canaries page.
You will be able to see:
- list of issues related to canary runs,
- detailed information about the individual canary run with indication which steps have passed and which failed,
- if during the canary creation, you kept the "Take screenshots" option checked, you will have also the screenshots from each of these individual steps at your disposal (which is great for investigation when any of the steps have failed),
- HAR file capturing a web browser's interaction with a site (HTTP requests, response headers, response times..).
Frequently asked questions
1. Do I have to run canaries manually or can I schedule their runs?
In the example mentioned above, we configured our canary to run just once (right after it becomes created). We could then run the canary again manually how many times we wanted from the CloudWatch synthetics web console.
Should you need to run your canaries more often (e.g. to ensure continuous monitoring of your web application), you can configure them to run recurringly once every 1-60 minutes.
2. Is it possible to setup alarms so that I am informed if canaries fail to execute the defined workflow?
Yes, it is possible. For each of your canaries, you can setup CloudWatch alarms that can monitor the following metrics: SuccessPercent, Duration, Failed.
3. What about the canary pricing? Are canaries expensive?
Within the Free-Tier programme, first 100 canary runs per month are free of charge. Each additional canary run will cost you around $0.0012 - $0.0022 depending on the AWS region in which the canary runs. See the pricing page on AWS.
4. It looks like that running Synthetics canaries can become quite expensive. Is there any way how to optimize the costs?
Let's say we have 100 canaries and each of them setup to run just once per hour. In the month having 31 days, there will be 74.400 runs in total. As first 100 runs are free, the total monthly costs will be approx. 74.300x$0.0015 = $111.45 + costs associated with CloudWatch alarms + costs associated with storing screenshots and other metadata in S3 buckets.
I agree this can be much if you are paying it from your own pocket.
Of course, there are ways how you can optimize these costs. For example:
- you can group multiple canaries into a single canary (so instead of having 1 canary for testing the login procedure and 1 canary for testing the logout procedure, you can have 1 canary in which both login & logout workflows are included),
- you can setup canaries to run just once and trigger their runs not from Amazon CloudWatch Synthetics but rather from custom Lambda function. You can then setup Amazon CloudWatch rule to trigger run of your custom Lambda function. So instead of min. 24 runs / day which can be achieved via Amazon CloudWatch Synthetics, you can reduce it to few runs / day which may be sufficient for your use case (this is because scheduling using CloudWatch rules is far more flexible, you can set it up so that it triggers canary runs just every few hours per day).