How to reduce UI-Test tests running time from several hours to a few minutes

This is about how to port UI/Selenium tests to AWS Lambda and to reduce running cost and running time and thereby reducing the probability of failures. In some cases all tests were developed using Selenium and run on an AWS EC2 node. But this method is highly prone to failures since deployments/changes may happen while tests are running since it takes a lot of time (6-12 hours) for a full test suite to complete its execution. There is a high probability that an ongoing deployment in the same environment may pause/stop/terminate a UI test suite that is running. In most cases tests are run daily or after a feature deployments. Any system that run for a long time without necessary supervision is prone to failures.

I am using python here and same can be done using Java also.

Diagram:

Step1:

Download Chromedriver, Headless Chrome and Selenium

Make sure you download correct versions. This is very important for the components to work together. Below given versions are subjected to change upon newer releases.

Chromedriver: 2.41

Headless-Chromium: v67-69

Selenium: 3.141.0

Now AWS environment needs to be set. Access to following AWS components are required

• Lambda
• S3
• AWS Systems Manager
• IAM Role

AWS systems manager is to trigger a command in ec2 instances. For this to work either ec2 should have amazon linux as its base image version or aws systems manager should be installed separately in case of any other linux flavour running in ec2. Amazon linux comes with aws systems manager pre installed.

AWS lambda have only basic libraries of programming language. In the case of python it have packages like os and io but third party packages needed is missing. This can be addressed by adding layers to lambda functions which act as libraries. A lambda function can have up to 5 layers. Here, below components are added as layers.

• Chrome Driver
• Headless Chromium
• Selenium
• Custom Libraries

Selenium can be bundled with custom libraries as a single layer also as all are python packages. What needs to be taken care while adding python packages as layers is the directory structure. For python packages, following directory structures should be followed. If directory structure is incorrect, then packages cant be imported. Python version should be same as the runtime selected for lambda function. It is recommended to select single runtime in lambda. Here runtime is python 3.6. Packages can be installed with pip with --target parameter also and then zipped and uploaded as a lambda layer. Here packages are installed with pip install <package> and then zipped.

Chromedriver and Headless Chromium are also zipped separately. All zipped files are uploaded to aws lambda as separate layers. Use the layers option of lambda to create a new layer and upload compressed files. Once all layers are created up to 5 such layers can be added to a single lambda function. By default compressed files in layers are extracted to /opt mount point.

Once layers are ready, S3 and database setup should be done. This includes creating an s3 bucket and creating an IAM role to grant permission on s3 bucket to lambda function. Same should be done for database also. Note that aws entities like lambda and s3 already have a permission system and iam role mapped to it at the time of its creation. This role can be edited to grant permission to the specified resource.

Once all the above are set, test case can be run on lambda. Following is a sample test case

import os
import boto3
from selenium import webdriver
from selenium.webdriver.chrome.options import Options

# setup chrome headless
chrome_options = Options()
chrome_options.add_argument("--headless")
chrome_options.add_argument('--disable-gpu')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--single-process')
chrome_options.add_argument('--disable-dev-shm-usage')
chrome_options.binary_location = '/opt/headless-chromium'



# create boto3 client to interact with s3
s3 = boto3.client('s3')

browser = webdriver.Chrome(options=chrome_options,executable_path='/opt/chromedriver')

def lambda_handler(event, context):
    # this is a simple test case
    browser.get('https://google.com')
    search_button = browser.find_element_by_name('btnK')
    assert search_button.text == 'Google Search'
    # save screenshots using driver.save_screenshot()
    browser.save_screenshot('/tmp/scrnsht.png')

    # upload file to s3 whenever needed or let selenium  
    # handle upload when tests fail
    s3.upload_file('/tmp/scrnsht.png', '<s3Bucket>', '1.png')

Now sample test case can be run on lambda. After test case is fired following issues may be observed. Issues and solutions are listed below.

• Timeout - To fix this, increase timeout value to maximum possible which is ~15 minutes. In most cases lambda functions wont take more than 5 minutes.
• Access issues between lambda and s3/other aws services - As mentioned in steps above make sure lambda have access access to s3 using iam roles and policies.
• Insufficient Memory - No direct error may say that memory is not sufficient. Instead of raising any memory related warnings/errors lambda function might timeout. Try allocating maximum possible memory (~3GB) to lambda function.
• Chrome binary not found - Make sure to assign path to headless_chromium browser to variable chrome_options.binary_location as in the example code.
• Chrome not reachable - Make sure versions of headless chromium, selenium and chromedriver are compatible with each other.
• File/Bucket not found in s3 - Create a bucket in s3 with name same as one specified in code.

Now setup is complete and test cases can be executed on top of aws infrastructure. Note that pytest can be used to execute tests once pytest package is compressed and added as a layer to lambda.