Continuous Testing using JetBrains TeamCity Server

As Automation Tests becomes integral part of software development pipeline, continuous testing is an very important skill for an aspiring tester. In this article I am focusing on a simple goal for beginners looking to learn CI/ CT skills - "How to integrate a test framework to a pipeline created in TeamCity".

Continuous testing is a process of executing software tests in a software delivery pipeline provide and reduce risk associated with the release candidate code. The saying goes 'An effective way to improve quality assurance (QA) is to "fail fast." This is truly achievable with the different kind of tests suites we write and with tools that help run these tests continuously.

These tests can be Acceptance Tests, EndtoEnd Tests, UI tests, API Tests, BDD Tests or performance Tests. The goal here is that the respective test framework should run on every PR merge to a given branch and generate pass/fail reports that can be stored and accessed in Team City. The success of a Continuous Testing depends on Test Framework that is good ft for the pipeline.

Generally a software release pipeline will contain couple of environments not limited to Dev, Test, Stage/UAT and Production.

When a developers commits code to a shared repo or to a master environment, ideally the code is build and application is deployed to the Dev environment and then Unit Tests are execOnce all the Unit Executed. QA environment is made available where functional Automation Tests are run and Pass/Fail. reports are shared. Below are the list of steps to integrate a suite to TeamCity and enough information for testers to understand on how to setup your own pipeline.

Step 1: Test Framework Project

For this article we will going to consider a functional automation suite testing the open Json placeholder APIs as our Application Under Test. The End to End Test Suite is developed using C#, specflow, html reports following a BDD style of Automation. The two important things to setup are to setup a startup script or an out-of-the-box script to trigger the tests in Command Line Interface. Secondly, to setup a reporting capability that can be consumed by the build tool, could be xml, json etc.

Step 2 :Setup team city 

 Download team city from here: https://www.jetbrains.com/teamcity/download/#section=section-get and follow instructions provided to setup your own cloud team city account.

Step 3 : Some important concepts you would need to know in team city 

Build Agent: This is important to understand. There could be a lot of build agents available. Build Agent is a piece of software that actually executes a build process. They can have different platforms, operating systems and some pre-configured environment that you may want to test your software. You can also run your tests in different kinds of platforms and check the results. 

TeamCity Server: As developers integrate their work into a version control, this server is responsible to distribute them to all the Agents. I guess the server handles all the upgrades to agents automatically. Not much to worry about here. 

Build configuration: These are set of settings that define the build procedures. You would need to know where your code base resides, how do you run your tests from a CLI, environment parameters, any build trigger you want to define, build artifacts and any other advance configurations etc.

Step 4 : Understanding Basic workflow 

The TeamCity server detects a change in your VCS root on the PR merges. Team city server finds a idle compatible build agents and assigns a build agent. The agent executes the Build Steps. While the build steps are being executed, the build agent reports the build progress to the TeamCity server sending test reports to the server and available on the build step for us to check the failures. Now let's create a project.

Step 5: Create a Team city project and build configuration

Click on New build configuration.

Define version control setting

Click on the the source repo , provide your credentials and the path to where the code to the automation tests resides.

Setup up your test runner

Step 3 /3 : run tests is where you would choose an appropriate test runner for the tests to run. In our case we choose .net

Some good tips here are

• when you click auto detect build steps, it would detect if its .net or. maven build etc
• you can add any number of build steps
• all the steps that would run are in sequence, you can change the order of them to reflect the order of tests you want to run
• You can define dependency on build steps
• Other configurations such as test case filters and any other command line triggers can be defined here. For example if you have a shell script that needs to be triggered for reports can be defined as a last step of this test run

Define parameters

Environment variables can be defined here.

Build features

Adding xml reporting is a good build feature.

With all the above step, setup is done. Click on DSL to view the script.

package _Self.buildTypes

import jetbrains.buildServer.configs.kotlin.v2019_2.*
import jetbrains.buildServer.configs.kotlin.v2019_2.buildSteps.dotnetBuild
import jetbrains.buildServer.configs.kotlin.v2019_2.buildSteps.dotnetTest
import jetbrains.buildServer.configs.kotlin.v2019_2.buildSteps.nuGetInstaller
import jetbrains.buildServer.configs.kotlin.v2019_2.triggers.vcs

object TcSpecflow : BuildType({
    name = "TC-specflow"

    vcs {
        root(HttpsGithubComKirthiraj12tcSpecGitRefsHeadsMaster)
    }

    steps {
        nuGetInstaller {
            toolPath = "%teamcity.tool.NuGet.CommandLine.DEFAULT%"
            projects = "TestRepoSpecflow.sln"
        }
        dotnetBuild {
            projects = "TestRepoSpecflow.sln"
        }
        dotnetTest {
            name = "run test"
            param("dotNetCoverage.dotCover.home.path", "%teamcity.tool.JetBrains.dotCover.CommandLineTools.DEFAULT%")
        }
    }

    triggers {
        vcs {
        }
    }

    features {
        feature {
            type = "xml-report-plugin"
            param("xmlReportParsing.reportType", "mstest")
            param("xmlReportParsing.reportDirs", "dir/**/*.xml")
        }
    }
})

Step 6: Execute the Build

Click on the Run option on the right corner of the build setup, this opens a small window.

You can choose the Agent you want to run the build against and click on RunBuild.

Build History

Once the tests are Run, build history is shown as below.

Reports /Artifacts

Click on one of the build runs, you will see the test results and build logs and other artifcats

Click on Tests tab on a recently triggered build, you will see all tests pass/fail history. Some good features I like in team city tests reports are

• Time duration taken for every test run
• Test run history. Very helpful check which regression test failed with which build commit.
• Test build logs. Very useful if you add loggers to your tests, you can see request and response for every test.
• Generally build logs are huge, team city provides option to go to the section of logs from the build step.

Artifacts tab: Click on artifacts tab, here all the artifacts generated by the tests can be found. This is especially useful if you coded for html reports and the index page of this report can be directly shared to the shareholders.

Option to download test results in excel format. For my project, I had to provide excel reports as test results and TC provided a one click option to download these reports.

Build logs

Conclusion

As part of my cloud learning journey, one of my goals is to learn how to setup continuous testing pipelines using different integration build servers. Knowing how to setup pipeline for testing purpose and providing my requirements to DevOps has greatly helped Clients enable the DevTestOps pipeline faster and adopt CT practices with ease. You can view quantifiable results in

• Increase in your team’s productivity
• Shorten time to delivery
• Increase application reliability in production