CI/CD for Test Engineers: How Deep Should We Go?

The Growing Importance of CI/CD Pipeline in Automated Tests

A few years ago, CI/CD was considered a nice-to-have skill for test engineers. Many relied on DevOps teams to set up and maintain pipelines, and it was common to hear statements like:

• "I know what CI/CD is."
• "My company has a CI/CD process in place."
• "Our DevOps team helps us configure pipelines."

However, the industry has evolved. Just as test engineers are now expected to have strong coding skills, they are also expected to design and maintain their own CI/CD workflows. CI/CD is no longer just an operational concern—it’s a core part of modern test automation. To stay competitive, test engineers may go beyond using CI/CD and learn how to architect it effectively.

Evolution of CI/CD Platforms

Previously, CI and CD were usually managed on separate platforms. For instance, Jenkins was used for CI to build and publish artifacts, while tools like Octopus Deploy handled CD workflows. This required additional effort to integrate multiple platforms for seamless deployment. Today, many companies have consolidated CI/CD into a single platform, such as Azure DevOps, simplifying management and reducing integration overhead. The decision to use a unified platform or separate tools depends on team standards and legacy infrastructure.

Where to Integrate Automation Tests in the CI/CD Pipeline?

Automation tests can be placed in either the CI pipeline or the CD release, depending on the testing needs:

• CI Pipeline: If tests focus on unit, integration, or early-stage validations (especially when test code is in the same repository as development code), they should be executed in CI. This ensures early detection of issues before deployment.
• CD Release: If tests focus on end-to-end (E2E) testing in a dedicated test environment, running them in CD is ideal. This allows tests to execute after all microservices have been deployed, facilitating comprehensive regression testing.

While this is a recommended approach, the actual implementation should align with project requirements. Here's what I've been using in different projects so far in recent years, hopefully it'll give some inspiration.

CI/CD Implementation for Automated Testing

Before diving into specific implementations, it's essential to understand the fundamental differences between VM-based and container-based deployments:

• VM Deployment: This traditional method involves provisioning virtual machines (VMs) to run automated tests. It provides a stable, controlled environment with direct access to system configurations and dependencies. VM-based deployment is easier to set up and debug but lacks the scalability and flexibility of containerized solutions.
• Container Deployment: This modern approach packages tests into lightweight, portable containers that can run in any environment. It enhances scalability and consistency while reducing dependency-related issues. However, container-based deployment introduces complexities in authentication, secret management, and Kubernetes orchestration, requiring a deeper understanding of DevOps principles.

VM Deployment (ADO) – .NET-Based (SpecFlow/Reqnroll, Selenium, RestSharp, Appium, C#)

CI Pipeline

• Define tasks using a YAML file.
• Choose a VM image.
• Acquire NuGet authentication.
• Select the appropriate .NET Core version.
• Restore dependencies, build the project, and copy build artifacts to a target folder.
• Publish artifacts for use in the CD release.

CD Release

• Use Azure DevOps’ GUI to define tasks.
• Separate automation tests into different stages (API, UI, Mobile, etc.).
• Define agent jobs, specify the agent VM, and select artifacts.
• Install dependencies and create environment variables for sensitive data.
• Deploy the test environment and execute tests.
• Generate test reports for stakeholders.

dotnet test "$(System.DefaultWorkingDirectory)/{}.AutomatedTests.API.dll" --logger:{Reporter} --filter:TestCategory={tag}        

VM Deployment (ADO) – .Node.js-Based (Cucumber, Playwright, TypeScript)

CI Pipeline

• Select a VM.
• Unlike .NET-based tests, no NuGet authentication or build tasks are required.
• Publish the test code as an artifact without executing tests in CI.

CD Release

• Define agent specifications and download artifacts.
• Install Node.js and dependencies.
• Install required browsers.
• Execute tests and store results.
• Configure environment variables for sensitive data.

Container Deployment (Docker, Kubernetes, ADO, ACR, AKS) – .Node.js-Based (Cucumber, Playwright, TypeScript)

As containerization gains traction, many teams adopt Docker to build images and Kubernetes to manage deployments. Test engineers need to align with these architectures to ensure automation tests integrate smoothly.

Benefits of Container Deployment

• Speed: Faster setup and execution.
• Scalability: Can scale tests efficiently.
• Portability: Ensures consistency across environments.

However, container deployment requires additional expertise compared to VM deployment.

If you are not very interested in the DevOps domain, the fact is that for automated testing CI/CD, VM deployment is sufficient for most projects.

As for how to implement container deployment, we need to configure a few things:

Key Considerations for Container-Based CI/CD

Container-based deployment introduces additional complexities compared to VM-based deployment. Several key areas require careful planning:

Container Registry Management

• Use Azure Container Registry (ACR), Docker Hub, or another registry to store images.
• Ensure secure authentication for pushing and pulling images.
• Configure service connections to enable CI/CD pipelines to interact with the registry.

Kubernetes Cluster Management (AKS or Equivalent)

• Kubernetes orchestrates containerized applications, requiring YAML-based job definitions.
• Secure authentication is required to pull images from the container registry.
• A Kubernetes secret may need to be created to store authentication credentials for image access.
• Pods must be correctly configured with environment variables and command-line arguments to execute tests reliably.

Secret Management with Key Vault

• Unlike VM-based deployment, where environment variables can be used easily, Kubernetes pods cannot directly access Azure DevOps variables.
• Kubernetes Secrets are only Base64-encoded by default but can be secured with encryption at rest, RBAC, and external secret stores like Azure Key Vault.
• Recommended Approach: Azure Key Vault Integration:
• Cluster-Level Integration: Enable Secret Store CSI Driver for secure, seamless secret access (requires cluster-wide configuration).
• Code-Level Integration: Directly access Key Vault from the test framework using: @azure/identity for authentication with DefaultAzureCredential, @azure/keyvault-secrets for retrieving stored secrets.
• Configure the necessary Kubernetes service accounts and Azure Workload Identity bindings to grant access to the Key Vault.

CI Pipeline

• Build the test automation container image.
• Push the image to the container registry (e.g., ACR).
• Publish YAML job definitions as artifacts.
• Create necessary service connections for secure registry integration.

CD Release

• Deploy test containers using Kubernetes.
• Authenticate Kubernetes clusters to pull images securely.
• Deploy secrets to Kubernetes via Key Vault or Kubernetes secrets.
• Execute test jobs and retrieve logs for debugging and reporting.

But deploying test containers is not the end of the process. We need to ensure the tests execute as expected and retrieve relevant logs. There are two ways to monitor the logs.

One approach is to add an additional task in the release to fetch logs from the AKS cluster; Another approach is to check logs directly from AKS by examining the pods created by the job, attaching a Sidecar to capture output or using tools, such as Lens/OpenLens, offering an intuitive UI to view logs and troubleshoot issues efficiently.

The last option is to implement health checks (livenessProbe, readinessProbe) and automatic rollback policies to handle failures.

Conclusion

For most automation projects, VM-based CI/CD deployment is sufficient. However, as teams migrate to containerized architectures, test engineers must adapt to ensure automated testing integrates seamlessly. Mastering CI/CD enhances a tester’s ability to contribute beyond traditional testing roles, making them valuable assets in DevOps-driven environments.