Building An Automation Software Testing Team: Hiring, Evaluation, And Management

By some estimates, up to?100%?of software companies use automation testing in some capacity. There’s a lot been said about the importance and benefits of automation, including on our own blog. Plus, we’ve already talked about building an efficient?test automation strategy?and choosing the right set of?automation testing tools. Today we are going to talk about another core aspect that is at the heart of any test automation project: an automation testing team.

What does an automation testing team do, how to build one, and how to ensure its effectiveness throughout the entire automation project? We’ve got all the answers you need right here.

Who should do automation testing: a manual QA team, a development team, or a separate team?

Test automation is a unique subset of software engineering: it’s a piece of code that is created with the sole purpose of testing another piece of code. This is why many software companies feel like it’s only natural to entrust the process of automation testing to the developers. Other organizations go in a different direction by creating an automation team from scratch. And some companies choose to grow their own automation testing departments using their available manual QA resources. So which strategy is the right one?

The answer to this question depends on the company’s available resources and both short-term and long-term goals. For example, when the company is planning a long-term automation project that is expected to grow both in size and complexity, and there is already a skilled and motivated manual QA workforce, then helping them?switch to automation seems like a reasonable move.

The downside of this solution is that it takes time to acquire at least some knowledge of automation, and in the end, you’ll get Junior or Strong Junior automation engineers at best while still having to employ the services of a Senior Automation QA or a Lead. However, those can be arranged on an outsourced or freelance basis, so it shouldn’t become an issue.?

In the case of?software developers turning into automation engineers, those situations are fairly rare and may not yield the best results for one reason: after spending so much time honing their coding skills, software developers may unconsciously try and write code that is guaranteed to pass the tests, not discover as many defects as possible. Therefore, the results of testing can be biased.

All things considered, establishing an automation software testing team from scratch is probably the best option, provided that you have the resources and are confident that the scope of the project will keep your new automation QA department occupied long-term. You can hire an in-house team, choose to outsource instead, or use a combination of the two methods — more on those later in our article.

Typical roles and responsibilities in an automation QA team

The automation test team structure in software testing is not that different from the setup of a manual team, although the exact composition and responsibilities of an automation software testing team structure will still be different from manual QA. These are the must-have automation testing roles and responsibilities:

• Automation QA Lead. This is the person responsible for setting up the automation project and overseeing it throughout all the phases. An Automation Lead mostly deals with the big picture, such as identifying the scope of testing, selecting the framework and the tools, and ensuring smooth communication between every party, including the project stakeholders. It’s worth noting that the work of Automation QA Leads goes way beyond writing the code, even though they are definitely capable of it.
• Senior Automation QA. When the project is big enough, there is certainly room for both a Senior and a Lead Automation QA. However, sometimes organizations choose to have just one of the two. A Senior Automation Engineer also deals with the big picture, but a lot of their work also concerns the day-to-day execution of the project strategy, as well as helping junior members of the team get a full understanding of the task in front of them and the tools they need to use. Typically, senior engineers will also write code for tests.
• Infrastructure Engineer. This is a dedicated person for handling the technical details concerning the launch and progress of an automation project, such as the health of an automation test suite and the framework, as well as remote test execution. This position is usually reserved for large-scale and long-term projects, as there may simply not be enough for an Infrastructure Engineer to do on a smaller project. On a mid-sized project, this role can go to Lead or Senior QAs, and on larger projects, the DevOps department can handle these tasks.
• Junior and Middle Automation QAs. This is the biggest part of the workforce of a typical automation testing project. Junior and Middle engineers are involved in every stage of the project, from developing the test suite and setting up the automation tools to generating and analyzing the reports, using the data to make the testing process more efficient and resource-effective.

Other project team members may include a project manager, a product owner, and a software developer for handling code-related issues.

What’s the right size for an automation team?

When discussing how to build a software testing team that will deal with automation, the topic of team size is one of the most controversial yet most important ones. The reason why organizations care so much about the ideal team size is completely understandable: on the one hand, they don’t want an overblown staff; on the other hand, they don’t want to stretch their available resources too thin.

Unfortunately, there aren’t any formulas with a 100% success record that can help find out the ideal size or composition of an automation testing team — ultimately, it depends on the specifics of the project. However, many automation experts agree that having?one to two automation engineers per four or five software developers is an approach that usually works.

Of course, it’s also important to take into account the seniority level of the automation QA: the higher the level, the more challenging tasks they will be able to complete in the same amount of time. However, “one automation QA to four or five developers” is a formula that can help you get started on a brand new project and then adjust your resource allocation as you go.

In-house or outsourced: Which one is right for you?

As it’s often the case with outsourcing your software needs or choosing to do everything in-house, this question is deeper than it may initially seem and requires thorough consideration before any decisions can be made. Here are a few reasons why outsourcing all or some of the test automation can benefit your project:

• Niche expertise. When you are no longer limited to your local area in your hiring decisions, you get to work with engineers with a specific set of skills or one-of-a-kind expertise that can be rare to find in the industry.
• Financial benefits. Compared to the US and other Western countries, the work of an outsourced automation QA in Eastern Europe or Asia can cost you several times less for the same selection of services. Moreover, you don’t have to worry about office space, insurance, and other overhead costs.
• Flexibility. Working with an outsourced automation QA engineer or a whole team always happens on your terms, as you get to determine the scope and duration of the project, team size, and so on. You can even create a unique automation setup at your organization — for example, have the in-house team work for you full-time and outsource the Automation QA Lead position to an offshore engineer for 10 hours a week — which is rarely possible with a fully in-house project.

“Who is automation outsourcing primarily for? It’s an excellent solution for companies who don’t have the resources to set up an automation process from scratch and want to build mature processes instead of spending time and money on trying out different approaches and testing tools before finding the right one. Outsourcing also works well for organizations that need to automate only specific types of testing, such as load and performance, and simply do not need an entire automation department to work on a permanent basis.”

Maxim Khimiy, Automation QA Lead, TestFort

One of the concerns keeping some organizations from fully taking advantage of the outsourcing model is that managing and evaluating a remote team is not as easy as when everyone is working from the same premises. However, if the past three years have taught us anything in terms of job models, it’s that remote and hybrid work are only going to become more common and that different locations and time zones do not stand in the way of efficient management.

How to build an automation QA team that meets your needs

When building an automation software testing team from scratch or strengthening your existing setup with new team members, selecting and hiring the right people is a task of utmost importance. Here is how to develop high-performing teams for automation.

1. Create a solid vacancy description

The key here is to make the description as relevant to the job as possible, and not just to copy and paste the requirements from another vacancy. A difference in the requirements and duties between the job description and the actual interview will create a lot of confusion and damage the candidate’s opinion of the company. This is why it’s always a good idea to collaborate with someone who is actually working on the same project to outline the description, as well as ensure smooth two-way communication between the departments in case the project is to be launched from scratch.

2. Allocate enough time for hiring

Assembling an automation team from scratch or getting one or two strong additions to the team is not something that happens overnight. Things can get even more challenging when you are looking for senior-level engineers or automation leads, as they are no longer motivated to frequently switch jobs and are typically pretty happy where they currently are. The two things more likely to make them change their mind are an interesting tech stack and higher compensation.

The rule of thumb is that the more seniority you want to see in the position, the more time it will take to sign the contract. On average, it takes around a month to hire one senior automation engineer. It’s also worth noting that a currently employed candidate may need to fulfill their existing contract requirements, which can take another two to four weeks.

3. Know what to look for in an automation engineer

A competent automation QA combines both hard and soft skills that make them good at their job, and you can usually tell about some of those skills being present or missing already at the screening stage. Here are some of the things to pay attention to in an automation engineer’s CV:

• How many years of experience they have in total, how many years they’ve spent on average in every organization, and whether their experience in the last workplace is a logical continuation of their overall experience;
• How detailed the description of the last projects is and whether the type of product, platform variety, tech stack, and toolset match the requirements of the project;
• Whether there are any inexplicable gaps in the professional experience;
• The overall look and feel of the CV: it doesn’t have to be overly polished or overly creative, but it does need to be well-written, informative, and free of mistakes purely out of respect for anyone reading it;
• Whether the candidate goes into detail when explaining their role, responsibilities, and accomplishments on the project, or simply lists the basic project specifications that may not even fully match their seniority level.

If you want to get a better idea of the candidate’s personality, you can also check the section where they list their qualities and hobbies outside of work. However, it’s only natural for people to try and present themselves in a more positive light whenever possible, so take those words with a grain of salt.

4. Complete the interview stage

Ideally, the interview — especially the technical one in case the company uses multiple interviews to assess one candidate — should once again be conducted by a person with immediate knowledge about the process and the tech stack of the upcoming or current project. That way, the company will be able to choose a candidate with the most relevant skills and experience. But since an interview is a two-way process, it’s also important to give the candidate a realistic picture of what the project entails and what is expected of them.

5. Assign the leadership role

Any automation testing team, whether it has two, five, or twenty members, should have a clearly identified leader. And it’s not always the person with the most seniority or the person with the longest CV. It should be the person who is experienced in building test automation from scratch, maintaining it, and training other members of the team. Moreover, this should be a person with a realistic idea about automation in general, thinking of it as a helpful tool rather than the ultimate answer to every quality-related question or a?complete replacement for manual QA?in general.

6. Consider bringing additional specialists on board

Many organizations want to go the lean route when establishing an automation QA department, but the truth is that it’s nearly impossible to build high-performing teams for automation without eventually adding niche specialists on board.?

This can be a principal developer with a strong background in methodology, a project manager with experience encompassing software development and quality assurance, an infrastructure engineer, or a person who will deal with setting up the reports and processing them for maximum efficiency.

7. Lead by example

Whether every member of your automation team is well-versed in their craft or there are some junior members who need time to master automation as a whole and the project specifications in particular, the best way to launch an effective automation QA operation is to go all-in.

Identify the scope of tasks for each member, assign the people responsible for different aspects of the project, tell your junior team members who they can turn to in case they have questions, and launch the project, showing how it should be done.

“To me, one of the red flags is when the candidate is changing companies every year or so — this can be alarming for different reasons. I am also wary of candidates who apply for Senior-level positions while listing skills and qualifications that are too mundane for their advertised professional level, such as putting HTML or Chrome DevTools as their go-to tools or listing regular tasks, such as “Increased test coverage”, in the achievements section. These are things that are simply too basic for an engineer who claims to be on a high enough level.”

Maxim Khimiy, Automation QA Lead, TestFort

Automation team culture: How to create a good one and why it matters

Team culture is not always the number one idea in the mind of a person running an automation department, and it’s completely understandable: with so many technical matters to take into account and so many goals to meet, culture risks becoming an afterthought. So does team culture matter for automation QA, what does it entail, and how to maintain it at the right level?

The important thing to note here is that within an automation testing project, team culture is more about sharing the same work ethic and approach to solving problems than about sharing the same interests outside of work or having a similar sense of humor, although those definitely help in establishing a good rapport within the team.?

When you build high-performing teams for test automation, team culture can be fostered in the following ways:

• Knowing the end goals of the automation project and which results will matter the most;
• Fully believing in the importance of automation as the key way to achieve those goals;
• Demonstrating firm leadership but also knowing when and how to delegate tasks;
• Promoting the idea of constant learning and mentorship among the team members;
• Creating an environment where people can communicate and exchange ideas openly;
• Gradually lowering the amount of control as the project progresses successfully;
• Locating the ideal balance between speed and quality without compromising one or the other;
• Adopting the practice of stopping for a moment to get a bird’s eye view of the project;
• Making an effort to get to know each other not just from the professional, but also from the personal standpoint;
• Encouraging the team to spend time outside of work, even in a virtual setting, but not insisting on it;
• Establishing effective communication across teams — most importantly, with the software development and the manual QA departments;
• Avoiding the desire to make the team completely homogenous, as variety and thinking outside the box fuel innovation and drive progress.

“One of my own best practices when launching a new automation team is communicating as much as possible in the early stages of the project, including on non-work related topics. It’s even better when the team has the opportunity to hang out together outside of the office and make everyone feel included. From a professional perspective, I try to always have extensive documentation that outlines the principles and organization of the work process, project milestones, communication schedule, and other parameters concerning every team member.”

Taras Oleksyn, Head of AQA, TestFort

How to manage an automation team for maximum efficiency

“How to develop high-performing teams” is one of the key interests for thousands of organizations globally, including ones that deal with automation. When you assemble a dream team of automation QA engineers, the last thing you want is to have them not live up to their full potential due to poor management or wrongly distributed tasks. Here are a few tips on how to make the most of your automation software testing team.

1. Choose roles and set responsibilities correctly

There are few things more discouraging to a team member than having to deal with a task that doesn’t match their area of expertise or seniority level, and this can go either way: a Junior team member may be overwhelmed by an overly complex task, while a Senior team member may experience burnout from having to repeatedly deal with tasks below their skill level.

This is why any automation QA project needs to have an experienced Lead in the role of a manager. The Automation Lead will be able to correctly identify the skill level and area of expertise of each team member and assign tasks accordingly.

2. Allocate time for training and team building

Whether you are starting an automation project with a team assembled specifically for this task, a team of manual QAs ready to take on some new responsibilities, or a team of software developers with an interest in automation testing, you cannot expect every single team member to just jump head first into a completely new work environment and thrive there.

Reasonable training time should be mandatory for any new automation-related project. This can be the time for the team members to familiarize themselves with the scope of the project, the objectives, the framework and the tools, and, more importantly, with each other. These are going to be the people working on the same project day in and day out, so make sure they know each other on a decent level.

3. Keep your team informed throughout the project

We’ve already talked about the importance of good communication within an automation team, and that involves communication not just between the team members themselves, but also with the management.?

The situation where each team member is constantly in contact with all of the stakeholders of the project may definitely be overkill. However, keeping your team in the loop at every stage of the project is integral for their complete understanding of the project specifics and their work towards the end result. This also includes investing effort into communicating the goals of the project to every team member in the very beginning, so that everyone involved knows what they are doing it for.

4. Don’t get too focused on metrics

There are different ways to measure the effectiveness of a test automation team, and various metrics are among the most commonly used ones. The exact set of metrics can largely depend on the project’s characteristics, although some, including the number of test cases automated, total test duration, number of user stories covered in a week/month, number of defects located through automation, are applicable to nearly every QA project.

At the same time, there is a risk of becoming too dependent on metrics. When the team’s leadership is all about the metrics and the team is aware of that, there is a real risk of the team treating metrics as the only way to know if they’re good at their job. Instead, the team members and the management should work together towards a bigger goal, such as the quality of the product overall.

5. Promote out-of-the-box thinking

The work of an individual automation QA engineer or the whole team should not be just a sequence of tasks needed to get from point A to point B with a predetermined selection of tools and solutions. While following standard procedures is definitely beneficial to a project to some degree, creativity and out-of-the-box thinking are at the heart of being a successful automation QA.

This is why the person managing an automation testing team should always encourage inventiveness and help the team members form their own vision. This is exactly how efficient, innovative automation testing engineers are born and how the automation QA industry is constantly moving forward, powered by the creative thinking of its players.

“I’ve found that most automation QA engineers fall into one of two categories: a pioneer and an executor. Both roles are indispensable on a project, but the ratio of executors to pioneers can be even more important than the sheer number of team members who play that role. You can usually tell whether a person is a pioneer or an executor already in the interview stages and can adjust the ideal composition of the team from that standpoint. However, people are also known to change their roles throughout the duration of the project, even due to some transformations in their personal lives.”

Taras Oleksyn, Head of AQA, TestFort

How an automation team can change and evolve with time

A test automation team is not some fixed mechanism that performs tasks in a robotic manner and only changes when someone with more seniority tells it to change. A well-established, well-functioning automation team will transform and evolve naturally as part of its course. The question is which direction to choose for the transformation.

Perhaps, the most straightforward one is adopting new technologies to the mix to make testing more efficient and all-encompassing. For example, ideally, a senior-level automation QA needs to:

• Know two programming languages — an object-oriented one, such as Java or C#, and a scripting one, such as Ruby, Python, or JavaScript;
• Have experience with automation at different levels — UI, API, Unit, etc;
• Be familiar with different automation platforms — web, desktop, mobile;
• Have both the desire to manage a team and knowledge of how to do it;
• Be able to act as a visionary on a project, not just as a person to execute other people’s vision.

It’s also worth mentioning that not every automation QA wants or needs to assume a senior or management role, just as not every manual QA wants or needs to go into automation testing. However, with adequate leadership within the team and enough opportunities for improvement, each member of the team can locate and execute a comfortable growth trajectory.

“In my experience, around 50% of manual QA engineers eventually become interested in automation, so the transformation from manual to automation QA is not a given. Another way to grow as a QA is to become a performance engineer: this position doesn’t require as much knowledge of coding but can still make a world of difference in the quality of the product.”

Maxim Khimiy, Automation QA Lead, TestFort

Final thoughts

When done right, automation testing can dramatically improve the quality of the product while saving time and other resources on getting there. However, the key element and the most important aspect of an automation QA project is the automation team you’re working with. There are different ways to set up the automation operation at your company: hiring an in-house team, outsourcing the whole project, or choosing a combination of the two.?

What matters the most is that each team member is there for the right reasons, knows the objectives of the project, and strives to become better at automation testing week after week. With that in check, and with talented management, the only way to go is up.