How to build an experimentation team

E-commerce and digital businesses are finally waking up to the fact that experimentation can and should be the primary way they make all their important decisions.

Amazon, Google, Booking.com - they all do it and look where it got them!

Unfortunately though, many companies are stuck in the old days of 'CRO' (Conversion Rate Optimization) by which I mean the use of A/B testing in a purely tactical way to tweak a few landing pages and CTAs, often using external agencies or consultants who work in a black-box external to the business.

Amazon and Netflix don't 'do CRO' - they have research and experimentation at the very core of their culture and business, and they are where they are because of it. They have fully developed operations to support this way of life.

Whether you do this in-house or using one or more agencies, or a combination of all, there are a core set of skills needed to get it right.

However, to get started you don't need all of them, in fact you can start out with virtually none of them. Also, different skills doesn't necessarily mean different people. It would be hard but not impossible to find a single person who has some degree of all the skills you need.

The aim of this post is to explain the skills needed to build an experimentation function, but in such a way that you can see which are fundamental and which are nice-to-have or can be worried about later.

Introducing the skills map and its 3 main categories

The different skills required fall roughly into 3 main areas:

• Analytics & Research - the ability to listen to and understand customers using multiple different sources of data, translate the data into insight and ideas and generate hypotheses for experimentation.
• Design & Creative - the ability to both visualize and realize the application of a test hypothesis on the front-end user experience.
• Development and Technology - the ability to actually build and produce the designed experience, as well as support the implementation and maintenance of the various technologies required for the whole operation.

As we shall see, project management and/or operations is something which straddles all of these things.

The reason the map is divided into concentric circles is that there are certain aspects of these skill groups which are more fundamental than others, and so should be tackled first.

Project Management - The glue that holds it all together

As you will see, experimentation involves a lot of different skills coming together in unison. This can't happen without some kind of project management and central process to hold it all together and make everything run on time.

However, this does not mean you need a dedicated project manager, or even anyone who will call themselves a project manager, at least not to begin with. It certainly doesn't mean you need someone with Prince2 skills or anything like that.

All it means is that someone needs to take ownership for running the overall show and pulling together the different things which need to happen.

Most people start out with a Conversion Manager/Specialist or such, whose background is likely to be marketing, analytics, UX or a bit of all of those. That person is going to be doing other things as well, but they are going to also hold the role of project manager whether they like it or not.

A project management system aids this greatly, but that is the topic of a different post.

Analytics & Research skills - Essential

Analytics and research represents the ability to:

1. Listen to and understand customers using a range of different data sources and techniques.
2. Translate that data into insight and hypotheses for experimentation.
3. Measure and analyse the results of the tests statistically and understand how these results support decision making.

There are 3 core skill sets which are compulsory. Again though, they do not need to be different people when you are starting out:

• Statistics - An experiment is basically a statistical thing and not much else. You are looking at the result of a sample of data and trying to decide whether that same result would happen outside the sample (if you really did that thing) - this requires statistics. This is both the most common thing which people misunderstand and get wrong in testing, and also the most important aspect of it. Doing it wrong because of statistics is no better than not bothering. You don't need a statistician but you do need someone who is going to take the time to understand it all to the best of their abilities.
• Digital analytics - this generally relates to the ability to analyse and understand quantitative data, especially from tools such as Google Analytics, but possibly from a range of tools like heat mapping, session recording tools, survey tools etc. You won't get very far without someone who can process this information, however being able to translate that data into insight and ideas is a long way away from the ability to just analyse it, so hire wisely.
• User testing & research - I'm using this as a catch-all for the more qualitative aspects of research and analysis which require somewhat more human/emotional skills. If analytics understands what is happening, you also need to understand why, and that means talking to customers in more conversational ways. Again, possibly the same person to begin with.

Analytics & Research skills - next level

Eventually there are other skills which are going to be required:

• Psychology and behavioral economics - user research and testing requires some understanding of user psychology and behavior analysis, but to do it justice you need someone with a specific background in these areas. Understanding, in a given situation, the likely psychological triggers required to nudge towards a goal, is an art form. The better you can do this the more likely the solutions you design will result in success.
• Data programming and reporting - digital analytics tools work by putting a nice user interface over real data, so that a range of people can interrogate it without needing real data-analytics skills. You will, however, eventually come unstuck without these skills, especially when anything requires looking at customer data or other offline source of information. SQL, Python etc are data programming language skills which you will eventually need.
• Design thinking - if the outcome of insight and analysis is hypotheses and ideas, then design thinking is it's next level. I won't go into exactly what it is here but Google it for an overview. In addition, once experimentation is embedded into your culture you are going to want to apply the processes and methodology to many things, not just your website. Design thinking is your way into thinking about processes, products and other aspects of your business.

Design & Creative skills - Essential

When starting out with experimentation, you can get by testing a lot of ideas which require no design skills or development skills at all. A/B testing tools have WYSIWYG editors which allow anyone to make basic changes to content and copy on websites and, sometimes, the most basic changes can make enormous differences.

However, this approach will get exhausted pretty quickly and whoever is responsible for building tests is going to feel pretty frustrated by knowing what they want to do and not having the skills to do it. The first two things to get a handle on are:

• Copy Writing - on the one hand, there is a lot of evidence to suggest that consumers don't actually read very much of what you write on your website. On the other, seemingly small copy changes can have a huge impact on your sales and revenue. But writing is a skilled profession, especially when it comes to selling and the associated psychology of it. Getting this right can be very important for the success of the overall programme.
• UX Design - For more complex changes, and before hitting up a developer, someone needs to design the experience. For the most part, you are going to be modifying the structure and flow of an existing experience and will be able to get away with just a UX Designer producing some wireframes.

Design & Creative skills - Next level

Eventually you need to extend these skills with:

• Visual design - if you are designing something completely new (as opposed to modifying structure/flow etc) then your developer is going to want PSD files showing the exact final design, which involves more visual/graphic design work.
• Customer Experience Design & Design Thinking - you will miss a trick with experimentation if it only concerns itself with the web or app experience, and also if it only tries to improve the conversion rate. There are huge benefits to be had by applying an experimentation mindset to other areas of the business as well

Development & Technology skills - Essential

Most companies, when they start out with experimentation, will tend to use the WYSIWYG functionality of their A/B testing tools in order to make simple front-end changes. These tools work a little like a CMS and allow changes to be made with drag and drop style functionality.

However, you will quickly exhaust these tools and feel frustrated at not being able to do more complex things. This is where you are going to need a front-end developer, or at least someone with very good front-end coding skills.

There is also an important consideration to make here about the tools and technology you use for testing, because that will determine to some extent the complexity of what you can do. However that is for a different post.

There are two different aspects of development which it is important to consider for experimenting on websites and apps:

• Experimentation development - coding inside an A/B testing tool is still coding and primarily uses HTML, CSS and JavaScript. However, it is a very different experience for a developer who is used to coding in a production environment. Also, the QA and testing processes surrounding this kind of development are and should be somewhat different. Finding a developer who both can and wants to develop experimentation is pretty challenging. They also need to understand and appreciate the rest of the process of experimentation and the other skills required.
• Production development - once you have run a test which you want to then push live, you need to be able to get that pushed to the real production website. Most people have internal developers or an agency, but the important point to consider is the process, workflow and prioritization for this. If you are constantly giving your winning tests over to production and they never get done then the whole thing is pointless.

Development & Technology skills - Next level

Once you get more advanced there are a couple of other skills here which may become useful:

• Back-end development - particularly if you are using server-side experimentation, then there are a range of other more complex things you can test, and coding and tech skills beyond front-end become useful
• Technology implementation - some would probably argue this is essential and maybe it is, but most people can get quite far without worrying too much about their tagging and tracking, or they have an agency who can do it for them. But, eventually, this is going to a) drive you mad and b) limit what you can do in terms of insight. It's not just a case of coding to apply tags because someone needs to understand exactly what those tags are meant to do and where they should be sending data.

Summary

Most companies who want to in-house CRO tend to think that hiring a CRO manager will achieve that. It's a start but it unfortunately ends up with that manager feeling incredibly frustrated by not having resources to lean on to do all the myriad things which need to be done.

Planning out exactly how these skills are going to be tapped into when they are needed is an incredibly important first step towards getting to the kind of experimentation culture that Amazon and the likes have and to which everyone aspires.