Better CRO: The best practice of creating a experiment document that will be the record of reference for all things having to do with a CRO test.

The Business Challenge

 Conversion rate optimization (CRO) is the process by which a given user facing experience, be that a web site, an app, or a marketing campaign, is studied for improvement. Once an improvement is theorized an alternate to the “control” or “the how it works now” version of the “control” is created. Some people call it the “challenger” experience and that is the term to be used in this document.

Let’s take a very simple look at what that might mean in business practice. Imagine a product detail page (PDP) and it has glossy pictures of the product, a blurb that gets to the value proposition of why the customer would want to buy the product in the first few sentences and an add to cart button. One element, the add to cart button, is green in the “control” experience, but some studies have shown that red, orange and yellow are colors relating to taking an action. Based on this information a “challenger” add to cart button is created in orange to test against the green button’s “control” experience.

One might conjecture that this sounds simple enough, what is the challenge at hand? Please take a moment to imagine such an experimentation program scaled up 10 times. That’s 5 tests running simultaneously and another 5 in development or QA awaiting a result from the 5 tests that are running. That’s the simplified story, in practice there will be experiments that get initial approval and then disapproval by stake holders, and a diligent CRO practitioner will always be creating new test ideas and presenting them to stake holders so the experimentation pipeline does not run dry.

It’s all got to be documented, and that’s what this article is about: how to create and manage “experimentation documents”, or “exdocs” for short, in order to create great reporting for stake holders on an experiment’s details, and save the CRO practitioner a great deal of time.

Naming Convention

 The first step in keeping an enterprise level CRO program nice and tidy is to create an agreed upon naming convention for all documents about, and references to, a given experiment. In this example a fictional large international airline will be used called “Pronto Air”.

Now Pronto Air has many lines of business, but, for the sake of simplicity, only two URL domains will be used in the example experimentation program and only one will provide “on page” examples.

The CRO practitioner’s remit is to improve conversions for two urls: prontoair.com (B2C passenger facing) and prontocargo.com (B2B site for air freight reservations). The naming convention starts broadly defining the domain to be tested, in this case, we need to differentiate prontoair.com and prontocargo.com immediately in the experiment document, or, exdoc.

So let’s start with the domain and build from there.

“PA” will represent the retail passenger facing site prontoair.com and “PC” will represent the cargo business web site. This naming example will only use the retail site as most readers will be familiar with the purchase process of a plane ticket. The theoretical exdoc has a pefix “PA” to define which domain is being experimented upon. Now let’s build it out.

Pronto Air has several divisions that are pretty siloed and almost treated as separate entities. Those divisions are hotel and car reservations, the loyalty program oriented pages, and gift cards. Since each division operates independently each will have a short hand, like the URL shorthand, for brevity of characters and ease of understanding. Experiments on the hotel and car booking pages will be called “HC”, the loyalty program will be shortened to “LP”, gift card related pages will be referred to as “GC” and, of course, the flight booking path which will be known as “FB”. Let’s choose FB, again, for familiarity’s sake.

So our exdoc name has grown a little. Now we have the domain (PA) and the business division (FB). The file name will start with “PA_FB”.

In this experiment it is believed that the seat selection tool in the flight booking path is causing confusion, and thus, drop offs from the purchase funnel resulting in fewer sales. The seat selector is not available for all flights, so it’s appearance on the page is dynamic depending on the flight selected. However the flight selector does appear on “Step 3” of the purchase funnel and that nomenclature is understood by all. For brevity let’s refer to step 3 as “S3” and the seat selector as “SS”.

The exdoc name has grown some more to identify the precise location of where the experiment will be taking place. It should now read “PA_FB_S3_SS”, or, colloquially, the experiment takes place on prontoair.com (PA) in the flight booking path (FB) on step 3 of that path (S3) and in the seat selection tool (SS). This may seem trivial but it is not. In a healthy enterprise level CRO program dozens, perhaps hundreds, of experiments will be conducted in a given year and if, say, one wished to look up all the tests that have occurred on the seat selector tool they need only search by “S3_SS” to bring up the relevant files.

Now for the friendly words, the user friendly words, that is. All that brevity of the file name, using 2 letter acronyms, is for a reason, because the user friendly part of the experiment name is going to bloat the size of the file name. Let’s say our hypothesis is that when users select their preferred seat, the seat changes color to indicate as much, but the shade is too light. For certain visitors and for certain hardware display settings it may leave the visitor wondering if the selection happened at all. So the experiment will test a more visible seat selection color, the “challenger”, against the current design, “control”. So our experiment is called “brighter selected seat”. Note that one only names the challenger and not the control.

Our file name has grown to “PA_FB_S3_SS_Brighter_Selected_Seat”

Lastly will come the date. The date should be changed as the experiment moves through the production process if the process results in a change to the test and the date will be fixed when the test goes live when there should no longer be changes. Some examples of that:

“PA_FB_S3_SS_Brighter_Selected_Seat_2020.12.25” date stamp reflects stakeholder sign off.

“PA_FB_S3_SS_Brighter_Selected_Seat_2020.12.26” date stamp reflects requested change to audience targeting.

“PA_FB_S3_SS_Brighter_Selected_Seat_2020.12.27” shows the go live date and at this point the file name is locked down for the length of the experiment.

Hoping that makes sense as it’s likely more than an individual wanted to read about experiment taxonomy.

 The problem Statement

 Every experiment document (exdoc) must contain a few fundamentals to identify the “why” of an experiment. The problem statement is one of those essentials.

The problem statement identifies what part of the user experience might be causing friction in a purchase funnel or another type of conversion. This statement can be gleaned by several methods, one of them is looking at web analytics, AKA quantitative data. By slicing and dicing the web analytics one can discover points of struggle for the customer to get to the next page or other goal you have set out for them to complete. Using the seat selector example, it has been noted that flights without the seat selection option convert at higher rates.

However, the best data is often found in non-numeric, qualitative data. Qualitative data consists of things like survey comments, customer service calls, and focus groups. Why is qualitative data so important? Because it’s a direct response from the customer. While the web analytics can show us that the funnel drop off is pronounced at the seat selection tool, it cannot say why. The all important “why” comes from people based qualitative data.

In the seat selection example, we have the analytics showing the funnel drop off at the seat selection tool, but no “why”. Going through dozens of survey comments user has commented “can’t select seat, made reservation on the phone”. Now we have qualitative data from the web analytics and a qualitative data point backing that up. It’s not that visitors don’t want to choose their seat (take that option away and watch the complaints flood in) it’s that they feel they cannot do so given the current user experience.

With that information our problem statement can be made: “Based on evidence of higher drop off rates for flights involving the seat selector and a user observation the seat selector is causing confusion and leading to high drop off rates.”

The Hypothesis

The hypothesis is quite simple, it’s the “what” of the experiment. Since the ending has been spoiled and we already know our hypothesis as described in the naming convention section of this article, we shall none the less repeat it here.

Hypothesis: “upon selecting a seat for a given flight, the color of the selected seat only changes slightly and may lack visibility, the lack of certainty in this step may cause users to switch channels and use the phone or abandon the purchase entirely”

The Mock Ups

“Make it look like this” is the best specification one can give a programmer. Instead of getting bogged down in long confusing descriptions and red boxes with arrows coming out of them indicating a feature’s new position as one might see in other experiment specifications, the best way is to “make it look like this” in a mock up design.

PIxlr.com is a great tool for creating mock ups. It’s free and it has most of the major functionality of Photoshop. If you don’t know Photoshop, pixlr.com is a great place to learn the basics.

Once the mock up is created it is placed in the exdoc along side of the control experience. In this manner a business practitioner can easily compare the changes to the current design. And you can tell the programmer “make it look like this”.

 The KPIs and KBRs

 It is always key to define the KPIs (key performance indicators) and KBRs (key business requirements) for each experiment. The KBRs are more global in nature, the KPIs are more tactical. For example, an ecommerce site that sells electronics might have a KBR of “increase take rate of protection plans” and KPIs of “purchased with plan”, “put plan in cart” and “visited plan page”. The KPIs are ordered by importance, “purchased with plan” is the main target for success, as it directly relates to the KBR. However, should the experiment not show any lift on the “purchased with plan” metric then we can look to the other KPIs for lift in engagement. Those data points will then be further used to determine what got the visitor interested enough to put a plan in their cart or learn more about the plans, but it the visitor’s curiosity didn’t make it to checkout.

 Technical Requirements

 The technical specs are largely for the programmer’s use although they could easily be of interest to other stakeholders. Since we already have our “make it look like this” mock up, the technical specs may not be necessary at all except for details of the changes to the control. Using our Proto Air seat selector example, the technical specs could be as basic as “Color change: #32a892” indicating the hex value of the color so the programmer is not left “guessing” what color might be intended in the mock up.

In some workflows a programmer might develop the code but not QA the work or put the code into an experimentation platform. In that case the technical requirements page would be the location for the code change along with instructions on where to swap it out in the challenger.

Other data that would be good to put here: any excluded browsers from the experiment, audience targeting details and so forth.

Experiment Data

 Once the experiment is up and running it will begin generating data after a short lag period. If you see no data coming into an experiment an hour after it has been sent live, there’s a problem and the test should be paused and the cause of the broken data feed to be investigated. If the experiment is taking data then care should be taken to disregard, or at least be skeptical of, the first 48 hours of data. Users leave old browser tabs open which could lead to a browser refresh which could cause the visitor to see the control on visit #1 and possibly the challenger on refreshed visit #2 making for a confusing user experience, the experiment challenger might not have replicated to all the content delivery networks (CDNs), and other reasons. Just be patient and wait for things to normalize.

A test may run for less than a week (not recommended) but it may have several weeks of data that needs to be updated and entered into the experiment data page. The experiment data page should be the master record of data for a given experiment. The CRO practitioner needs to tightly control the data. Using Pronto Air as an example, what if the New York Jets and their staff bought up all the first class and a big chunk of business class seats on a given flight to accommodate the players’ large physiques? Supposing the person booking is doing so in the challenger experience. That would lead to an outlier in the data making a test appear more successful than it really is. If one does not control the data false narratives and assumptions can become a problem.

If, using the Jets’ first class seat purchases as an example, when you remove the outliers (or any other data smoothing technique) for presentations to stakeholders, always note the filtering method on the experiment data page. Total honesty and transparency are critical for the experiment data page. As data comes in and is smoothed (if necessary) the page becomes the source of information and analysis for meetings and communications with stake holders.

 Experiment Conclusions

 The experiment conclusions page should be blank up until a given experiment has been taken down for underperformance, or, it is put in line for base code development to make the successful challenger experience the new control experience.  On this page of the exdoc, the CRO practitioner will summarize the findings from the experiment data page: special insights, trends, and to put the data into word format because some people just don’t like numbers.

 Next steps

The next steps page of our exdoc will take the information from the experiment conclusions page and iterate on the findings for possible next tests. To go back to our seat selection example test for Pronto Air it might say something like “the increased brightness of a selected seat had a positive impact on flight bookings, due to this finding it is recommended to revisit all CTA buttons and selection experiences in general. One interesting place to start would be….” The purpose of the next steps page of is to keep the program running with fresh test ideas and to communicate that this is an ongoing process.

In Summary

If the CRO practitioner is organized and follows the method above for making an exdoc, that practitioner will have everything the practitioner needs in one place for the full life cycle of a test and the testing program’s history. It will contain the problem and the hypothesis to ensure experiments are not chosen randomly, that there was logic to why a given experiment took place. It will contain the mock ups so it’s 100% clear to business users and developers what the changes are and how they will look on the page. The KBRs and KPIs page ensures that there is an understanding of overall business needs (the KBR) and that there is agreement to the KPIs chosen as the primary and secondary goals for a given experiment. The technical specs will communicate to development resources what to put in their code, or, conversely, what code a developer has created for the practitioner to enter into an experimentation platform. The experiment data page is perhaps the most important page. It will contain all the data, in numbered format, that shows how a test performed. It needs to be communicated clearly that the experiment data page is the master record and not the testing platform as test data sometimes requires refinement to get to the truth. The experiment conclusions and next steps pages will show that the test has provided insight (even if the challenger experience was negative on the KPIs) and the new learning, or learnings, will lead to a more informed testing program that stays on a steady pace.