How to improve Optimizely

Welcome to a new series of 'How to improve'. In this series, I review various technology services and platforms and suggest how I would improve their offering, either in terms of user experience (UX) of the service or analysis of strategic direction.

First up is Optimizely.

Who are Optimizely?

Optimizely was founded by two former Google product managers, Dan Siroker and Pete Koomen. Dan served as the Director of Analytics during the Obama 2008 presidential campaign. While there, his team relied on the use of A/B and multivariate testing to maximize e-mail sign-ups, volunteers, and donations to raise more than $100 million in additional revenue for the campaign.

But optimization was hard — you needed technical skills and know-how to run even the simplest of tests. After the campaign, Siroker teamed up with Koomen to create a world-class optimization platform that was easy to use in an effort to provide a platform for businesses to be able to conceive and run experiments that helped them make better data-driven decisions.

Since completing YCombinator in the winter of 2010, Optimizely has seen strong and growing demand.

What is my verdict of Optimizely's service?

I love Optimizely. They have a free offering but also have a solid enough platform to support enterprise clients too. They have achieved some really great feats in their time. If you are only going to do one thing to improve your website for your visitors, I would highly recommend AB Testing - Optimizely offer a world class service for this.

.... but, as Optimizely will also tell you, every company has opportunity for improvement. Here are some of the things I would change about Optimizely to improve their offering;

Improvement

1. Submitting support tickets

A little while ago, Optimizely made the decision to move away from supporting clients via email and instead directed customers to their support portal. I understand why they did this; it makes managing support tickets far easier on their end. It also drives customers towards their community and academy areas, which help self-service and enables customers.

Some clients prefer to email support issues, rather than raise tickets though. When responses came back from Optimizely that they would no longer read support tickets raised via email, it was disappointing (although I may add that I do still get responses via email). 

Improvement - Openly accept any form of contact from customers reaching out for support.

2. Adding/creating goals

One of the most important aspects of using Optimizely is to add goals to experiments. This is a little painful seeing as their 'Experiment Goals' lightbox in their editor is quite small. Furthermore, they should have the 'add saved goal' and 'add new goal' buttons at the bottom of the lightbox next to the 'close' button - not at the bottom of the scroll frame. 

Improvement - buttons for adding goals should be outside of the scroll frame and next to the close button, to make use of available space more effectively. They should also increase the size of the lightbox.

3. Mobile app

Anything which prevents users from needing to launch a new web browsing session, logging into a portal and fumbling with a full website is a good thing in my eyes. It would be very useful if Optimizely had an official mobile app. I have been made aware that enthusiastic developers outside of Optimizely have built an app for iOS users, but what about us Android folk? Get behind mobile Optimizely, we all know it's here to stay! 

Improvement  - Build an official app for iOS/Android to allow users to check in on their tests and edit test settings. Bonus points for integrating your blog on it, your support forum and other useful content.

4. Editor animations

The editor is very simple to use for customers with little or no development experience. This is perfect. For those who are looking to get a little more juice out of Optimizely, but still don't possess developer skill, it can be frustrating. One thing I would love to see Optimizely offer in the editor is animations. Wix.com offer this in their WYSIWYG editor and it works perfectly. It allows users to add a variety of animations to any content on the page.

Improvement - Add Wix style animations within Optimizely's editor to help non-technical users get more juice out of their experiments when setting up tests. It would also be great if Optimizely's developers could add coding macros to the editor. This is a one off job for them which would benefit many users multiple times.

5. Editor primary goals

You can set goals in Optimizely's editor, but you can't set a primary goal. The user needs to launch the experiment, go to the results screen of the experiment, then set a primary goal. This is something which should be done when setting up the experiment, not at the results view.

Improvement  - Add ability to set primary goal in the editor.

6. Editor negative goals

Currently you can set positive goals in Optimizely. This is where you try to produce an uplift which is reported as a winner. What if you're trying to lower the interaction with content though, such as clicks on an Email opt out box? If you conducted a test to decrease clicks on an opt in box and it was successful, Optimizely would report it as a 'loser' and also state 'Original is currently beating variation', even though your variation is really winning.

Improvement - Have option in editor when adding goals to set the goal as a negative goal.

7. Automatic results per segment

Optimizely reports the overall results of your experiment. You have the option to then segment the results by a variety of different user conditions (Tablet, UK, Returning visitors etc). This is very useful, but I would love it if Optimizely automatically ranked how the test variation performed for all the various segments. This would provide greater clarity of results and not pass the onus on the analyst to check this information.

Improvement - Maxymiser does this one well; show the results in a long list and how the variation performed vs each segment.

8. Anonymous Competitor benchmarking

I had an insight review from one of Optimizely's insight analysts. It was very useful and I got a lot of value from the session. One thing that piqued my curiosity was some anonymous competitor benchmarking. This encouraged me to build an optimisation road map and notably ramp up my AB Testing activity.

Improvement - When new customers join Optimizely, get permission from the customer to use their data anonymously, in return, offer them quarterly updates on how their activity ranks with their anonymous competitors.

9. Offer development resource as a service

Optimizely's strategic goal is to get as many people AB Testing as possible: it's growth. This makes perfect sense to me and is very fitting with how companies get noticed in modern 'millennial generation' times (focusing on growth much more than profit). I agree with this approach, but no doubt one of the most common issues Optimizely customers must face is a lack of development resource to implement the more complicated tests which clients want to do. 

Improvement - I'm aware Optimizely offer preferred partners for development work, but it would be great if Optimizely hired staff internally whose only job is to code experiments for their customers, at a price, as a service.

10. Offer developer Optimizely training as a service

Developers client side normally have to go through the process of reading lots of material to learn how Optimizely works to feel confident enough to create functional experiments. This process could be improved and monetised.

Improvement - As above, the developers hired to code experiments for customers could also provide technical training to developers, at a price, as a service.

11. Show how heavy each test is

Even though the Optimizely tag can load asynchronously, best practice states it should be the first thing to load on a site and to load the tag synchronous, to avoid page flashing. The more experiments you run and the more goals you set is the larger that overall Optimizely tag will be. This can slow down the website load time and impact visitor user experience ('UX'). Optimizely show how big the tag is in bytes to help raise awareness. This is buried somewhat in their 'settings' tab on their dashboard. Website speed impacts user experience.

Improvement - Show how big (Kb) each test is as you are building it, so you know which tests are contributing most to the tag increasing in size.

12. Reuse custom views on the results page

I often create custom views of Optimizely results on the results page. My issue is, I'm forever re-creating the same custom views, as Optimizely doesn't allow me to save them for later use in other tests.

Improvement - Allow use of recently used created custom views for results page.

13. Result page navigation

When I have several tests running, I like to check in to see how they are getting on. I need to launch each result page for each test individually. This is a little annoying and seems clunky.

Improvement  - Be able to navigate to a different test result page, from the test result page, via the use of a drop down menu etc.

14. Archive test edit

In Optimizely you can edit a test from the overview page or preview it, but not from the archive page. This is annoying, as I need to unarchive it, launch the editor/preview page, close the page, then archive it again. I could accept why I can't load the results of an archived experiment if Optimizely told me why, but they don't. Below is normal vs archived view of the options for an experiment;

Improvement  - Allow archived tests to be edited directly from the archive screen, or use tool tip icons/messages to explain why this can't be done.

15. URL match validator

A very handy tool in Optimizely is the URL match validator feature it offers in the test URL targeting. This  to create goals (same as URL targeting)

Improvement - I would like to see whether pageview type goals work with query strings and various regular expression. I could use tools outside of Optimizely which help such as a regex validator, but it would be great to have this from within the goal creator in Optimizely.

16. Test checklist

When creating tests, it can be easy to forget something; 'Did I set up the integration to Universal Analytics?', 'Did I set the correct traffic allocation', 'Did I serve the test to the right audience?' etc. Ensuring I remember everything I need to do is a burden that Optimizely can take off my mind.

Improvement - Allow the user to create a checklist anytime during test setup, so that when they're done creating the test, they can blitz through the checklist and ensure they have done everything they need to do.

Lastly... 

Personalisation Strategy

For those of you who don't know, Optimizely are stepping into the world of personalisation. They are developing a service which will help deliver relevant experiences to users. See more from Dan Siroker's Keynote speech at Opticon.

This is very likely the right direction to go in, as I personally believe there is more potential here than in optimising for Mobile or Multi-Channel. This service will also complement their existing product very well.

I have no suggestions for improvement for this strategy and wish them the best of luck with this service. I'm keen to see what they come up with! (cough cough - Optimizely: invite me to the beta!)

Closing Thoughts

@Optimizely - I'm keen to hear what you make of my suggestions; do you find them useful? Are you working on any of them currently?

If you enjoyed this post...

Please let me know by hitting thumbs up at the top of this post and/or sharing it with others to read - thanks! More of the 'How to improve' series coming soon!

Over to you!

As always, it's your turn - Do you have any ways you would improve Optimizely? 

Keep in touch!

Connect with me on LinkedIn or follow me on Twitter!

Other Posts by Daniel

• 25 reasons your homepage is failing
• How to improve your website in 4 hours
• The 10 Conversion Commandments
• How to make more money online straight away
• How to get insight quickly using Google Analytics
• The 1 problem of optimising conversion rate (CRO) - and the solution!
• How to use data to improve customer experience online