A/B testing | Optimizely wins and fails

There’s possibly some stigma attached to the idea of Conversion Rate Optimisation. The name alone is a little daunting and then there’s all those other acronyms floating around the search landscape; SEO, RCS, WTF!? It’s hard not to get caught up in a wash of web analytics. Put simply, conversion rate optimization helps you to sell more online by giving your customers exactly what it is they are looking for. Patrick McKenzie summed it up well in last year’s SearchLove presentation when he explained CRO with a charming anecdote about his wife;

’having asked one woman to marry me and that one woman saying yes, gives me a 100% conversion rate for the proposal’.
Here at Distilled, we’ve been running some tests to try and improve our conversion rates for the DistilledU landing page. We felt it would be interesting to share not only the tools we used to do so but also, what results we saw [or didn’t in some cases] and how we went on to improve this.

So, ‘what is DistilledU?’ you might ask



Born out of an idea of building on our culture, DistilledU is an online platform for marketing training that we launched in beta, June 2012. The thinking behind such a program? We hire smart people and want to help them get to the top. Personal development beats development plans any day so why not use this as a recording of learning SEO “The Distilled Way”. So why make this platform available to you guys? External paying customers will drive us to build the best thing we can.

The most important thing when trying to improve your conversion rates from your website is to understand what your users most want from your service. We’re constantly looking for your feedback on the product and have set up this Public Roadmap to track the latest modules as well as getting input directly from our community on what they’d like to see next. But we’re not just looking for your suggestions, we also want to use this feedback and turn it into action to build the best training platform for online marketing we can. The vast majority of feedback we’ve had from cancelling customers is that either they’ve finished all the modules or they wanted more advanced content. Many also explicitly requested more video content.

So, what did we do? [We listened]

The first step we took towards improving conversion rates was to improve the value of membership as a whole by adding streaming video; full members to DistilledU are now able to stream over 80 hours of conference footage from their paid accounts including our most recent SearchLove event. Having added all of this value, you might think that all we would have to do to get the conversion rates benefits would be to simply mention video over on the landing page. Well, this is the actual process that we went through...

One of our consultants had been using Optimizely to prove the worth of some on site changes to his clients, so what better way to test out the new DistilledU copy [not to mention, a tool that requires absolutely zero coding or programming background!]

What is Optimizely and how do I use it?

Optimizely  Dashboard

Launched around the same time that the beta version of DistilledU came out, Optimizely is essentially an easy way to measure and test your website performance through A/B testing. The super easy interface lets you change any icon on the page —so you can move, resize, re-color, or even remove any element on an existing website in an instant. You can move site text higher or lower on a page using the application’s drag and drop technology. When you’re happy with the variation you came up with, give it a name and save it to your account.

DU experiment 1

By setting up a series of A/B tests through Optimizely, we were able to track the effectiveness of the new copy in terms of conversions and visits on site. As you can see from the above screen grab, there is limited space on the landing page above the fold and so we wanted to make sure the message came across in the most effective way, encouraging new sign ups and  new visitors.

All sounds good so far, right?

But, the seasoned Distilled blog reader will know that sometimes, CRO tests can fail and our first run of the new video copy certainly wasn’t the right one, scoring a huge 69% conversion rate below the original. After ten days of running this first experiment, we felt it was pretty safe to concede and take some time to reconsider what we could have been behind such a massive drop, despite the mention of a whole wealth of new advanced content.

It’s all semantics

Taking a look at the two different versions of the copy uploaded onto the site, it’s interesting to note the difference in the overall call to action.

Version 1 [without mention of the video]

‘Enrol in our online SEO university and master the art of search marketing. Learn at your own pace and in your own time, with lessons presented in an easy to digest format.’

Version 2 [with mention of video content]

‘Now with over 80 hours of advanced video content, sign up to our online SEO university and master the art of search marketing. Learn at your own pace and in your own time...’

In version two, the emphasis of the whole sentence now lies with the content [which is great] but does detract somewhat from the main call to action. Not so good for CRO but something that we certainly wouldn’t have anticipated without running the tests.

A second bash at the landing page copy with my initial Optimizely findings in turn leads me to this phrasing:


It’s now clear that we are inviting the user to sign up and try out DistilledU as well as later telling them why. A second look at the results and you can see the difference from the original.

After leaving this test running long enough to be sure, we rolled this change out to the live site and we’ve been delighted to see that February actually ended with a conversion rate 68% higher than January. We’re really loving how easy Optimizely is to use and would definitely recommend it.

Have you used Optimizely on your own site? How did you find the experience when it came to CRO or would you recommend another best practise? I’d love to hear your opinions in the comments below.