A/B testing | Optimizely wins and fails

There’s possibly some stigma attached to the idea of Conversion Rate Optimisation. The name alone is a little daunting and then there’s all those other acronyms floating around the search landscape; SEO, RCS, WTF!? It’s hard not to get caught up in a wash of web analytics. Put simply, conversion rate optimization helps you to sell more online by giving your customers exactly what it is they are looking for. Patrick McKenzie summed it up well in last year’s SearchLove presentation when he explained CRO with a charming anecdote about his wife;

’having asked one woman to marry me and that one woman saying yes, gives me a 100% conversion rate for the proposal’.
Here at Distilled, we’ve been running some tests to try and improve our conversion rates for the DistilledU landing page. We felt it would be interesting to share not only the tools we used to do so but also, what results we saw [or didn’t in some cases] and how we went on to improve this.

So, ‘what is DistilledU?’ you might ask



Born out of an idea of building on our culture, DistilledU is an online platform for marketing training that we launched in beta, June 2012. The thinking behind such a program? We hire smart people and want to help them get to the top. Personal development beats development plans any day so why not use this as a recording of learning SEO “The Distilled Way”. So why make this platform available to you guys? External paying customers will drive us to build the best thing we can.

The most important thing when trying to improve your conversion rates from your website is to understand what your users most want from your service. We’re constantly looking for your feedback on the product and have set up this Public Roadmap to track the latest modules as well as getting input directly from our community on what they’d like to see next. But we’re not just looking for your suggestions, we also want to use this feedback and turn it into action to build the best training platform for online marketing we can. The vast majority of feedback we’ve had from cancelling customers is that either they’ve finished all the modules or they wanted more advanced content. Many also explicitly requested more video content.

So, what did we do? [We listened]

The first step we took towards improving conversion rates was to improve the value of membership as a whole by adding streaming video; full members to DistilledU are now able to stream over 80 hours of conference footage from their paid accounts including our most recent SearchLove event. Having added all of this value, you might think that all we would have to do to get the conversion rates benefits would be to simply mention video over on the landing page. Well, this is the actual process that we went through...

One of our consultants had been using Optimizely to prove the worth of some on site changes to his clients, so what better way to test out the new DistilledU copy [not to mention, a tool that requires absolutely zero coding or programming background!]

What is Optimizely and how do I use it?

Optimizely  Dashboard

Launched around the same time that the beta version of DistilledU came out, Optimizely is essentially an easy way to measure and test your website performance through A/B testing. The super easy interface lets you change any icon on the page —so you can move, resize, re-color, or even remove any element on an existing website in an instant. You can move site text higher or lower on a page using the application’s drag and drop technology. When you’re happy with the variation you came up with, give it a name and save it to your account.

DU experiment 1

By setting up a series of A/B tests through Optimizely, we were able to track the effectiveness of the new copy in terms of conversions and visits on site. As you can see from the above screen grab, there is limited space on the landing page above the fold and so we wanted to make sure the message came across in the most effective way, encouraging new sign ups and  new visitors.

All sounds good so far, right?

But, the seasoned Distilled blog reader will know that sometimes, CRO tests can fail and our first run of the new video copy certainly wasn’t the right one, scoring a huge 69% conversion rate below the original. After ten days of running this first experiment, we felt it was pretty safe to concede and take some time to reconsider what we could have been behind such a massive drop, despite the mention of a whole wealth of new advanced content.

It’s all semantics

Taking a look at the two different versions of the copy uploaded onto the site, it’s interesting to note the difference in the overall call to action.

Version 1 [without mention of the video]

‘Enrol in our online SEO university and master the art of search marketing. Learn at your own pace and in your own time, with lessons presented in an easy to digest format.’

Version 2 [with mention of video content]

‘Now with over 80 hours of advanced video content, sign up to our online SEO university and master the art of search marketing. Learn at your own pace and in your own time...’

In version two, the emphasis of the whole sentence now lies with the content [which is great] but does detract somewhat from the main call to action. Not so good for CRO but something that we certainly wouldn’t have anticipated without running the tests.

A second bash at the landing page copy with my initial Optimizely findings in turn leads me to this phrasing:


It’s now clear that we are inviting the user to sign up and try out DistilledU as well as later telling them why. A second look at the results and you can see the difference from the original.

After leaving this test running long enough to be sure, we rolled this change out to the live site and we’ve been delighted to see that February actually ended with a conversion rate 68% higher than January. We’re really loving how easy Optimizely is to use and would definitely recommend it.

Have you used Optimizely on your own site? How did you find the experience when it came to CRO or would you recommend another best practise? I’d love to hear your opinions in the comments below.

Cheri Percy

Cheri Percy

Cheri joined Distilled as a community intern and now heads up the Marketing department in the London office. She has co-ordinated and project managed some of Distilled's biggest content pieces to date and has doubled its social media growth.  When...   read more

Get blog posts via email


  1. WOW! I'd never heard of optimizely before but you better believe we're going to be incorporating it with our clients. What a useful article. Thanks!

    reply >
  2. Thanks for sharing, both your thoughts and the numbers.

    I was just wondering.. why do you start with testing a rather micro-level element, such as a few lines of webcopy?
    Wouldn't it be better to start with some major layout changes and then drill down to more specific elements such as colors, font-sizes, webcopy variations, call to action text, etc...

    Of course, the result more or less proves me wrong, since an uplift of almost 70% is a great result. But still.. I'm curious... :-)

    Would love to hear your thougts on this.

    reply >
    • Hello Sven,

      The hypothesis behind the micro level testing was that this was something that could provide significant value and wouldn't necessarily need ample time from the product developers or creative team i.e we could implement and test this change on site ourselves. In the long run, we do have a few larger scale tests lined up [namely on DU landing page with the introduction of a few new versions of the hero shot] so as to test these major layout changes alongside the copy discussed here.

      Hope that helps and thanks for commenting.

  3. Hi Cheri

    Did you do any issue gathering before you started (kissinsights etc)? - that's normally the first step in the chain. How did you know what was the best element to start testing?

    btw, I have been using VWO which has also been amazingly easy to use


    reply >
    • Hello Stephen,

      Thanks for your comment on here. In fact, our system is set up so that we capture any feedback from people through completions of DistilledU modules or through their cancellations; we used this feedback to offer up what a lot of people felt they were lacking, in this case more advanced content. This case study explains how we worked to build the video content into the DistilledU platform and integrate the announcement onto the landing page to drive more conversions.

      Hope that helps!

  4. Cheri , honestly i I didn't really know what optimizely can do , will definitely give it a try.

    reply >
  5. Not often you get a review of a product with some real and useful results.

    I've been looking for a tool for optimizing conversion rates and this seems a good choice.


    reply >
  6. Optimizely seems like a fantastic program for conversion rate optimisation. I don't know of too many programs that provide all these features combined with using such an easy interface.

    reply >
  7. Hey Cheri,

    Didn't know of Optimizely. I usually do Google A/B testing to test site pages and am ok with it. Is Optimizely as a tool worth the money?

    reply >
  8. I'll be grateful if you continue this in future. Many people will be benefited from your writing. Full Spectrum Lighting http://www.cuimianxiehui.com/forum.php?mod=viewthread&tid=55898

    reply >
  9. I'm doing some research on Optimizely for my employer right now. Did you notice any hit from page load and/or performance?

    reply >
  10. Hmm, it looks like Optimizely may be what I need to run some tests on my own site... Thanks for the article!

    reply >

Leave a Reply

Your email address will not be published. Required fields are marked *


You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>