Craig Bradford's Posts

Since you’re reading this on the Distilled blog, you probably know Distilled has an SEO split-testing platform, DistilledODN.

One of the important features of the platform is taking care of the statistics and measurement automatically. Any time you set up a test, the platform will automatically:

• Create a forecast of expected traffic to the group of pages you’re testing on
• Create statistically similar buckets of control and variant pages
• Measure the organic traffic to the two buckets
• Calculate whether any difference in traffic is statistically significant

When we first built the tool, we used a measurement model that was based on the paper Inferring Causal Impact Using Bayesian Structural Time Series, which was published by Google.

Until now it’s not been possible to measure the impact of SEO and CRO at the same time. Today we’re proud to announce a new feature of Distilled’s Optimisation Delivery Network that we’re calling full funnel testing.

Our ODN platform launched with a focus on SEO testing. You have probably thought about this by comparing it to tools like Optimizely that allow you to do CRO testing. If you want to know more about how SEO testing works and how it’s different to CRO, you can read more in this post on what is SEO testing.

The trouble with just using one or the other is you don’t have any insight into how they impact each other.

Last week I tweeted an explanation of how we know that an increase or decrease in SEO performance was caused by a change that we made or by an external factor like seasonality, competitors, Google updates etc. People found it helpful and it generated a lot of questions so I thought it would be useful to post a more detailed explanation on what exactly SEO split-testing is as there seems to be a lot of confusion/misunderstanding.

One quick thing: This is deliberately a simple example with a basic explanation of the maths that we use. In reality, the maths is a lot more complicated and based on this research by Google: Inferring causal impact using Bayesian structural time series.

Have you ever been in a situation when your or your client’s traffic has dropped and you’re not sure what to do? How do you know if it’s a real problem or just seasonality?

Being able to recognise those true traffic drops, as opposed to red herrings like broken tracking or the aforementioned seasonality, is a critical skill for marketing consultants and in-house marketers. You’ve got to know when to reassure your boss/client, and when to suggest action.

This post will walk you through the process of confirming what actually happened and understanding why it happened in the first place.

The “what” - Was there really a drop at all?

Analysing historical traffic data

The image below is a real Google Analytics account. The client was worried that there was a traffic drop around October to December.

A couple of months ago I was on my way to work; it was 8am on a Monday and I was on a busy train heading to Waterloo station. I received an email that read something along the lines of:

“Hi Craig,

Blah blah blah, legal legal legal, please reply to confirm you are ok to go ahead with the searches and approve the payment of £X.”

My response was:

“Please go ahead.”

Although a simple enough email, writing it annoyed me. I had to open the Gmail app, click ‘reply’, type the response and hit ‘send’.

I realise this sounds petty, but we live in a world of speed, and that process isn’t fast. Typing on a mobile device isn’t fast. Fast would be clicking a button that says ‘approve’ or ‘confirm’ and not having to type at all. Fast would be an interface like Tinder that lets me answer ‘yes’ and ‘no’ by swiping left and right or a Google Now style card: