What Might Be Next For (not provided)

On May 7th, Mozilla announced that Aurora (Firefox’s public beta) is beginning to roll out their HTTPS-by-default Google search behavior. It won’t be long before this is default in Firefox. As their announcement states, the user will be none the wiser (save for that tiny ‘s’ after http, and the secure lock icon). Webmasters, on the hand - let’s just say (not provided) is not making any friends.

I spoke on advanced analytics at #SearchFest back in February and the topic of (not provided) was on everyone’s minds. The question - “What can be done about (not provided)” - is and will remain inescapable in any analytics discussion. My advice now and at the time: ignore it. Any ‘hack’ is really just extrapolation of known keyword data, and was a pretty irresponsible calculation on which to be making business decisions. It’s dirty data. At the very least, worrying about (not provided) is probably just a waste of time and energy. The slides from that presentation and my writeup are here.

I tend to lean toward a pessimistic outlook of the future of (not provided). One in which Google eventually owns your keyword data, and the only way to get real, hard numbers is by buying AdWords. Which, by the way, Google had to create a workaround for the standard way https operates in order to still pass keyword data, as Danny Sullivan says (Caller ID, below, is the full referrer string that contains keyword data),

Let me be very clear. Google has designed things so that Caller ID still works for its advertisers, but not anyone else, even though the standard for secure services isn’t supposed to allow this. It broke the standard, deliberately, to prevent advertiser backlash.

But I’ve been thinking about a rosier possiblity that I thought I’d share. Before getting into it, I want to cover where we are now and where we’re likely headed in the short term. I think it’s also important to understand the motivations at play, which solidify my rosier possiblity (I hope).

So let’s start with:

The State of (not provided)

Remember this?
“even at full roll-out, this would still be in the single-digit percentages of all Google searchers on Google.com”
That’s Matt Cutts on record as the secure search default was rolling out. As it turned out, when the full roll out was complete, a spot check of 100s of Google Analytics accounts revealed that not one of the websites had under 10% (not provided). Distilled.net was sitting at roughly 20%.

So in November if 20% of Google searchers are logged in and using secure search, it wouldn’t be far off to assume that today that number would have climbed. After all, Google is certainly taking every opportunity to have you sign up for Google+ (and now Google Drive). More people signed in, more (not provided).

Not having run the numbers in a while I was surprised when on April 25, Mr. Casey Henry of SEOmoz tweeted:

Ew.

Sadly, Distilled was worse:

Yikes.

The ugly Distilled graph:

not provided graph for Distilled

So (not provided) is currently greater than 50% for both Distilled and SEOmoz. Granted, our users are likely a savvy-ass group, but that is a drastic increase. Now throw in Firefox’s subset, fast forward a year as Google continues to push Google+ signups… It ain’t pretty.

So,

Why, Again, was this change made?

Privacy, of course! Well, that’s what the Google PR would lead you to believe. There’s almost certainly some truth to it, but a few astute bloggers picked up on some possible ulterior motives that sounds pretty convincing to me. Danny Sullivan’s “Google Puts a Price on Privacy” is probably the definitive round up of motivations.

The strongest non-privacy reason for making this change has to do with Google’s bread and butter: ads. Before the change, 3rd party ad networks (AKA Google’s competition for ad space) that had inventory on a publisher’s site were privy to users’ queries. This allowed them to target and retarget ads across their network to match a user’s search history. Creepiness conversation aside, it _works_.

With keyword data no longer available for these ad networks, their relevancy and overall performance might suffer. It would be difficult for advertisers to target their ads at users who have shown a propensity to search for concepts that are closely related to their services. Relevance could be hard to come by. UNLESS, of course, you choose to use Google’s ad network: the only game in town with the data to know just how to serve your ad.

Another motivation: web analytics. There are a number of players out there, all of which have their own version of (not provided). The Google change does affect all analytics software the same, though. Whether its an on-page JavaScript or server log processing, Google’s “&q=” parameter in the referrer string is empty. No way around it. But how does this benefit Google? I’ll get to that in a bit.

The Future of (not provided)

Well, I think it’s fairly obvious that the Firefox change will lead to a modest bump, I’m guessing anywhere from 5 to 10%. Even after that, I would expect it to steadily climb as user’s are forced wooed into signing up for Google+ and other Google services.

But wasn’t there a rosy outlook?

Google Webmaster Tools

GWT has been pumping out updates and improvements over the past few months. The team recently revealed an overhaul to its dashboard, has improved the usefulness of server error reports, and has increased its communication with registered webmasters.

When the https change was first announced, Google was quick to point to GWT as an option for webmasters fretting about losing their keyword data. At the time, we weren’t thrilled with this idea, as GWT’s data search query data has proven to be rounded and sampled to the point of uselessness. But perhaps this is where Google gives back what hath been taken from webmasters by improving this data.

The SEO reports in Google AnalyticsThe “Search Engine Optimization” reports in Google Analytics that receive data from Webmaster Tools is currently pretty useless. I rarely utilize it. In this scenario, it becomes the go-to report for organic search marketers.

Obviously the flaw in this idea is one of the three motivations mentioned above for Google making the change in the first place: privacy.

But looking past that:

- How can Google provide the only ad network in town with real keyword data powering its targeting and retargeting? Turn off referrers.

- How can Google become the only web analytics provider with real keyword data? Turn off referrers. Pass keywords through to Google Analytics with GWT integration. GA Premium ends up looking pretty attractive compared to other paid platforms, no?

In the end, the only people unhappy: privacy advocates, 3rd party ad networks like Chitika and Chango, and 3rd party web analytics platforms like Omniture and Coremetrics.

And, what about that privacy thing? Ian Lurie pretty much nails it at Search News Central:

Don’t try to say this is a privacy thing. It. is. not. How exactly does this protect privacy, when you tie the text of e-mails to your advertising platform? How does this protect privacy when you’re photographing people’s streets, homes and whatever else you can lay your hands on?

Don’t get me wrong - I’ve not opposed e-mail ads, or Street View. But you can’t shut down search query data and then protest privacy. That’s like leaving one bite of steak on your plate and saying you’re a vegetarian.

Are you honestly telling me you had no way to deliver anonymous counts of keyword searches by signed-in users? You’ve never found a way to do this? ‘Cause that sounds like a load of horse hooey, if ever I’ve heard one.

Me, too.

This is all my opinion, not Distilled’s, and I could very well be dead wrong. For the sake of our keywords, I hope I’m not.

I would love to hear your take below or on Twitter.

Get blog posts via email

20 Comments

  1. Any confirmation that GA premium Provides the data otherwise lost to the (not provided) bin for their free offering? I've heard speculation and assumption but never anyone that could verify it one way or the other...

    reply >
  2. Mike Pantoliano

    Kris,

    No web analytics package is privy to (not provided) data. Including GA premium.

    reply >
  3. Barry Adams

    Pretty spot-on, Mike. It's only going to get worse. I do disagree with you on the uselessness of finding ways to extrapolate from (not provided) keyword data. In specific circumstances when analysing sites on a page performance level it can be quite useful. Especially as the share of (not provided) data starts to grow.

    reply >
  4. Jacob

    Superb article, thanks.

    It's nice to see someone coming out and saying that analyzing 'not provided' is quite dubious. This isn't mathematics, it's not like you can truly solve x.

    I guess we still have Bing, Yahoo and Ask referral data ... I prefer clouds with bigger silver linings(!)

    reply >
  5. this has been a massive problem for us - so does the GA premium version not have the same issue?

    reply >
  6. Awesome article, and GREAT quote from Ian... I echo that. It's like Google randomly decides when to take privacy seriously. The odd thing to me is, if Google allowed advertisers (us) to see this data instead of displaying the wretched (not provided) keyword pool, Google would probably make more money due to higher CTRs derived from more keyword insight. It boggles my mind why in THIS case, Google is prioritizing privacy over more dough in their pockets... Nice post Mike, you've definitely struck a chord with a lot of fellow SEMers out there including myself.

    reply >
  7. Good sum up mike, funnily enough just before I read this article I just downloaded 26,000 keywords across 402 websites from analytics (using @anakyticscanvas) these are for SME websites, the figure.... 7.56% (not provided)

    reply >
  8. An astute perspective. however I don't see a link to where I sign up $and pay$ for the new Google Analytics PLUS account... the new one that provides 'extra' features like "search queries" ;-)

    reply >
  9. When (not provided) was rolled out, I was pretty sure about Google's plan that they will gradually shut down viewing search queries at all from GA, the time is not far away when you will see 100% (not provided) queries. If this happens and assumes that you are not an adword user, Let's see how it affects.
    1) You couldn't find which query provides you huge traffic r.
    2) You can't find out which pages ranked on irrelevant organic queries.
    3) Other metrics that are part of CRO, e.g. bounce rate, avg. time on site, page/visits, % new visits etc, can you figured out about these metric on most convertible queries?

    reply >
  10. I've always been a big supporter of Google and their innovation. They have, after all, made my life easier and from Google Apps to Google earth I've always signed up. When Google brought out Wave I tried it and kept pretty quiet about it's inadequacies and subsequent big flop. When they brought out Google +, I blogged about it and promoted it to all my non-marketing friends to help them take the battle to Facebook. When "In the Plex" was published I bought, it, read it and passed it on to friends.
    I like many marketers out there have always championed Google and been a member of their unofficial fan club.

    This is diminishing and I'm rapidly losing faith. Are they bothered? Obviously not!

    Great article Mike

    reply >
  11. Mike

    Really great article, yesterday we were wondering with some clients the whereabouts of "not provided" , and I find that your description in this article, is clear enough so to answer at least a couple of questions. Maybe not the whole picture, would love to be able to predict future... but clear enough. Anyway, future in this industry is so quick that we could find a completely different scenario in a couple off years.... maybe months?

    Thanks again.

    reply >
  12. I find it much more likely that they are pushing advertisers into adwords (their existing cash cow) than GA premium. There is a lot more money in adwords and look at how they have hosed the worth of organic rankings over the years. This is just one more way to make SEO (read unpaid) less valuable.

    reply >
  13. I think it also, conveniently, helps Google by further destabilising the SEO environment and reinforcing that PPC is the way to earn quantifiable returns online.

    SEO was always hard to monetise. Now the ranking factors are more unstable than I can recall, there's no solid rankings to appease clients and you can't even say how much traffic came through keywords.

    The recent changes in Adwords (e.g. automatic accounting for misspellings/plurals, Quality Score breakdowns) may make things easier but they also reduce an advertiser's capacity to demonstrate his skill, and lead to a number of points of optimisation becoming homogenised across competitors' accounts.

    Net result: More people advertise in Adwords + More homogenised Adwords accounts = bid prices rocket.

    My 0.2 obviously...

    reply >
  14. So about a week ago I read a story in the New York Times about the FTC going after Google for abusing their monopoly of search. Here is the article if you are interested. http://www.nytimes.com/2012/04/27/technology/google-antitrust-inquiry-advances.html?_r=4&pagewanted=2#
    Of all the things I have seen Google do this is the biggest item where it seems like Google is purposely making decisions to push their competition out of the picture- this is applicable to other online add agencies and to other analytics packages. The other things will be harder to prove in my opinion but on this one case it seems they have stepped over the line and are abusing their power.

    reply >
  15. Great article a lot of good points. I agree with Barry there are lots of things that can be learned from (not provided). I do agree that there must be away to pass keyphrases data anonymously.

    reply >
  16. Hi Mike,

    You say:
    "GA Premium ends up looking pretty attractive compared to other paid platforms, no?"

    but then ...
    .... "No web analytics package is privy to (not provided) data. Including GA premium"

    I know that GA Premium is affected by "(not provided)" (see http://goo.gl/m31O9), and that there are some advantages (unsampled extracts being one of the), but I just wondered what you meant by GA Premium "looking pretty attractive"?

    Cheers
    Roland

    reply >
  17. Mike Pantoliano

    Roland,

    Indeed, no web analytics, including premium, has access to (not provided).

    However, in the scenario I discussed above with (more accurate) GWT data flowing into Google Analytics, GA Premium would become the only enterprise-level web analytics with full keyword data.

    reply >
  18. Mike,

    Do you have any idea when this more accurate WMT data is coming? Or if it is?

    Also, is there much word on the entry price for GA prem?

    Thanks.

    Rob.

    reply >
    • Mike Pantoliano

      Robert,

      It may never come. This was a hypothetical based on some educated guessing.

      Also, I've been hearing upwards of 10k per month for GA Premium, though I think its largely customized depending on the amount of data being tracked.

  19. I tend to work with a lot of tourism-related websites, whose audience is a pretty wide cross-section across the population. I tend to see (not provided) percentages from 10% to 20% these days, but trending upwards a bit. It is almost always at the top of the list, but I still have plenty of data to draw out some conclusions and recommended actions.

    reply >

Leave a Reply

Your email address will not be published. Required fields are marked *

*
*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>