SEO Isn’t Dead, But The Low-hanging Fruit is No More

by Mike Levin SEO & Datamaster, 10/02/2013

I am Mike Levin, the SEO… but not lately, not so much. This article will discuss that, and what you can do in light of the seismic shifts in SEO over the past few years. What are my credentials? I conceived, designed, programmed, and installed the HitTail SEO product—right down to configuring the servers and installing them in the colocation facility (long before the cloud could handle such an app) back in the day. It’s been sold and moved onto the cloud since. But it was my baby.

HitTail is a product whose stated purpose is to give you a practical alternative to paying for search hits (in opposition to being forced into AdWords)—based on the now-turned-off passage of keyword data on search hits from mostly Google.

That was 2006. In addition to the influx of new HitTail customers discovering it through word-of-mouth that seemed to be constant, they have a core of loyal users who love the product and go back to those days 7 years ago. So I figure that it’s time that I weigh in and talk about Google’s recent referrer data hoarding.

For 7 years prior to my move to New York, I was one of those in-house jack-of-all-trade Webmasters running the entire company’s newly developed (by the skin of my teeth) business systems—everything but Accounting and Payroll. I handled the public website, reseller channel orders, product assembly, web orders, customer service plans, and the internal website that tied it all together. I did this with an agile framework that I developed on VBScript.

From this very powerful position of running everything, I also did SEO on the side and drew a nice penny from a percentage of worldwide gross revenues. This performance-based variable income (a commission) motivated me like nothing I ever experienced before or since. These days, although I haven’t been working on that sort of deal for ages now, I never forgot lost that fierce sort of competitive SEO spirit and knowledge of how to manipulate the system to avoid paying for formal marketing products. I became one of those SEO ninjas, then took it agency-side when I moved to New York.

That’s what made me a good SEO. This “skin-in-the-game” history I think really separated me out from the consultants who did SEO for employers because they didn’t have enough going on in their own life to have to do SEO for themselves. At Scala, promoting digital signage WAS my main business. SEO was a means to an end—a necessary and recurring touch-point in the sales process. The knowledge to look-at referrer data, and technical savvy to do so in an easy and addictive way was my 15-year running secret weapon of SEO.

From it, I was one of those SEO ninjas that could talk a client through any developments or changes at Google, so long as they kept passing on that all-telling referrer data. Losing this data for any one particular site is damaging to “natural search” endeavors. But imagine companies that relied on that for special competitive advantage—perhaps aggregating referrer data across vast families of sites to yield insight—almost insider-knowledge—on the nature of the tweaks to the Google ranking algorithms.

I first started playing with referrer data and tracking gifs since 1998, realizing Urchin (later Google Analytics) was expensive, but all based on an easy-to-exploit “web bug”. Access to referrer data on a search truly did feel like a bug. I was like: “Wow, do they mean to do that?” I have wondered for all these 15 years how long the powers that be (I assumed the browser people and owners of the http protocol) would close this bug. I hadn’t imagined (at least, not that far back) that it would be Google. Google was still trying to keep you from noticing that they tracked your click-through back in those days, hiding it in a then-uncommon onclick event of the link.

It became obvious somewhere along the line that the web bug that I so dearly relied upon all these years was going away—as was my secret weapon, and I immediately started reinventing myself for what I believe to be the new age — the new SEO. I got off of Microsoft and onto Linux/Python and created 360iTiger to start playing with the API mash-ups that I saw as a large part of the future. But I was still firmly an SEO.

Then about a year ago, I felt something lose its grip—and no, it’s not my sanity from being an SEO all those years. The thing losing its grip was the mindless acceptance by the observable population that Google was always to be used as the automatic, default web search that everyone inevitably and inescapably used. A golden age of centralized authoritative search that was born with Google’s rise to dominance on the desktop was winding down. The magical moment of search… the pay-dirt moment for AdWords… was in jeopardy.

The magical moment is when you search and Google can look at your search plus profile, then instantly deliver precisely the correct advertisement for you at that moment. This is as fundamental a development in advertising as radio and TV commercials. It made everything different. It made Google a superpower. And this magical moment of inquiry is their gravy train—like over 90+ percent of their revenue stream. This is the family jewels that must be protected long enough for Google to reinvent itself—right while all us SEOs do the same. That’s fair.

How is Google losing its grip over assured default-search dominance? It’s under the strain of native mobile apps and explosive availability of data from the original and most credible sources via APIs—often making what’s gathered in Google’s increasingly antiquated-seeming web crawler method look like the trash it really is. That’s because web crawler-based relevancy systems without built-in verification and accountability are an infinitely corruptible sea and an invitation to pirates to come rob all the journeymen. And the sea was/is full of pirates robbing with impunity. Wolfram Alpha has been singing this tune for awhile, but the world is only just catching up with what the decentralization of authoritative search really might/can/and probably SHOULD mean.

Over time, users who have been made nervous and feel intimidated by all these spam-pirates on the sea of web-search (not to mention viruses, Trojan horses and identity-theft), became hungry for an authoritarian father figure in tech to keep them safe. For some time, this was Steve Jobs. He helped usher in the decentralized best-of-breed search approach to search with Siri, but still managed to not turn his back on Google’s highly demanded search brand, thus keeping it the default search in Safari. But the iTunes ecosystem, and eventually the App Store and the iPhones themselves were pitted in direct competition to the web/PC experience. Sure, iOS is closed and proprietary beautiful prisons. But when’s the last time a virus took down your iPhone?

What about the past nearly 20 years of Web infatuation? Is that all gone and for naught? Do you really like “the web” all that much more than iTunes, the App Store, and your overall iPhone experience? Could the web as we know it become a marginalized also-ran of tech in that sort of landscape? In fact, aren’t you outright delighted when some particular app comes out in exactly the area you’ve been waiting for (exercise, investing, sports, whatever)? The web is getting squeezed into a forever smaller mindshare space on mobile.

And these apps get hooked to original-source data entered, curated, or at least overseen by a “domain expert” on that topic. It’s just often better and cleaner data than the results of data culled from increasingly random-seeming web crawls. A news app for news, a product app for products, and so on. No matter your interest, there will be better native data sources for it that don’t require an old-fashioned wild-west web crawl to discover. And some talented developer will be lured into creating a kick-ass native iOS or Android app.

How will these apps and data sources actually get discovered and used? Word of mouth! Just like how Google rose to preeminence. Radically different and better things win in the end, if they can start to pick up tribe, help drive them forward, and get rapidly improved and iterated in a fashion satisfying to the tribe.

Specifically, people who know each other in some particular field-of-interest will refer others to the location of that data source (via tweeting and sharing) and the validity of the data will be asserted through Google+ and other identity verification systems, and then Likes and +1s will be used to vote up or down the data, and popularity of the data owners will be sorted out through their addition-to and membership-in Google+ circles. Google is trying to take control of the word-of-mouth process. Google Google+ Ripples to understand. There’s a lot of the future of SEO in ‘dem ‘dere ripples.

Okay, so a superimposed system of identity and tons of word-of-mouth idea-spreading vectors is how apps get popular. But does social happenstance discovery really replace search? No, but because of all the superior-sources of data than a web crawl, and because of proprietary ownership of that data by countless companies and organizations, there will be a giant custom search brokering marketplace. Your search could actually be BETTER than someone else’s because you chose to pay for some premium data sources.

These data sources will then plug it into their digital assistant—so, network communication software and task automation will be enormously important technologies. These new “digital assistant” modeled user interfaces like Siri and Google Now throw search’a future into question. And don’t rule out IBM with applications like Watson as a major player here. Half the function of tomorrow’s smart phones may be to run some hypothetical future version of Watson that can tend to your entire life—a lot like Jarvis in Tony Stark’s Iron Man suit.

A lot of what you see today is jockeying for the strongest position when AI-like digital assistants really hit. They will be the stewards of tomorrow’s search, because who’s going to bother to type a query when you can just say “look that up and advise me” to some disembodied tech slave. To whom will such digital assistants owe their core allegiance to? Not you, if you can’t truly turn the device off, that’s for sure.

The fascination with the web-facet of the Internet is gradually wearing off—to be replaced by what some are calling the “Internet of Things” replete with radically new, or even seemingly nonexistent, user interfaces where the focus goes off of dedicated generic information devices like iPhones, and instead into just any old device that has processors, network interfaces, and a vast array of tiny sensors plopped into them. A smart fork still works just like a fork. But now, it reports on your eating habits and might vibrate to tell you something.

I predict that in the not-to-distant future, you will be able to just ask of the thin air: “Where’s my keys?” and have a ghostly accurate response. It will not be expensive or difficult to achieve that. It will just be the new state of things with some phone upgrade or other. THAT’S SEARCH!

So right as Google’s grip on default search loosens, and the REAL battle over artificial intelligence-LIKE digital assistants still a bit too-off in the future to be meaningful, WHAT IS the battle? It’s the battle to create dependencies on services and platforms and ecosystems and payment systems. It’s the battle for various companies to keep strong relationships with you through all this mess, which they can then monetarily leverage in a host of different ways in the future.

It’s the battle over lifestyle and life-desision lock-ins that can only today be achieved by the likes of Apple and Google… and maybe Amazon, because it is also the battle over the infrastructure for what is the oil of the information age: data.

So, should you beware Objective-C and Java or native code written for the Dalvik virtual machine (Native iOS and Android development, respectively)? They are not standard. Or should you steer clear of generic Web HTML5/CSS3 development because it will never be as cool as a native app? It seem you can’t win. It’s a Catch-22, really. It’s one of the big ironies of mobile. There always has to be an irony: like Google and Facebook not actually being the same company. And so with becoming a tech, there’s very little clear place where you should be investing your know-how. Everything’s got a gotcha’s. You just gotta choose your gotcha’s wisely.

The state of search is one very big gotcha right now. Right as Google’s iron grip over default search wavers—as soon as you’re done in Amazon or iTunes, you still soon go back to Google. THAT’S the conundrum right now. Google is making very gutsy moves to solidify whatever industry advantage it still has today from its dominance AT LEAST as the always-second thing to search. On desktop, the reason to use Google may very well be because it’s the best search. But on mobile, the reason is brand. How long would Apple keep Google as the default web search if they didn’t have to for brand reasons? Google has the same mobile vulnerability that Facebook has, which was so dazzlingly demonstrated during its lackluster IPO.

Google, like Apple and Amazon, are trying to make itself your personal infrastructure. It did it with Google Docs for consumer-like users. Amazon did it with the cloud for business service customers. Google’s venturing into Amazon’s turf with Google App Engine. And Amazon has in the past dabbled in Google’s territory with their A9 search company (yeah, remember that?).

The problem is that nobody is really quite so expert or far-along in the other company’s core area of expertise. Apple doesn’t have datacenters or hardgoods fulfillment on the scale of Amazon, but has utter control of a platform from top-to-bottom. Amazon doesn’t have hardware or a platform like Apple, nor generic search dominance like Google—but it’s got the datacenters and hardgoods fulfillment (a critical but “dark horse” ingredient to the formula). Google has the datacenters of Amazon, but not the mastery of consumer hardware or user interfaces like Apple nor a hardgoods fulfillment infrastructure like Amazon. Here, you have Catch-22, 33 and 44.

And it is against this backdrop that Google turns dark the infinitely valuable referrer data, coming in through the “web bug” which many thought was an overly generous-with-data privacy-violating protocol mistake of the http protocol. In this action, Google turns off a firehose of the highest-value, most actionable, most free data source in marketing upon which I based the HitTail service.

At first glance, it now seems like bleak days for the SEO. I remember in my HitTail heyday watching the referrers flow in from the collective set of sites that ran the HitTail tracking. It was a very SciFi moment, at the dawn of Ajax but before websockets or any form of http server push. I still pulled it off in 2005 state-of-the-art. It was a real thrill seeing enough data pour in front of you in an almost Matrix-like flow. Sometimes I felt I could see the woman in red. And if it were a trending topic, I probably could.

Imagine the industry insights people who run networks like Quantcast, Statcounter, CrazyEgg, Compete, Optimizely and others can glean across their vast networks of tracking pixels. I know. I was a human brain looking for patterns against the giant flow of data coming in from the HitTail counter. Partially, I could tell a lot about the sites using HitTail. But more importantly, I could tell a lot about changes in the Google Algorithm. HitTail was (and really still is) uniquely adapted to alert you to changes at Google. Why?

See, HitTail only recorded the FIRST TIME any search hit ever reached your website. After that, it incremented a counter on that keyword, so you could tell total hits. Whenever a new search hit led to your website that NEVER DID BEFORE, it was a pretty big deal because it resulted in a new record being created (a new keyword being recorded). It was an incredibly efficient way to monitor for changes. Seeing ANYTHING happen suddenly on your keywords tab if you haven’t added content recently was a big signal.

This was a natural algorithm change early warning system because with any given algorithm, a site “settled down” after some time and stopped producing new keyword records. However, whenever the alg was tweaked, it stirred things up, an a whole new set of keywords became recorded for the site. So, I could sit back and watch an alg-tweak play out as all the dirt from the bottom of the lake get stirred up and muddy the waters, then watch it clear up. The clearing-up process with HitTail was watching seeing all the new keywords that COULD NOW lead to the site, inevitably lead to it (odds being what they are).

After that was done, I knew what happened with the alg. Were these new long-searches leading to the site? Were they new short-phrases? Is old content becoming active that previously was being ignored? Is previously good-performing content suddenly no longer performing well? Picture a giant river of keywords scrolling in front of me, with patterns just able to be picked out by the human mind. I was almost plugged into the changes, reverse engineering what they must have done, almost intuitively in my head.

Okay, now if I was doing this just to keep myself sharp in my industry, imagine how many other people and companies were using this data in more direct ways. You could (and I have) build automatic content-creation and aggregation systems that zero in on the best, most underutilized but promising search terms. You can also key off of long tail search terms that are suddenly on the rise, but yet-undominated by a serious competitor. Usually these spam cannons are justified as topic page generators. Smart ones could also retire pages no longer needed or beneficial.

These sorts of auto-content-generators that relied so much on the keyword referrer had just been struck a major blow. It was some of the few automate-able endeavors in marketing that didn’t require some sort of major crowd-sourcing (ala Pinterest) or much human oversight or energy (ala editorial staffs) to do well in natural search. When cleverly designed, such page generators could automatically respond to algorithm changes, and still be effective, because there’s something that’s always almost nearly working for you, and all you need to know is how to optimize one or two factors that still really matter (like title tag and headline-like element).

And so, here we are cut off from granular keyword data on our own systems. The data still resides (sometimes, when the stars are aligned just-so) on Webmaster Tools and Google Adwords for review and download. But it’s not in the same granular form as in your log files, or that could be culled from a tracking pixel. Causality is lost. The anecdotal story of each visitor that could be gleaned by looking at their initial search hit then journey through the site, and eventual return visits, has been lost. Original attribution to natural search is no more. We must go with our hat in our hands to download an inferior distilled version of this data from Google services.

Blah, forget it. SEO was interesting to me because it was science. Observation and experimentation could let you rapidly zero in on what content worked best, then help you do more of it. Now, sites can’t nearly be optimized so easily on the long-tail keyword front. You can still do stuff, but you have to bring the much less precise and much greater guesswork of statistical analysis to bear the problem. Direct cause-and-effect evidence left in your own log file data is lost in this new deal. It is now a problem of massive correlation and guesswork. You might as well put that energy elsewhere.

I used to get a sort of thrill out of brilliant and unexpected site-shaping chisel strikes that you could do based on this referrer data. It was low-hanging fruit. It was immediately actionable to-do items. It was striking-distance endeavors. Now, it’s people just popping into existence inside your home. What inquiry brought them here? I don’t know! Just that they came from Google.

Although I have positioned myself out of the field of SEO to some degree over the past year, SEO isn’t dead yet, and my mind keeps getting drawn to the problem for which SEO was once the slam-dunk solution. What are the most intelligent and efficient marketing plays today?