Sometimes it's Hard to Tell, But What Is Clear is That Google Costs Small Biz $Millions
So your website's being dropped by Google. Perhaps you had thousands of pages indexed, and you can see the number plummeting...or perhaps you're already totally out of the index: search for site:yourdomain.com and Google responds with "Your search - site:yourdomain.com - did not match any documents."
The first reaction for most site owners is to assume that they have been banned by Google, that they've done something wrong, something "blackhat," that Google has discovered their nefarious deeds and punished them by dropping them from the index.
But hold on a moment...that's not always true. It's not always clear whether Google has "banned" a site, or whether something else is going on, and it's close to impossible to get Google to let you know. What is clear, though, is that it's costly for the business community, hurting thousands of innocent business owners, to the tune of tens of millions of dollars each year.
The Nightmare Scenario - Your Site Drops
Let me give you one little example. A one-time consulting client called me in a panic a few months ago. I had set them on a path, and things had been going well; more and more pages were getting indexed by Google, and more and more pages were turning up high in the search results...until one day, things started to drop. Google appeared to be dropping the company's pages out of its index, and the few pages that remained weren't doing well in the results.
"Don't worry," I told them, "Yhings will recover. This is all part of the natural Google cycle. Things go up and things go down." But as is common in these situations, the company was convinced it must have done something "wrong." Some change to the site must have displeased the Google Gods, and now it was being punished. "No," I said, "there's nothing you've shown me that should get you in trouble with Google. I think it will come back. Give it a couple of weeks."
This company spent literally thousands of dollars on the problem: executives wasting valuable time in meetings, discussing what had to be done; Web designers being paid by the hour to suggest to the client (incorrect) solutions and then make (irrelevant) changes to the site; money paid to me to sift through dozens of emails between the client and the Web designers, trying to figure out what had gone wrong and what, if anything, should be done about it. And what happened in the end? After running around in circles for a few weeks, the company eventually saw the site suddenly pop back into the Google index, and the traffic started flowing again.
Google's Response - Too Little, Too Late
Now, I understand Google can't guarantee anyone a particular position in the search results. But when this sort of thing happens, it's very expensive -- and disturbing -- for the businesses involved. We're talking tens of millions of dollars of expenses incurred to deal with Google fluctuations like this. I also understand that there are people within Google who know of this problem, and are concerned by it, which is good. What isn't so good is that Google's response to the problem has been inadequate and slow; too little, too late, one might say.
Let's look at a scenario. Imagine for a moment that Joe the Website Owner has seen his site suddenly drop out of the Google index. He must have done something bad, right? But no, not necessarily. As we've seen, sites drop out and come back, with no significant changes, and Google itself has publicly stated that this can happen now and then.
So, what to do? Well, perhaps someone tells Joe about the Google Site Status Tool which, according to Google, "lets you check the index status of your site, and also tells you when your home page was last accessed by Google." Perfect, Joe thinks, exactly what he needs! So he opens this tool, enters his site's URL, clicks the Next button, and Google responds with this:
Googlebot has successfully accessed your home page.
Potential indexing problems:
We do not know about all the pages of your site. You can submit a Sitemap to tell us more about your site.
Well, that's not so helpful; in fact, it's utterly worthless. Joe knows Google has successfully accessed his home page: it used to be indexed, after all. He's also well aware that there are indexing problems. (Not just potential problems!) And Joe has already submitted a sitemap. So this tool is, in effect, a complete waste of electrons.
So what next? Joe could try logging into his Webmaster Tools account, where he submitted his sitemap, and see if he can find any clues. What does he find? Well, not much. He doesn't see any errors, and although Google sometimes sends messages to site owners through the Message Center, warning them of infractions, in Joe's case there's nothing. He does notice one thing, though: Google is continuing to index the site and to download his site's sitemaps. Joe's spirits are buoyed. Perhaps this is just a temporary glitch. Maybe the site will be back soon! In fact, Google might continue downloading sitemaps and crawling the site every day, even if the site remains out of the index for months.
Google's Webmaster Guidelines - Part of the Problem
So what is Joe supposed to do? Well, these days Google's preferred process is to have you read their Webmaster Guidelines, find out what you are doing wrong, correct the problem, then contact them using the Request Reinclusion form in your Webmaster Tools account. But there are plenty of problems with this process.
First, it's not always easy for a non-expert, such as a small-business owner like Joe--in this case a plumber--to figure out what the guidelines actually mean. In fact, the guidelines are ambiguous enough for different people even within the SEO business to hold different interpretations. For example, many people believe that "cloaking" -- sending one page to Google's crawlers, and a different page to browsers -- is an absolute no-no that will get your site banned. This belief is based on Google Webmaster Guideline statements such as, "Don't deceive your users or present different content to search engines than you display to users, which is commonly referred to as 'cloaking.'" However, there are often good reasons for cloaking, and upon a deeper read of Google's documents you discover that Google doesn't object to cloaking in many circumstances. They just object if the intention is to mislead them.
Or how about hidden links? Google doesn't like things that are designed purely for the search engines and not users, and thus is suspicious of links that it can find but that users cannot. But there are often good reasons to have "hidden" links, such as in-navigation tools that only appear upon request -- a pop-up sitemap helping users find their way through the site, for instance. Such a device would definitely not be against the spirit of the Google guidelines. But could Google find it and interpret it as a trick?
Thus the Google Webmaster Guideline's ambiguities raise two questions:
1. Has Joe done something in his optimization that is borderline trickery and that Google objects to?
2. Has Joe done something totally innocent that Google has misconstrued as being devious?
Now, put yourself in the Joe's shoes, in the shoes of this non-expert business owner. (Perhaps you're already in those shoes.) Either Joe has to decipher the guidelines himself -- and quite frankly, this is not possible for the average business owner -- or he has to hire someone to decipher them. But who can Joe hire? The average Web designer or Web-design firm knows next-to-nothing about SEO (even though they claim otherwise), and as for SEO firms themselves. Well, too many SEO firms are either outright cons or perhaps trying their best but only knowing enough about SEO to be dangerous. So, if our non-expert business owner asks five "experts" to review the site and come up with suggestions for what's wrong, he's likely to get five different answers, most of which -- and perhaps all of which -- will be wrong. And then, to cap it all, a friend tells Joe, "Yep, this happened to me too. My site dropped out of the index for two months last year, and then it came back. Never did figure out why."
The Human Factor
So has Joe done something wrong? Is this all a Google Glitch? Or perhaps something has been misinterpreted by a Google employee? Google likes to have its algorithm do most of the work, automatically ranking sites in a purely objective manner. (It's objective once the algorithm is released, of course. At some point subjectivity is employed to write the code.) It also employs people to check some sites manually, to subjectively rate them. Thus your site may have been killed by an employee who didn't fairly apply the rules, or who is inexperienced and doesn't fully understand what he's doing.
If you use Google Adwords, you may have seen this in action with another Google review process -- the Adwords editorial review -- in which Google employees may too often misinterpret the rules. I recall one case in which a PPC ad that mentioned the town of Superior, Colo. was blocked by an editor because the word "superior" was a superlative. And in another case the publisher of charitable-donation software had an ad blocked because he wasn't displaying his charitable tax-exempt status on his website, which of course he neither had nor needed, because his company wasn't a charity.
Everyone who's done more than a few Adwords campaigns seems to have a "crazy editor" story, so if Google's process for manually checking websites for spam has the same level of accuracy as its process for checking ads for compliance with editorial standards, then too many sites are being unfairly blocked.
Anyway, let's imagine Joe has picked a couple of things that his advisors think might have caused problem. He's fixed them -- or paid someone to fix them more likely. Then Joe used the Request Reinclusion form. Then what? Probably nothing. Joe's chance of getting a response from Google is pretty low. Sure, he got a boilerplate response:
We'll review the site. If we find that it's no longer in violation of our Webmaster Guidelines, we'll reconsider our indexing of the site. Please allow several weeks for the reconsideration request. We do review all requests, but unfortunately we can't reply individually to each request.
But there's a good chance that's the last Joe will ever hear from Google on the subject. Of course, Joe will probably try posting a message in the many SEO forums, including Google's own Webmaster forums, something Google actually advises. But so much of the advice is conjecture, conflicting, or just plain confusing (and what Joe doesn't realize, often just plain wrong), that Joe leaves in disgust.
Joe's Next Step
So now what is Joe going to do? Well, eventually he gives up. He can't find anyone at Google to talk to, he's changed a few things on his site, and still he's out of the index. He could start over with a brand new website, of course, but in this case Joe does something a little different. Yes, he moves his site, and in fact, he reverts to his original content -- under the theory that changing his content hadn't fixed the problem anyway- - to a new domain name. But he also does a 301 redirect from the old site to the new one.
"No!, Joe," I hear some of you shouting, "say it ain't so! That old domain name has been banned; a 301 redirect will transfer the ban from the old site to the new, and poor ole Joe's just going to kill the new site, I betcha!" I can also hear some of you saying, "What does Joe, a plumber, know about 301 redirects?" Not much, I'll admit, but for the sake of illustration I'm assuming one of the Web designers implemented the redirect for him.
But that's not what happens to Joe's site. In this case (and many similar cases, in fact), Google begins crawling the new site and adds it to the index. That's right, sometimes the "banned" domain doesn't transfer its curse to the new domain.
So where does all this leave Joe? First, hundreds, if not thousands, of dollars in the hole. He's paid Web designers to misinform him, and he's wasted time on this whole process -- time he could have spent fixing people's bathrooms. He's paid a designer to change a few things on his site, he's lost business for a couple of months, and he's paid to move the site.
But in addition, it leaves him confused. Was his original site even "banned"? Well, it wasn't in the index then, but it is now, and questions are still hanging.
Google's Responsibilities
Google, through a combination of great ideas, outstanding innovation, massive infusions of money, and being at the right place at the right time, finds itself in an enviable position; it's a virtual monopoly. If you're in business online, Google has to be part of your marketing strategy. The great majority of Web searchers query a database managed by Google, either on one of Google's various national websites, or on one of Google's partner sites. Search at AOL or Earthlink, for instance, and you're searching Google. Around 75% of all search results come from this company.
Monopolies and near-monopolies have special responsibilities. All businesses have responsibilities, of course, but when you're the only game in town, it's even more important that you act responsibly, because people can't just go elsewhere. Right now, Google is hurting businesses. Yes, yes, I understand that Google has to protect the integrity of its index, and there are plenty of people playing games to trick Google. But that's generally not who's getting hurt, anyway. The true "black hatters" expect to have sites banned. It's just part of the cost of doing business. It's often businesses who don't fully understand SEO that are getting hurt, or businesses tripping over Google's ambiguous guidelines, or businesses using techniques that Google found acceptable yesterday, but not today.
What, then, could Google do to make life easier for these people? Well, the first thing Google could do is create a genuine Site Status Tool, one that provides useful information. A site owner should be able to use a tool, behind the password-protected wall of the Webmaster Tools, in which he can ask Google about his site's status. Better still, this information should be a primary component of the Webmaster Tools, showing you site status when you log in.
The status should include information such as whether your site is in the index or not. If so, how many pages are indexed. If not, it should give some indication of why, along with detailed explanations and suggestions, such as the following:
Again, telling someone about a problem will rarely do much harm. For example, telling someone that they shouldn't be using hidden text on a page hardly tips off the person using black-hat techniques. (These guys are way beyond that!) And reportedly Google does sometimes inform people, through the Webmaster Tools Message Center of these types of problems. And Google already tells people things that they shouldn't be doing in the guidelines, so what's wrong with telling Joe which of the (sometimes ambiguous) guidelines is the problem?
For instance, Google says that cloaking pages is sometimes okay and sometimes not. Perhaps I think my use is okay, but Google thinks it's crossed the line. Is it so unreasonable, when the issue is a fuzzy rule, to expect some feedback? Or perhaps Google has found some hidden links in my site, links that are clearly not intended to mislead the search engine but that have been interpreted as doing so. Is it so unreasonable to expect a tip off that this is the problem?
Innocent Before Proven Guilty
Imagine for a moment that the firm in the first example I gave in this article had gone into their Webmaster Tools account and seen the message "Googlebot has crawled x pages, but due to normal fluctuations these pages are not currently displayed." That one little piece of information would have saved the firm thousands of dollars.
Here's another thing Google could do: at least tell people when their Request for Reinclusion has been seen, so you don't just leave them hanging. The current process of expecting people to drop requests into a black hole reflects badly on Google, leading people to believe that Google probably doesn't read them, that Google doesn't care, or even that Google intentionally blocks some sites in order to boost its AdWords business. (That maybe be a flakey conspiracy theory, one to which I do not subscribe, but it's a common claim.)
Furthermore, if a site has been blocked manually, there should be some kind of review process, the SEO version of habeas corpus. It shouldn't be possible for a low-level Google employee to manually "lock up" someone's site without some kind of recourse, some kind of judicial review that will free the innocent.
Now, one could argue that all this doesn't really matter, that if you're not playing tricks you won't get banned, that only the bad guys get hurt. But that's demonstrably false. Clearly many innocent sites are dropped from the index, sometimes temporarily, sometimes for extended periods. It's similar to the argument made by Ed Meese, Ronald Reagan's attorney general: "If a person is innocent of a crime, then he is not a suspect." In fact Google's own Quality Rater Guidelines contains the phrase, "We prefer that a 'guilty' page remain unlabeled [as spam] than an 'innocent' page be labeled." Well, come on, then, Google, step up. You're hurting people, when you have it in your power to help them.
About the Author: Peter Kent is the author of Search Engine Optimization for Dummies, The Complete Idiot's Guide to the Internet, and several dozen more books about the Internet and online business. He consults with businesses large and small in the areas of SEO and e-commerce. You can find more information at PeterKentConsulting.com.