So the title is a little bit incendiary in my opinion mainly because of my attitude toward Google lately. When Google first appeared on the WWW stage it was a breath of fresh air. Finally there was a site that seemed to want to provide a quality service that didn’t just care about money unlike many of the search engines of the time. My pre-google favorite was Alta Vista and without any hesitation I dumped AV and ran to Google. Besides having a clean interface, accurate results, and of course who could forget the, “I feel lucky” button made me want Google to succeed. This was all back in about 1999, if my memory serves me correctly. In a very short period of time Google was the most popular search engine and in my opinion the quality of their search warranted it.
That seems so long ago now. The once playful attitude and quality search has, in my opinion been replaced with a huge corporation that feels that it has the duty to police the web. Where they fit into the internet landscape of what I remember to be the Wild Wild West of web sites, they won the battle. They saw the deficiencies and worked within them to become number one. Good on you, Google.
This post is based on an article that I read at The Next Web entitled, “Google wants to make the 25-year-old robots.txt protocol an internet standard.” For anyone not sure what robots.txt is, it’s basically a way for a web designer to instruct bots how to behave when they come to the site. It can deny or allow bots from indexing the site or can give instructions about whether to include or exclude separate files or folders. It is a very weak method of a code of behavior for the bot. I say it’s weak because there isn’t any way that you can control the bot, the robots.txt has no teeth and can’t stop a bad bot from doing what it wants. Now Google has decided to make standards for the robots.txt. My very first thought is, what? My second was, “who the hell is Google to set standards?” I get it, their business is mostly, if not solely on the web and they are a huge corporation, but that doesn’t give them the right to make rules and then possibly enforce them through denying someone’s site from being listed. In fact all I see is a huge corporation trying to make their job easier.
I want to be clear, I get it. Google has a right as a private company to do what it wants. Some, or even all of their rules make sense at some level and could make the WWW a safer place, maybe. What I dislike is that I trusted them so long ago and the result was that I helped create a monster. A corporate giant with the desire to control something that they have no right to control. I want a little chaos on the web, I want the freedom to do what I want, and mostly to not have anyone tell me that my site is bad because Google doesn’t list it. It’s not Google’s job to judge, they should index the web and display the results. Period.
Afte
r I have said my piece about Google, I started to envision people sticking up for them. “Come on, it’s a good thing. Google is just pushing for what good designers are already doing.” In theory I agree, this is nothing in the big picture. A robots.txt file is nothing but a simple text file that wouldn’t take 2 minutes to complete. The core of my displeasure over what Google is doing is I can see a time when we lose the freedom to surf the net. That sites like google will create a tiered system of web sites that supports large corporate sites and omits or neglects to serve smaller sites. I will take it a step further too, there may be in the very near future legislation that controls who can add a site, or ban a site if it doesn’t fit their standards (whoever “their” is). What if Google decides that it should ban a candidates site in the upcoming election? That, I hope we can agree on, would be a very bad thing. How far are we from this happening now?