MovableType Blogs, TypePad Hosting, Comment Redirects & Google Ignoring Robots.txt Files

Jun 18th
posted in

What is up with Google indexing all the TypePad comment redirects. Clearly the robots.txt file says Google should not be indexing those.

Is ignoring the robots.txt file an accident, or a normal feature at Google?

I have a rather small blog, with about 1,000 posts on it. Google is showing 5,000 pages from my site in it's index. Some of my normal pages are already not being cached because Google is indexing my site less aggressivley due to seeing no unique content on the pages where THEY IGNORE THE ROBOTS.TXT PROTOCOL. Pretty evil shit, Google.

Now I need to figure out how to do some search engine friendly cloaking or somehow issue Googlebot 403 errors when it tries to spider those URLs. Way to suck Googlebot.

Perhaps this issue would have been noticed and addressed by a MovableType employee if they didn't have blind trust in search engines and think all SEOs are scum.

Many TypePad hosted sites & MovableType sites are being screwed / partially indexed due to this problem. MovableType owes it to their paid customers to ensure problems like these are not happening.

Published: June 18, 2005

New to the site? Join for Free and get over $300 of free SEO software.

Once you set up your free account you can comment on our blog, and you are eligible to receive our search engine success SEO newsletter.

Already have an account? Login to share your opinions.

Comments

June 18, 2005 - 8:44am

It would seem you're not alone Aaron - I sent an email to Google about the problem and suddenly they all disappeared from the index before I received a response from them citing that they were unable to replicate the problem. I knew I should have grabbed a couple screenshots.

kp
June 18, 2005 - 7:07pm

They do index the URL, not the content, that's how Google index always works.

June 18, 2005 - 7:56pm

>They do index the URL, not the content

where is the value in that? especially when it leads to partial indexing of pages with content on them?

June 18, 2005 - 9:49pm

| They do index the URL, not the content

That wasn't the case for me, they all out indexed my images and offered them up in search results...

and I'd raise the same question as Aaron, what's the point?

New to the site? Join for Free and get over $300 of free SEO software.

Once you set up your free account you can comment on our blog, and you are eligible to receive our search engine success SEO newsletter.

Already have an account? Login to share your opinions.

  • Over 100 training modules, covering topics like: keyword research, link building, site architecture, website monetization, pay per click ads, tracking results, and more.
  • An exclusive interactive community forum
  • Members only videos and tools
  • Additional bonuses - like data spreadsheets, and money saving tips
We love our customers, but more importantly

Our customers love us!






    Email Address
    Pick a Username
    Yes, please send me "7 Days to SEO Success" mini-course (a $57 value) for free.

    Learn More

    We value your privacy. We will not rent or sell your email address.