Robots.txt Overview, Generator, & Analyzer

We recently compiled a 6 page robots.txt primer and added a robots.txt generator and robots.txt analyzer to our suite of free SEO tools.

Published: July 21, 2008 by Aaron Wall in seo tools

New to the site? Join for Free and get over $300 of free SEO software.

Once you set up your free account you can comment on our blog, and you are eligible to receive our search engine success SEO newsletter.

Already have an account? Login to share your opinions.

Comments

July 21, 2008 - 3:51pm

Nice work Aaron, this will be a great help to people not usually used to using robots.txt.
The generator in particular is user friendly and helpful.

Thanks.

July 21, 2008 - 4:09pm

thanks, aaron. i didn't know that spaces in user-agent strings are a bad idea.

July 23, 2008 - 3:26pm

This post and the http://tools.seobook.com/robots-txt/ post are excellent - succinct and cover some very helpful elements.

Thanks Aaron.

Ben

July 24, 2008 - 1:21am

Glad you found them helpful Ben. :)

July 24, 2008 - 1:48pm

great tool.It really helps. I tried it myself.Also went to see the robot.txt tutorial. There are lots of goodies here.Thanks a lot. It really helps especially newbies like me.

August 19, 2008 - 10:44pm

Nice tool Aaron.

I thought our robots would pass your analyser and I was embarrased to find some faults! So thanks!

Mike

New to the site? Join for Free and get over $300 of free SEO software.

Once you set up your free account you can comment on our blog, and you are eligible to receive our search engine success SEO newsletter.

Already have an account? Login to share your opinions.

Gain a Competitive Advantage Today

Your top competitors have been investing into their marketing strategy for years.

Now you can know exactly where they rank, pick off their best keywords, and track new opportunities as they emerge.

Explore the ranking profile of your competitors in Google and Bing today using SEMrush.

Enter a competing URL below to quickly gain access to their organic & paid search performance history - for free.

See where they rank & beat them!

  • Comprehensive competitive data: research performance across organic search, AdWords, Bing ads, video, display ads, and more.
  • Compare Across Channels: use someone's AdWords strategy to drive your SEO growth, or use their SEO strategy to invest in paid search.
  • Global footprint: Tracks Google results for 120+ million keywords in many languages across 28 markets
  • Historical data: since 2009, before Panda and Penguin existed, so you can look for historical penalties and other potential ranking issues.
  • Risk-free: Free trial & low price.
Your competitors, are researching your site

Find New Opportunities Today


    Email Address
    Pick a Username
    Yes, please send me "7 Days to SEO Success" mini-course (a $57 value) for free.

    Learn More

    We value your privacy. We will not rent or sell your email address.