Robots.txt Overview, Generator, & Analyzer

Jul 21st

We recently compiled a 6 page robots.txt primer and added a robots.txt generator and robots.txt analyzer to our suite of free SEO tools.

Published: July 21, 2008

New to the site? Join for Free and get over $300 of free SEO software.

Once you set up your free account you can comment on our blog, and you are eligible to receive our search engine success SEO newsletter.

Already have an account? Login to share your opinions.

Comments

July 21, 2008 - 3:51pm

Nice work Aaron, this will be a great help to people not usually used to using robots.txt.
The generator in particular is user friendly and helpful.

Thanks.

July 21, 2008 - 4:09pm

thanks, aaron. i didn't know that spaces in user-agent strings are a bad idea.

July 23, 2008 - 3:26pm

This post and the http://tools.seobook.com/robots-txt/ post are excellent - succinct and cover some very helpful elements.

Thanks Aaron.

Ben

July 24, 2008 - 1:21am

Glad you found them helpful Ben. :)

July 24, 2008 - 1:48pm

great tool.It really helps. I tried it myself.Also went to see the robot.txt tutorial. There are lots of goodies here.Thanks a lot. It really helps especially newbies like me.

August 19, 2008 - 10:44pm

Nice tool Aaron.

I thought our robots would pass your analyser and I was embarrased to find some faults! So thanks!

Mike

New to the site? Join for Free and get over $300 of free SEO software.

Once you set up your free account you can comment on our blog, and you are eligible to receive our search engine success SEO newsletter.

Already have an account? Login to share your opinions.

  • Over 100 training modules, covering topics like: keyword research, link building, site architecture, website monetization, pay per click ads, tracking results, and more.
  • An exclusive interactive community forum
  • Members only videos and tools
  • Additional bonuses - like data spreadsheets, and money saving tips
We love our customers, but more importantly

Our customers love us!






    Email Address
    Pick a Username
    Yes, please send me "7 Days to SEO Success" mini-course (a $57 value) for free.

    Learn More

    We value your privacy. We will not rent or sell your email address.