« Unintended Consequences
Poor .info Domain Name Strategy, Afilias »

Robots.txt Overview, Generator, & Analyzer

Jul
21

We recently compiled a 6 page robots.txt primer and added a robots.txt generator and robots.txt analyzer to our suite of free SEO tools.

Subscribe to our blog via email or RSS to get more great posts like this one.

Posted in:

by Aaron Wall

Your Thoughts? [6]

July 21, 2008

comments

New to the site? Join for Free and get over $300 of free SEO software.

Once you set up your free account you can comment on our blog, and you are eligible to receive our search engine success SEO newsletter.

 

Already have an account? Login to share your opinions.

Nice work Aaron, this will be a great help to people not usually used to using robots.txt.
The generator in particular is user friendly and helpful.

Thanks.

Diablos on July 21, 2008 03:51 PM

thanks, aaron. i didn't know that spaces in user-agent strings are a bad idea.

corey on July 21, 2008 04:09 PM

This post and the http://tools.seobook.com/robots-txt/ post are excellent - succinct and cover some very helpful elements.

Thanks Aaron.

Ben

yet another ben on July 23, 2008 03:26 PM

Glad you found them helpful Ben. :)

Aaron Wall on July 24, 2008 01:21 AM

great tool.It really helps. I tried it myself.Also went to see the robot.txt tutorial. There are lots of goodies here.Thanks a lot. It really helps especially newbies like me.

axedbydax on July 24, 2008 01:48 PM

Nice tool Aaron.

I thought our robots would pass your analyser and I was embarrased to find some faults! So thanks!

Mike

mikesed on August 19, 2008 10:44 PM

New to the site? Join for Free and get over $300 of free SEO software.

Once you set up your free account you can comment on our blog, and you are eligible to receive our search engine success SEO newsletter.

 

Already have an account? Login to share your opinions.

Professional SEO Training

Improve your rankings, traffic, and profits today. The SEO Book training program offers you:

  • Over 100 training modules, covering topics like: keyword research, link building, site architecture, website monetization, pay per click ads, tracking results, and more.
  • An exclusive interactive community forum
  • Members only videos and tools
  • Additional bonuses - like data spreadsheets, and money saving tips
  • Every order comes risk free, and with the best selling SEO Book as a free bonus

Watch this video to learn more.

Join The #1 Online SEO Community

Hear what our members say about the #1 SEO Community. Here are some of our recent thread topics:

Improve your rankings today!