Robots.txt Overview, Generator, & Analyzer

We recently compiled a 6 page robots.txt primer and added a robots.txt generator and robots.txt analyzer to our suite of free SEO tools.

Published: July 21, 2008 by Aaron Wall in seo tools


July 21, 2008 - 3:51pm

Nice work Aaron, this will be a great help to people not usually used to using robots.txt.
The generator in particular is user friendly and helpful.


July 21, 2008 - 4:09pm

thanks, aaron. i didn't know that spaces in user-agent strings are a bad idea.

yet another ben
July 23, 2008 - 3:26pm

This post and the post are excellent - succinct and cover some very helpful elements.

Thanks Aaron.


July 24, 2008 - 1:21am

Glad you found them helpful Ben. :)

July 24, 2008 - 1:48pm

great tool.It really helps. I tried it myself.Also went to see the robot.txt tutorial. There are lots of goodies here.Thanks a lot. It really helps especially newbies like me.

August 19, 2008 - 10:44pm

Nice tool Aaron.

I thought our robots would pass your analyser and I was embarrased to find some faults! So thanks!


Add new comment

(If you're a human, don't change the following field)
Your first name.
(If you're a human, don't change the following field)
Your first name.
(If you're a human, don't change the following field)
Your first name.

Gain a Competitive Advantage Today

Your top competitors have been investing into their marketing strategy for years.

Now you can know exactly where they rank, pick off their best keywords, and track new opportunities as they emerge.

Explore the ranking profile of your competitors in Google and Bing today using SEMrush.

Enter a competing URL below to quickly gain access to their organic & paid search performance history - for free.

See where they rank & beat them!

  • Comprehensive competitive data: research performance across organic search, AdWords, Bing ads, video, display ads, and more.
  • Compare Across Channels: use someone's AdWords strategy to drive your SEO growth, or use their SEO strategy to invest in paid search.
  • Global footprint: Tracks Google results for 120+ million keywords in many languages across 28 markets
  • Historical data: since 2009, before Panda and Penguin existed, so you can look for historical penalties and other potential ranking issues.
  • Risk-free: Free trial & low price.
Your competitors, are researching your site

Find New Opportunities Today