Large websites tend to have many useless pages associated with them. They may be caused by any of the following

• poorly structured or poorly formatted user generated content
• content duplication due to content management issues
• canonical related issues
• dangling nodes which act as PageRank sinks
• navigational pages which are heavily duplicated and soak up link authority and do not provide a clean site structure

I recently have had a couple SEOs show me various navigational techniques which made thousands of thousands of somewhat similar mid level navigational pages.

Some pages make sense to be indexed and provide a great user experience if searchers land on them. Others provide a poor user experience.

Search engines do not like indexing search results from other engines, so if your navigational scheme has an element which acts similar to an internal search engine you probably do not want all those search pages getting indexed if they are heavily duplicates of one another.

I was talking to Stuntdubl the other day, and he stated one of the main things he likes to look at to get a general indication of the health of a site is to look at the ratio of quality pages indexed to total pages indexed from your site.

If lots of your indexed pages are heavily duplicated and/or of low value that may cause search engines to crawl or index your site less deeply and not index all your individual product level pages.

A few people have created free cool web based tools which allow you to search through the 20 million keywords AOL recently shared with the marketing community. http://www.aolsearchdatabase.com/ - allows you to sort data by:

• User ID
• Keywords
• Date of search
• URL

http://www.askthebrain.com/aol/

• allows you to sort data by TLD
• allows you to sort data by keywords leading to a specific domain
• by default the tool also displays the top few thousand URLs, the number of referrals to a URL, shows the top keywords leading to a URL, shows the keyword diversity ratio

http://www.dontdelete.com/ - search by keyword, keyword stem, or part of a keyword to find related keywords. For example, if you searched for dati it would return all keywords that had dating in them.

Anyone think I should add a link to any of these tools on my keyword research tool?

So I went to a party last night, and a friend of mine was not on the list, but got there before me and said he was my guest. Fair enough, and good for him. Usually being on the list is a good deal, but with internet marketing using the default list is a bad deal. Let's say there is a place that does cheap article submissions that has 50 places they submit articles to. Even if they did a kick ass job of it, no way would I want to submit my site to all 50 places if that list of submission places was the default one used by the industry.

I might consider submitting to the top 10 or 20 article sites, but the deeper you go the more abstract and lower quality of a link you are getting, and the more likelihood there is that those sites would probably be in a bad web neighborhood (ie: have low quality inbound AND low quality outbound links). In most fields only the top few sites are typically going to have many high quality links.

Also, if there is a potential gem that is not on the list (just for a rough example, say Buzzle.com as a decent article submission site) then maybe getting a few links from those types of sites that are not on default lists are worth more than just pushing to do what is easy to replicate and widely done by those using a default list.

That is part of the reason I do not keep my directory list up to date so much anymore. IMHO, many of the search engines have evolved far beyond that. Sure a Yahoo! Directory link is great, a Business.com listing could be money well spent, and maybe there are a half dozen or so other general directories that are going to be longterm useful and valuable to be listed in. But if you submit to 400 directories from a commonly used (and abused) industry list do not be surprised if your site does not rank as well as you would like.

As long as your mindset is stuck on a list, if you keep going deeper you may be digging a deeper hole. The more your marketing strategies, data sources, link sources, niche market selection ideas, keyword ideas, and content creation ideas are hidden from the common lists that everyone use the easier it will be to quickly create value and extract profit from your work.

If you go to an affiliate convention and see a merchant advertising all their offers to top affiliates that is a sign that their market already is or may soon be saturated beyond profitability.

I have even heard some people who sell AdSense keywords lists scrub the words with the highest easily accessible profit potential from their keyword lists.

And, this really is what makes writing a book about SEO and trying to keep it up to date somewhat hard. People want easy, simple, and straightforward lists that guide them toward success, but the more your marketing strategies, data sources, link sources, niche market selection ideas, keyword ideas, and content creation ideas are hidden from the common lists that everyone use the easier it will be to quickly create value and extract profit from your work.

The easiest way to get quality citations is through social relationships and unique original marketing ideas, but you really can't put those on a list unless people can infer how to extrapolate how an idea relates to their core strengths, potential customers, and/or marketplace.

AOL proved stupid by potentially violating the privacy of 650,000 searchers and the 20 million searches they did over a 3 month period earlier this year. While seasonal and limited to AOL data, this ties data together between searcher and search patterns as well as searches and relevant sites.

Imagine combining Alexa, Hitwise, and Wordtracker, and making it all free :)

The original data source has been pulled, but mirrors are up.

I frequently get asked to look at a page to see if I think it is perfectly optimized. But I rarely think you can tell if a page is perfectly optimized just by looking at one page. Most of the optimized pages I am asked to look at have no clear goal at hand. Is the page meant to be link bait? Am I supposed to buy from the page? How is the page integrated into your site? How do people find this page? What sort of brand equity have you built up?

No matter what you are doing you are always Electioneering. Does your site sell ad space? Are you trying to manipulate public opinion? In many ways just having open access to people who are willing to answer your questions is highly valuable even if that is the only aim of a page.

The key to doing well is to gain enough authority and mindshare to be in a self reinforcing market position or to have enough momentum to be able to jump from field to field. Examples:

• What is Home Depot? A retailer? Or a large home improvement ad network? They are willing to test adding ads to their site, which may allow them to leverage their brand for high margin revenue streams and discover new products and new markets while competing retail only stores are forced to close due to shrinking margins (I just tried shopping at the local Lowes but they closed down).

• The Wall Street Journal is going to sell front page ads.
• Some newspaper websites include syndicated links from blogs. Recently some newspaper publishers have even decided to add links to competing content near their stories.
• Ken McCarthy has been an internet marketer longer than I have known what the internet is about. He recently asked on his blog what Google did with their recent AdWords update. He must have got fifty to a hundred responses.

The issue with just looking at "is this page optimized" is that all markets are heavily manipulated and search frequently changes. If you just maximize one portion of your optimization process without considering the social aspects of the web, short and long term goals, or how your site fits into the web as a whole then when one of the market makers changes the rules you are relegated to leaving whiny comments about how that market maker is evil.

One of the reasons I never published SEO Book in print format is that my self image is quite low, and I never think what I do is good enough (I realize I should not have wrote that). But as time has passed I came to appreciate that the whole concept of optimization really is about predicting market changes and getting the most of the current market as you can. Any sort of optimization has associated opportunity costs and is only effective for a certain window of time. What was perfect optimization a year or two ago is now largely inefficient and ineffective as some of the market makers have closed some of their algorithmic holes.

Long term optimization would be "create high quality content that users like that would make a search engine inadequate if it was not ranked." But when you are starting from nothing, sometimes it helps to take advantage of a few market inefficiencies to help build the exposure necessary to build a self reinforcing position which may allow you to jump from field to field.

And the web is such a social medium that it requires understanding people far more than algorithms. At least for most businesses. And human nature changes much slower than the web and search algorithms do. But you can't understand how well a person speaks to / with their audience just by looking at one web page.

I might have a kick ass copywriter edit my sales letter to improve its conversion rate. I am not afraid of paying him to write it, but my biggest fear with optimizing it is that if I push conversion too much, then what will that do to the consistency of my voice across my site?