(GEEK STUFF) One of the largest problems many search engines run into is that after they get to a few hundred million documents their algorithms and hardware hit a wall.

For those companies that can afford the investment to get past this point they still run into the problem that each additional resource makes their job a bit harder.

One of the major ways around this problem is to take advantage of the natural patterns in human language. Using Latent Semantic Indexing allows indexing search results based on the pairing of like words within documents.

Many complex searches may lack exact matches in the results as well. Being able to find near matches will allow search engines to provide more comprehensive results.

Its hard to get computers to understand anything human, but the process of latent semantic indexing delivers conceptual results while being entirely mathematically driven.

There are two main ways to do this, single variable decomposition and multi dimentional scaling.

Some of the steps of the single variable decomposition process are to:

• create a database of all words in relevant documents
• remove common stop words
• stemming
• remove words appearing in all results
• remove words only appearing in one result
• create a database of relavent keywords
• weight the pages based on the frequency of keyword distribution
• increasing the relevance of terms which appear in a small number of pages (as they are more likely to be on topic than words that appear in most all documents)
• normalize the page to remove the pagelength as a factor
• create relevancy vectors for the keywords

The single variable decomposition process is not scalable enough to work on large scale search engines though as it requires too much processor time. Multi dimentional scaling allows us to take snapshots of the topicology of different documents. "Instead of deriving the best possible projection through matrix decomposition, the MDS algorithm starts with a random arrangement of data, and then incrementally moves it around, calculating a stress function after each perturbation to see if the projection has grown more or less accurate. The algorithm keeps nudging the data points until it can no longer find lower values for the stress function."

This does not provide exact results, but only a rough approximation. When combined with other factors this approximation improves scalability and quality of search.

Good Reading on latent semantic indexing

This technology is so amazing that it may eventually help lead to a cure for cancer. Already the technology is being refined for cognitive improvements and test grading!

Many sites do not have a custom 404 error page. When a site visitor clicks on a dead link the visitor is most likely gone. Here is a perfect article about creating the perfect 404 error page.

Did it has released a new free keyword suggestion tool located at http://www.did-it.com/suggest.php. They analyzed the meta tags of millions of websites to create keyword relationships between them.

While not as robust as WordTracker, this is a great tool for free!

Some recent articles ("Google's House of Cards" "A Perfect Storm for Pay Per Click") have been saying that the ROI for paid advertising is going away. It has been. It was not very competitive a few years ago, but now with over 150,000 people in the market you need to be more effective to extract profits from a campaign.

I honestly think many of the articles are suggested / written by people who want to make their own jobs easier and make more money while doing it. Some articles may even be written to scare away competition or drive leads to firms who provide the services.

In the past a sloppy website with low conversion rates was ok because there was little competition. Now some areas are requiring a smooth ad, which is well targeted, that leads to a smooth site, which has great usability, and is customer centric. In essence the shakeup of the organic listings and the rising costs of pay per click ads are forcing websites (and the internet as a whole) to be more functional.

There are few mediums which have feedback as rapid as AdWords does. Pay per click is here to stay. Those who know how to use it will make a ton of money.

The most important part of SEO is off the page factors, but for those of you heavily interested in on the page criteria...
Over at GoRank they created a tool for comaring the keyword density of various pages. In addition they have a small report about their keyword density findings in the post Florida Google.