Google Robots.txt Wildcard

Not sure if I have seen this mentioned before. Dan Thies noticed Googlebot's wildcard robot.txt support:

Google's URL removal page contains a little bit of handy information that's not found on their webmaster info pages where it should be.

Google supports the use of 'wildcards' in robots.txt files. This isn't part of the original 1994 robots.txt protocol, and as far as I know, is not supported by other search engines. To make it work, you need to add a separate section for Googlebot in your robots.txt file. An example:

User-agent: Googlebot
Disallow: /*sort=

This would stop Googlebot from reading any URL that included the string &sort= no matter where that string occurs in the URL.

Good information to know if your site has recently suffered in Google due to duplicate content issues.

Dan also recently an SEO coach blog on his SEO Research Labs site.

Google Jagger 3 Update

Matt Cutts announced the Google Jagger 3 update is live at 66.102.9.104.

It sure is amazing the number of large vertical sites, .edu, and .gov results I saw in a few searches I did. Although there will probably still be a good amount of flux most the stuff I worked on seemed to get through ok.

I did see a bit of canonical URL issues, as noted by others on Matt's blog. Someone named Jason also left this gem in Matt's comments:

Our site has been negatively affected by Jagger. Therefore we just requested the transfer of 30,000 site wide links (paid in advance until July 06) to our main competitor who is currently ranked extremely well in Google for our main keyword.

Our entire website is legit SEO so our site wide links are the only thing that could have caused such a drastic drop in our ranking.

In a thread on SEW DaveN responded to a similar webmaster

IN life there are 2 ways to get on :

1) Be the best you can and move to the top

2) Drag everyone who is above you too below your level ..

Both ways you end up at the Top, it depends on how you view life and how long you want to stay there.

As long as Google is going to announce their updates and data centers, has anyone made a free SEO tool to easily compare / cross reference all the search results at various data centers? (Perhaps something like Myriad Search, but focuses on just one engine and lets the users select which data centers to compare.) I can't imagine it would be that hard to do unless Google blocked it, but they haven't been too aggressive in blocking web based SEO related tools (just look at all the tools SEO Chat has).

Today is the Right Time to Buy Old Sites...

I work by myself, and am always a bit scared of spreading myself too thin, so I have not been to active on the old domain buying front.

Having said that, now would probably be a good time to buy old domains. Jim Boykin again mentioned his new love for oldies and Graywolf said

Came to the same conclusion myself, emailed about 150 people picked up 2 domains from 2000 for under $1K.

Think of how cheap those site purchases are. Decent links can cost $50 to $300 or more each, so buying whole sites for $500 is cheap cheap cheap! How cheap is it? Even the most well known link broker is recommending buying a few old domains.

Why now is the perfect time to buy old domains:

  • It is right before the Christmas shopping season and many people not monetizing their sites might be able to use a bit of spare cash.

  • Many older domains are doing better than one would expect in Google's search results, which means they may recoup their costs quickly.
  • As Andy Hagans said, "Some older sites seem to be able to get away with murder in Google's search results."
  • Link popularity flowed much more naturally to commercial sites in the past than it does now. This means buying something with the a natural link profile may be far cheaper than it would be to try to reproduce similar linkage data.
  • At different times search algorithms show you different things. Before the Christmas shopping season each of the last few year it seems Google rolled out a new algorithm that wacked many sites which SEO'ed their way to the top (IMHO via link trading and low quality linkage data). Most of the algorithm changes are related to looking at linkage quality, communities, and ways to trust sites. The most recent update seems to have (at least temorarily) dialed up the weighting on TrustRank or a similar technology, which has had the net effect of highly ranking many old/trusted/authoritative sites that may lack some query specific authority. If you shop for sites that fit the current Google criteria well then add some good SEO to it you should be sitting good no matter which way the algorithms slide.

Before MSN was launched GoogleGuy recommended everyone taking a look at the MSN search results:

I recommend that everyone spend their full attention coming up to speed on beta.search.msn.com.

It's very rare to get to see a search engine in transition, because that's the best time to see what the different criteria are for ranking.

Now that Google is in a state of flux it might be a good time to perform many searches to look for some underpriced ad inventory. If you know what you are looking for you are more likely to find it in the organic search results than in the AdWords system.

The search vs SEO cat fight:

and going forward...

  • creating causes

  • social networking
  • buzz marketing

I think there is way more competition and SEO is way more complex than when I started learning about it, but that is offset in part by:

  • more searches

  • greater consumer trust of online shopping
  • many channels discussing the topic of SEO
  • many free tools (SEO and content management)
  • lower hosting costs
  • the speed at which viral stories spread if you can create one
  • the vastly expanding pool of options to use to monetize your sites

Why Off Topic Link Exchanges Suck

So I recently got this email:

Dear Webmaster,

As you are probably aware, Google has changed its algorythm and now removes sites from its search results when they have exchanged links with sites that are not in EXACTLY the same category.

To prevent being from blacklisted in Google, it is imperative that we remove our link to you and that your remove your link to us!

Our url is www.SITE.com.

We are removing our link to you now. PLEASE return the courtesy and remove your link to us!

Note that Google is updating its results this week and failure to remove these links immediately will likely mean not showing up in Google for AT LEAST the next 4 months!

Thank you for understanding,
Site.com Partners

The email is bogusly incorrect, and I don't think I traded links with the site mentioned, but that is the exact reason why this email is extra crappy.

If you trade links highly off topic you increase your risk profile, and if it helps you rank:

  • Whenever there is an update your competitors can send these remove my link reminders out for you.

  • There are only a limited number of relationships you can have. If you link out to a million sites your links out to junky sites will be a higher percentage than most sites, you will have more dead links than most quality sites, and many of those people will remove their links to you.
  • Your competitors could pay people from Jakarta $3 a day to go through your link trades and trade the same links.
  • Quality on topic sites may be less likely to link to you if your site frequently links off to low quality resources.

I think most sites which recently went south in Google probably lacked quality linkage data, not because they had too many links.

Traffic Power Has Not Yet Replied to the Motion for Summary Judgement

I recently spoke with the lawyer again about the evil / shoddy anti free speech lawsuit the fine folks at Traffic Power thrusted upon me.

About a month ago my lawyer filed a motion for summary judgement.

Stuff in the court system tends to drag out, but my lawyer said Traffic Power's response is at least a week late & he filed a reply of non oposition [doc] requestion that the motion be granted.

Link Monkeys and Links Within Content

Jim Boykin continues his ongoing rants about links:

Since most people are still thinking "the numbers game" when it comes to obtaining links, most people are buying "numbers" from "monkeys" on crappy link pages.

When will the world wake up that the numbers game has passed the tipping point in Google. Engine are trying to get smarter with how they analyze sites. My overall thought is that they are working to identify, simply, "Links within Content and Linking to Content"

Not too long ago when I interviewed NFFC he stated:

This is what I think, SEO is all about emotions, all about human interaction. People, search engineers even, try and force it into a numbers box.

Numbers, math and formulas are for people not smart enough to think in concepts.

Amazon to Sell Electronic Book Access

Amazon to Google: we own books...

From the Journal:

[Amazon] is introducing two new programs that allow consumers to buy online access to portions of a book or to the entire book, giving publishers and authors another way to generate revenue from their content.

Although Bezos does not come right out and say it, clearly this is a shot across the brow at Google, especially with the timing of their recent print offering.

While Amazon Chief Executive Jeff Bezos wouldn't comment specifically on the Google Print controversy, he said, "It's really important to do this cooperatively with the copyright holders, with the publishing community, with the authors. We're going to keep working in that cooperative vein."

After Google develops their micropayment system I bet they also directly broker a large amount of media.

New Google User Profiling Patent

Loren does a good rundown of a new Google patent Personalization of placed content ordering in search results in his Organic Results Ranked by User Profiling post. Some of the things in the patent may be a bit ahead of themselves, but the thesis is...

GenericScore=QueryScore*PageRank.

This GenericScore may not appropriately reflect the site's importance to a particular user if the user's interests or preferences are dramatically different from that of the random surfer. The relevance of a site to user can be accurately characterized by a set of profile ranks, based on the correlation between a sites content and the user's term-based profile, herein called the TermScore, the correlation between one or more categories associated with a site and user's category-based profile, herein called the CategoryScore, and the correlation between the URL and/or host of the site and user's link-based profile, herein called the LinkScore. Therefore, the site may be assigned a personalized rank that is a function of both the document's generic score and the user profile scores. This personalized score can be expressed as: PersonalizedScore=GenericScore*(TermScore+CategoryScore+LinkScore).

For those big into patents: Stephen Arnold has a $50 CD for sale containing over 120 Google patent related documents.

I think he could sell that as a subscription service, so long as people didn't know all the great stuff Gary Price compiles for free. (Link from News.com)

MicroSoft Buys Into VOIP, Again

TheStreet reports M$ looking at VOIP again:

Microsoft said Thursday it has agreed to buy Media-streams.com, a privately held firm in Zurich, Switzerland. Financial terms were not disclosed.

Media-stream's VoIP technology, which enables telephone calls over the Internet, will become a core part of Microsoft's platform that enables workers to use the Web to collaborate on projects. Microsoft envisions such collaboration encompassing several different modes of communication, including email, instant messaging, Web conferencing and telephone calls via the Internet.

Media-streams is the second VoIP firm acquired by Microsoft in the last few months. In August, Microsoft acquired Teleo

MicroSoft Windows Live Launched

Windows Live software platform getting plenty of buzz.

Firefox Support coming soon. No surprise there. Search champ unimpressed. So are others.

If MicroSoft would learn to do the small things right they wouldn't need to try to create Monopoly 4.0 and Robert would not need to post why so many people do not trust them even though he knows how to fix it.

At least Robert has great job security. EVERYTHING should be secure.

Pages