DMOZ Shut Down

Last August I wrote a blog post about how attention merchants were sucking the value out of online publishing. In it I noted how the Yahoo! Directory disappeared & how even DMOZ saw a sharp drop in traffic & rankings over the past few years.

The concept of a neutral web is dead. In its place is agenda-driven media.

  • Politically charged misinformed snippets.
  • Ads cloaked as content.
  • Public relations propaganda.
  • Mostly correct (but politically insensitive) articles being "fact checked" where a minor detail is disputed to label the entire piece as not credible.

As the tech oligarchs broadly defund publishing, the publishers still need to eat. Aggregate information quality declines to make the numbers work. Companies which see their ad revenues slide 20%, 30% or 40% year after year can't justify maintaining the labor-intensive yet unmonetized side projects.

There is Wikipedia, but it is not without bias & beyond the value expressed in the hidden bias most of the remaining value from it flows on through to the attention merchant / audience aggregation / content scraper platforms.

Last month DMOZ announced they were closing on March 14th without much fanfare. And on March 17th the directory went offline.

A number of people have pushed to preserve & archive the DMOZ data. Some existing DMOZ editors are planning on launching a new directory under a different name but as of the 17th DMOZ editors put up a copy at dmoztools.net. Jim Boykin scraped DMOZ & uploaded a copy here. A couple other versions of DMOZ have been published at OpenDirectoryProject.org & Freemoz.org.

DMOZ was not without criticism or controversy,

Although site policies suggest that an individual site should be submitted to only one category, as of October 2007, Topix.com, a news aggregation site operated by DMOZ founder Rich Skrenta, has more than 17,000 listings.

Early in the history of DMOZ, its staff gave representatives of selected companies, such as Rolling Stone or CNN, editing access in order to list individual pages from their websites. Links to individual CNN articles were added until 2004, but were entirely removed from the directory in January 2008 due to the content being outdated and not considered worth the effort to maintain.

but by-and-large it added value to the structure of the web.

As search has advanced (algorithmic evolution, economic power, influence over publishers, enhanced bundling of distribution & user tracking) general web directories haven't been able to keep pace. Ultimately the web is a web of links & pages rather than a web of sites. Many great sites span multiple categories. Every large quality site has some misinformation on it. Every well-known interactive site has some great user contributions & user generated spam on it. Search engines have better signals about what pages are important & which pages have maintained importance over time. As search engines have improved link filtering algorithms & better incorporated user tracking in rankings, broad-based manual web directories had no chance.

The web of pages vs web of sites concept can be easily observed in how some of the early successful content platforms have broken down their broad-based content portals into a variety of niche sites.

When links were (roughly) all that mattered, leveraging a website's link authority meant it was far more profitable for a large entity to keep publishing more content on the one main site. That is how eHow became the core of a multi-billion Dollar company.

Demand Media showed other publishers the way. And if the other existing sites were to stay competitive, they also had to water down content quality to make the numbers back out. The problem with this was the glut of content was lower ad rates. And the decline in ad rates was coupled with a shift away from a links-only view of search relevancy to a model based on weighting link profiles against user engagement metrics.

Websites with lots of links, lots of thin content & terrible engagement metrics were hit.

Kristen Moore, vp of marketing for Demand Media, explained what drove the most egregious aspects of eHow's editorial strategy: “There’s some not very bright people out there.”

eHow improved their site design, drastically reduced their ad density, removed millions of articles from their site, and waited. However nothing they did on that domain name was ever going to work. They dug too deep of a hole selling the growth story to pump a multi-billion Dollar valuation. And they generated so much animosity from journalists who felt overwork & underpaid that even when they did rank journalists would typically prefer to link to anything but them.

The flip side of that story is the newspaper chains, which rushed to partner with Demand Media to build eHow-inspired sections on their sites.

Brands which enjoy the Google brand subsidy are also quite hip to work with Demand Media, which breathes new life into once retired content: "Sometimes Demand will even dust off old content that’s been published but is no longer live and repurpose it for a brand."

As Facebook & Google grew more dominant in the online ad ecosystem they aggressively moved to suck in publisher content & shift advertiser spend onto their core properties. The rise of time spent on social sites only made it harder for websites to be sought out destination. Google also effectively cut off direct distribution by consolidating & de-monetizing the RSS reader space then shutting down a project they easily could have left run.

As the web got more competitive, bloggers & niche publications which were deeply specialized were able to steal marketshare in key verticals by leveraging a differentiated editorial opinion.

Even if they couldn't necessarily afford to build strong brands via advertising, they were worthy of a follow on some social media channels & perhaps an email subscription. And the best niche editorial remains worthy of a direct visit:

Everything about Techmeme and its lingering success seems to defy the contemporary wisdom of building a popular website. It publishes zero original reporting and is not a social network. It doesn’t have a mobile app or a newsletter or even much of a social presence beyond its Twitter account, which posts dry commodity news with zero flair for clickability.

As a work around to the Panda hits, sites like eHow are now becoming collections of niche-focused sites (Cuteness.com, Techwalla.com, Sapling.com, Leaf.tv, etc will join Livestrong.com & eHow.com). It appears to be working so far...

...but they may only be 1 Panda update away from finding out the new model isn't sustainable either.

About.com has done the same thing (TheSpruce.com, Verywell.com, Lifewire.com, TheBalance.com). Hundreds of millions of Dollars are riding on the hope that as the algorithms keep getting more granular they won't discover moving the content to niche brands wasn't enough.

As content moves around search engines with billions of Dollars in revenue can recalibrate rankings for each page & adjust rankings based on user experience. Did an influential "how to" guide become irrelevant after a software or hardware update? If so, they can see it didn't solve the user's problem and rank a more recent document which reflects the current software or hardware. Is a problem easy to solve with a short snippet of content? If so, that can get scraped into the search results.

Web directories which are built around sites rather than pages have no chance of competing against the billions of Dollars of monthly search ads & the full cycle user tracking search companies like Google & Bing can do with their integrated search engines, ad networks, web browsers & operating systems.

Arguably in most cases the idea of neutral-based publishing no longer works on the modern web. The shill gets exclusive stories. The political polemic gets automatic retweets from those who identify. The content which lacks agenda probably lacks the economics to pay for ads & buy distribution unless people can tell the creator loves what they do so much it influences them enough to repeatedly visit & perhaps pay for access.

New gTLDs are Like Used Cars

There may be a couple exceptions which prove the rule, but new TLDs are generally an awful investment for everyone except the registry operator.

Here is the short version...

And the long version...

Diminishing Returns

About a half-decade ago I wrote about how Google devalued domain names from an SEO perspective & there have been a number of leading "category killer" domains which have repeatedly been recycled from startup to acquisition to shut down to PPC park page to buy now for this once in a lifetime opportunity in an endless water cycle.

The central web platforms are becoming ad heavy, which in turn decreases the reach of anything which is not an advertisement. For the most valuable concepts / markets / keywords ads eat up the entire interface for the first screen full of results. Key markets like hotels might get a second round of vertical ads to further displace the concept of organic results.

Proprietary, Closed-Ecosystem Roach Motels

The tech monopolies can only make so much money by stuffing ads onto their own platform. To keep increasing their take they need to increase the types, varieties & formats of media they host and control & keep the attention on their platform.

Both Google & Facebook are promoting scams where they feed on desperate publishers & suck a copy of the publisher's content into being hosted by the tech monopoly platform de jour & sprinkle a share of the revenues back to the content sources.

They may even pay a bit upfront for new content formats, but then after the market is primed the deal shifts to where (once again) almost nobody other than the tech monopoly platform wins.

The attempt to "own" the web & never let users go is so extreme both companies will make up bogus statistics to promote their proprietary / fake open / actually closed standards.

If you ignore how Google's AMP double, triple, or quadruple counts visitors in Google Analytics the visit numbers look appealing.

But the flip side of those fake metrics is actual revenues do not flow.

Facebook has the same sort of issues, with frequently needing to restate various metrics while partners fly blind.

These companies are restructuring society & the race to the bottom to try to make the numbers work in an increasingly unstable & parasitic set of platform choices is destroying adjacent markets:

Have you tried Angry Birds lately? It’s a swamp of dark patterns. All extractive logic meant to trick you into another in-app payment. It’s the perfect example of what happens when product managers have to squeeze ever-more-growth out of ever-less-fertile lands to hit their targets year after year. ... back to the incentives. It’s not just those infused by venture capital timelines and return requirements, but also the likes of tax incentives favoring capital gains over income. ... that’s the truly insidious part of the tech lords solution to everything. This fantasy that they will be greeted as liberators. When the new boss is really a lot like the old boss, except the big stick is replaced with the big algorithm. Depersonalizing all punishment but doling it out just the same. ... this new world order is being driven by a tiny cabal of monopolies. So commercial dissent is near impossible. ... competition is for the little people. Pitting one individual contractor against another in a race to the bottom. Hoarding all the bargaining power at the top. Disparaging any attempts against those at the bottom to organize with unions or otherwise.

To be a success on the attention platforms you have to push toward the edges. But as you become successful you become a target.

And the dehumanized "algorithm" is not above politics & public relations.

Pewdiepie is the biggest success story on the YouTube platform. When he made a video showing some of the absurd aspects of Fiverr it led to a WSJ investigation which "uncovered" a pattern of anti-semitism. And yet one of the reporters who worked on that story wrote far more offensive and anti-semetic tweets. The hypocrisy of the hit job didn't matter. They still were able to go after Pewdiepie's ad relationships to cut him off from Disney's Maker Studios & the premium tier of YouTube ads.

The fact that he is an individual with broad reach means he'll still be fine economically, but many other publishers would quickly end up in a death spiral from the above sequence.

If it can happen to a leading player in a closed ecosystem then the risk to smaller players is even greater.

In some emerging markets Facebook effectively *is* the Internet.

The Decline of Exact Match Domains

Domains have been so devalued (from an SEO perspective) that some names like PaydayLoans.net sell for about $3,000 at auction.

$3,000 can sound like a lot to someone with no money, but names like that were going for 6 figures at their peak.

Professional domain sellers participate in the domain auctions on sites like NameJet & SnapNames. Big keywords like [payday loans] in core trusted extensions are not missed. So if the 98% decline in price were an anomaly, at least one of them would have bid more in that auction.

Why did exact match domains fall so hard? In part because Google shifted from scoring the web based on links to considering things like brand awareness in rankings. And it is very hard to run a large brand-oriented ad campaign promoting a generically descriptive domain name. Sure there are a few exceptions like Cars.com & Hotels.com, but if you watch much TV you'll see a lot more ads associated with businesses that are not built on generically descriptive domain names.

Not all domains have fallen quite that hard in price, but the more into the tail you go the less the domain acts as a memorable differentiator. If the barrier to entry increases, then the justification for spending a lot on a domain name as part of a go to market strategy makes less sense.

Brandable Names Also Lost Value

Arguably EMDs have lost more value than brandable domain names, but even brandable names have sharply slid.

If you go back a decade or two tech startups would secure their name (say Snap.com or Monster.com or such) & then try to build a business on it.

But in the current marketplace with there being many paths to market, some startups don't even have a domain name at launch, but begin as iPhone or Android apps.

Now people try to create success on a good enough, but cheap domain name & then as success comes they buy a better domain name.

Jelly was recently acquired by Pinterest. Rather than buying jelly.com they were still using AskJelly.com for their core site & Jelly.co for their blog.

As long as domain redirects work, there's no reason to spend heavily on a domain name for a highly speculative new project.

Rather than spending 6 figures on a domain name & then seeing if there is market fit, it is far more common to launch a site on something like getapp.com, joinapp.com, app.io, app.co, businessnameapp.com, etc.

This in turn means that rather than 10,000s of startups all chasing their core .com domain name off the start, people test whatever is good enough & priced close to $10. Then only after they are successful do they try to upgrade to better, more memorable & far more expensive domain names.

Money isn't spent on the domain names until the project has already shown market fit.

One in a thousand startups spending $1 million is less than one in three startups spending $100,000.

New TLDs Undifferentiated, Risky & Overpriced

No Actual Marketing Being Done

Some of the companies which are registries for new TLDs talk up investing in marketing & differentiation for the new TLDs, but very few of them are doing much on the marketing front.

You may see their banner ads on domainer blogs & they may even pay for placement with some of the registries, but there isn't much going on in terms of cultivating a stable ecosystem.

When Google or Facebook try to enter & dominate a new vertical, the end destination may be extractive rent seeking by a monopoly BUT off the start they are at least willing to shoulder some of the risk & cost upfront to try to build awareness.

Where are the domain registries who have built successful new businesses on some of their new TLDs? Where are the subsidies offered to key talent to help drive awareness & promote the new strings?

As far as I know, none of that stuff exists.

In fact, what is prevalent is the exact opposite.

Greed-Based Anti-Marketing

So many of them are short sighted greed-based plays that they do the exact opposite of building an ecosystem ... they hold back any domain which potentially might not be complete garbage so they can juice it for a premium ask price in the 10s of thousands of dollars.

While searching on GoDaddy Auctions for a client project I have seen new TLDs like .link listed for sale for MORE THAN the asking price of similar .org names.

If those prices had any sort of legitimate foundation then the person asking $30,000 for a .link would have bulk bought all the equivalent .net and .org names which are listed for cheaper prices.

But the prices are based on fantasy & almost nobody is dumb enough to pay those sorts of prices.

Anyone dumb enough to pay that would be better off buying their own registry rather than a single name.

The holding back of names is the exact opposite of savvy marketing investment. It means there's no reason to use the new TLD if you either have to pay through the nose or use a really crappy name nobody will remember.

I didn’t buy more than 15 of Uniregistry’s domains because all names were reserved in the first place and I didn’t feel like buying 2nd tier domains ... Domainers were angry when the first 2 Uniregistry’s New gTLDs (.sexy and .tattoo) came out and all remotely good names were reserved despite Frank saying that Uniregistry would not reserve any domains.

Who defeats the race to the bottom aspects of the web by starting off from a "we only sell shit" standpoint?

Nobody.

And that's why these new TLDs are a zero.

Defaults Have Value

Many online verticals are driven by winner take most monopoly economics. There's a clear dominant leader in each of these core markets: social, search, short-form video, long-form video, e-commerce, auctions, real estate, job search, classifieds, etc. Some other core markets have consolidated down to 3 or 4 core players who among them own about 50 different brands that attack different parts of the market.

Almost all the category leading businesses which dominate aggregate usage are on .com domains.

Contrast the lack of marketing for new TLDs with all the marketing one sees for the .com domain name.

Local country code domain names & .com are not going anywhere. And both .org and .net are widely used & unlikely to face extreme price increases.

Hosing The Masses...

A decade ago domainers were frustrated Verisign increased the price of .com domains in ~ 5% increments:

Every mom, every pop, every company that holds a domain name had no say in the matter. ICANN basically said to Verisign: "We agree to let you hose the masses if you stop suing us".

I don't necessarily mind paying more for domains so much as I mind the money going to a monopolistic regulator which has historically had little regard for the registrants/registrars it should be serving

Those 5% or 10% shifts were considered "hosing the masses."

Imagine what sort of blowback PIR would get from influential charities if they tried to increase the price of .org domains 30-fold overnight. It would be such a public relations disaster it would never be considered.

Domain registries are not particularly expensive to run. A person who has a number of them can run each of them for less than the cost of a full time employee - say $25,000 to $50,00 per year.

And yet, the very people who complained about Verisign's benign price increases, monopolistic abuses & rent extraction are now pushing massive price hikes:

.Hosting and .juegos are going up from about $10-$20 retail to about $300. Other domains will also see price increases.
...
Here's the thing with new TLD pricing: registry operators can increase prices as much as they want with just six months' notice.
...
in its applications, Uniregistry said it planned to enter into a contractual agreement to not increase its prices for five years.

Why would anyone want to build a commercial enterprise (or anything they care about) on such a shoddy foundation?

If a person promises...

  • no hold backs of premium domains, then reserves 10s of thousands of domains
  • no price hikes for 5 years, then hikes prices
  • the eventual price hikes being inline with inflation, then hikes prices 3,000%

That's 3 strikes and the batter is out.

Doing the Math

The claim the new TLDs need more revenues to exist are untrue. Running an extension costs maybe $50,000 per year. If a registry operator wanted to build a vibrant & stable ecosystem the first step would be dumping the concept of premium domains to encourage wide usage & adoption.

There are hundreds of these new TLD extensions and almost none of them can be trusted to be a wise investment when compared against similar names in established extensions like .com, .net, .org & CCTLDs like .co.uk or .fr.

There's no renewal price protection & there's no need, especially as secondary market prices on the core TLDs have sharply come down.

Domain Pricing Trends

Aggregate stats are somewhat hard to come by as many deals are not reported publicly & many sites which aggregate sales data also list minimum prices.

However domains have lost value for many reasons

  • declining SEO-related value due to the search results becoming over-run with ads (Google keeps increasing their ad clicks 20% to 30% year over year)
  • broad market consolidation in key markets like travel, ecommerce, search & social
    • Google & Facebook are eating OVER 100% of online advertising growth - the rest of industry is shrinking in aggregate
    • are there any major news sites which haven't struggled to monetize mobile?
    • there is a reason there are few great indy blogs compared to a decade ago
  • rising technical costs in implementing independent websites (responsive design, HTTPS, AMP, etc.) "Closed platforms increase the chunk size of competition & increase the cost of market entry, so people who have good ideas, it is a lot more expensive for their productivity to be monetized. They also don't like standardization ... it looks like rent seeking behaviors on top of friction" - Gabe Newell
  • harder to break into markets with brand-biased relevancy algorithms (increased chunk size of competition)
  • less value in trying to build a brand on a generic name, which struggles to rank in a landscape of brand-biased algorithms (inability to differentiate while being generically descriptive)
  • decline in PPC park page ad revenues
    • for many years Yahoo! hid the deterioration in their core business by relying heavily on partners for ad click volumes, but after they switched to leveraging Bing search, Microsoft was far more interested with click quality vs click quantity
    • absent the competitive bid from Yahoo!, Google drastically reduced partner payouts
    • most web browsers have replaced web address bars with dual function search boxes, drastically reducing direct navigation traffic

All the above are the mechanics of "why" prices have been dropping, but it is also worth noting many of the leading portfolios have been sold.

If the domain aftermarket is as vibrant as some people claim, there's no way the Marchex portfolio of 200,000+ domains would have sold for only $28.1 million a couple years ago.

RegistrarStats shows .com registrations have stopped growing & other extensions like .net, .org, .biz & .info are now shrinking.

Both aftermarket domain prices & the pool of registered domains on established gTLDs are dropping.

I know I've dropped hundreds & hundreds of domains over the past year. That might be due to my cynical views of the market, but I did hold many names for a decade or more.

As barrier to entry increases, many of the legacy domains which could have one day been worth developing have lost much of their value.

And the picked over new TLDs are an even worse investment due to the near infinite downside potential of price hikes, registries outright folding, etc.

Most of the registration graphs for new TLDs are far uglier than the one posted above. China will not save the new gTLDs.

Into this face of declining value there is a rush of oversupply WITH irrational above-market pricing. And then the registries which spend next to nothing on marketing can't understand why their great new namespaces went nowhere.

As much as I cringe at .biz & .info, I'd prefer either of them over just about any new TLD.

Any baggage they may carry is less than the risk of going with an unproven new extension without any protections whatsoever.

Losing Faith in the Zimbabwe Dollar

Who really loses is anyone who read what these domain registry operators wrote & trusted them.

Uniregistry does not believe that registry fees should rise when the costs of other technology services have uniformly trended downward, simply because a registry operator believes it can extract higher profit from its base of registrants.

How does one justify a 3000% price hike after stating "Our prices are fixed and only indexed to inflation after 5 years."

Are they pricing these names in Zimbabwe Dollars? Or did they just change their minds in a way that hurt anyone who trusted them & invested in their ecosystem?

Frank Schilling warned about the dangers of lifting price controls

The combination of "presumptive renewal" and the "lifting of price controls on registry services" is incredibly dangerous.
Imagine buying a home, taking on a large mortgage, remodeling, moving in, only to be informed 6 months later that your property taxes will go up 10,000% with no better services offered by local government. The government doesn't care if you can't pay your tax/mortgage because they don't really want you to pay your tax… they want you to abandon your home so they can take your property and resell it to a higher payer for more money, pocketing the difference themselves, leaving you with nothing.

This agreement as written leaves the door open to exactly that type of scenario

He didn't believe the practice to be poor.

Rather he felt he would have been made poorer, unless he was the person doing it:

It would be the mother of all Internet tragedies and a crippling blow to ICANN’s relevance if millions of pioneering registrants were taxed out of their internet homes as a result of the greed of one registry and the benign neglect, apathy or tacit support of its master.

It is a highly nuanced position.

Update: Shortly after the sharp pricing increases were announced GoDaddy dropped Uniregistry domain names. Tucows later followed suit in dropping the new TLDs with exponential price increases.

Update 2: Many new TLDs exist primarily to arbitrage existing brands & monetized defensive trademarks. With that in mind, trademark holders have been more aggressive with trying to push trademark protections on the new TLDs even for nebulous misspellings. The EFF has recommended avoiding new gTLDs.

you'll want to think twice about registering in any of the newer global top-level domains (gTLDs), which provide brand owners access to a privately-run Trademark Clearinghouse that gives them veto powers that go far beyond those they would receive under the trademark law of the United States or those of most other countries.