When SEO Was Easy

When I got started on the web over 15 years ago I created an overly broad & shallow website that had little chance of making money because it was utterly undifferentiated and crappy. In spite of my best (worst?) efforts while being a complete newbie, sometimes I would go to the mailbox and see a check for a couple hundred or a couple thousand dollars come in. My old roommate & I went to Coachella & when the trip was over I returned to a bunch of mail to catch up on & realized I had made way more while not working than what I spent on that trip.

What was the secret to a total newbie making decent income by accident?

Horrible spelling.

Back then search engines were not as sophisticated with their spelling correction features & I was one of 3 or 4 people in the search index that misspelled the name of an online casino the same way many searchers did.

The high minded excuse for why I did not scale that would be claiming I knew it was a temporary trick that was somehow beneath me. The more accurate reason would be thinking in part it was a lucky fluke rather than thinking in systems. If I were clever at the time I would have created the misspeller's guide to online gambling, though I think I was just so excited to make anything from the web that I perhaps lacked the ambition & foresight to scale things back then.

In the decade that followed I had a number of other lucky breaks like that. One time one of the original internet bubble companies that managed to stay around put up a sitewide footer link targeting the concept that one of my sites made decent money from. This was just before the great recession, before Panda existed. The concept they targeted had 3 or 4 ways to describe it. 2 of them were very profitable & if they targeted either of the most profitable versions with that page the targeting would have sort of carried over to both. They would have outranked me if they targeted the correct version, but they didn't so their mistargeting was a huge win for me.

Search Gets Complex

Search today is much more complex. In the years since those easy-n-cheesy wins, Google has rolled out many updates which aim to feature sought after destination sites while diminishing the sites which rely one "one simple trick" to rank.

Arguably the quality of the search results has improved significantly as search has become more powerful, more feature rich & has layered in more relevancy signals.

Many quality small web publishers have went away due to some combination of increased competition, algorithmic shifts & uncertainty, and reduced monetization as more ad spend was redirected toward Google & Facebook. But the impact as felt by any given publisher is not the impact as felt by the ecosystem as a whole. Many terrible websites have also went away, while some formerly obscure though higher-quality sites rose to prominence.

There was the Vince update in 2009, which boosted the rankings of many branded websites.

Then in 2011 there was Panda as an extension of Vince, which tanked the rankings of many sites that published hundreds of thousands or millions of thin content pages while boosting the rankings of trusted branded destinations.

Then there was Penguin, which was a penalty that hit many websites which had heavily manipulated or otherwise aggressive appearing link profiles. Google felt there was a lot of noise in the link graph, which was their justification for the Penguin.

There were updates which lowered the rankings of many exact match domains. And then increased ad load in the search results along with the other above ranking shifts further lowered the ability to rank keyword-driven domain names. If your domain is generically descriptive then there is a limit to how differentiated & memorable you can make it if you are targeting the core market the keywords are aligned with.

There is a reason eBay is more popular than auction.com, Google is more popular than search.com, Yahoo is more popular than portal.com & Amazon is more popular than a store.com or a shop.com. When that winner take most impact of many online markets is coupled with the move away from using classic relevancy signals the economics shift to where is makes a lot more sense to carry the heavy overhead of establishing a strong brand.

Branded and navigational search queries could be used in the relevancy algorithm stack to confirm the quality of a site & verify (or dispute) the veracity of other signals.

Historically relevant algo shortcuts become less appealing as they become less relevant to the current ecosystem & even less aligned with the future trends of the market. Add in negative incentives for pushing on a string (penalties on top of wasting the capital outlay) and a more holistic approach certainly makes sense.

Modeling Web Users & Modeling Language

PageRank was an attempt to model the random surfer.

When Google is pervasively monitoring most users across the web they can shift to directly measuring their behaviors instead of using indirect signals.

Years ago Bill Slawski wrote about the long click in which he opened by quoting Steven Levy's In the Plex: How Google Thinks, Works, and Shapes our Lives

"On the most basic level, Google could see how satisfied users were. To paraphrase Tolstoy, happy users were all the same. The best sign of their happiness was the "Long Click" — This occurred when someone went to a search result, ideally the top one, and did not return. That meant Google has successfully fulfilled the query."

Of course, there's a patent for that. In Modifying search result ranking based on implicit user feedback they state:

user reactions to particular search results or search result lists may be gauged, so that results on which users often click will receive a higher ranking. The general assumption under such an approach is that searching users are often the best judges of relevance, so that if they select a particular search result, it is likely to be relevant, or at least more relevant than the presented alternatives.

If you are a known brand you are more likely to get clicked on than a random unknown entity in the same market.

And if you are something people are specifically seeking out, they are likely to stay on your website for an extended period of time.

One aspect of the subject matter described in this specification can be embodied in a computer-implemented method that includes determining a measure of relevance for a document result within a context of a search query for which the document result is returned, the determining being based on a first number in relation to a second number, the first number corresponding to longer views of the document result, and the second number corresponding to at least shorter views of the document result; and outputting the measure of relevance to a ranking engine for ranking of search results, including the document result, for a new search corresponding to the search query. The first number can include a number of the longer views of the document result, the second number can include a total number of views of the document result, and the determining can include dividing the number of longer views by the total number of views.

Attempts to manipulate such data may not work.

safeguards against spammers (users who generate fraudulent clicks in an attempt to boost certain search results) can be taken to help ensure that the user selection data is meaningful, even when very little data is available for a given (rare) query. These safeguards can include employing a user model that describes how a user should behave over time, and if a user doesn't conform to this model, their click data can be disregarded. The safeguards can be designed to accomplish two main objectives: (1) ensure democracy in the votes (e.g., one single vote per cookie and/or IP for a given query-URL pair), and (2) entirely remove the information coming from cookies or IP addresses that do not look natural in their browsing behavior (e.g., abnormal distribution of click positions, click durations, clicks_per_minute/hour/day, etc.). Suspicious clicks can be removed, and the click signals for queries that appear to be spmed need not be used (e.g., queries for which the clicks feature a distribution of user agents, cookie ages, etc. that do not look normal).

And just like Google can make a matrix of documents & queries, they could also choose to put more weight on search accounts associated with topical expert users based on their historical click patterns.

Moreover, the weighting can be adjusted based on the determined type of the user both in terms of how click duration is translated into good clicks versus not-so-good clicks, and in terms of how much weight to give to the good clicks from a particular user group versus another user group. Some user's implicit feedback may be more valuable than other users due to the details of a user's review process. For example, a user that almost always clicks on the highest ranked result can have his good clicks assigned lower weights than a user who more often clicks results lower in the ranking first (since the second user is likely more discriminating in his assessment of what constitutes a good result). In addition, a user can be classified based on his or her query stream. Users that issue many queries on (or related to) a given topic T (e.g., queries related to law) can be presumed to have a high degree of expertise with respect to the given topic T, and their click data can be weighted accordingly for other queries by them on (or related to) the given topic T.

Google was using click data to drive their search rankings as far back as 2009. David Naylor was perhaps the first person who publicly spotted this. Google was ranking Australian websites for [tennis court hire] in the UK & Ireland, in part because that is where most of the click signal came from. That phrase was most widely searched for in Australia. In the years since Google has done a better job of geographically isolating clicks to prevent things like the problem David Naylor noticed, where almost all search results in one geographic region came from a different country.

Whenever SEOs mention using click data to search engineers, the search engineers quickly respond about how they might consider any signal but clicks would be a noisy signal. But if a signal has noise an engineer would work around the noise by finding ways to filter the noise out or combine multiple signals. To this day Google states they are still working to filter noise from the link graph: "We continued to protect the value of authoritative and relevant links as an important ranking signal for Search."

The site with millions of inbound links, few intentional visits & those who do visit quickly click the back button (due to a heavy ad load, poor user experience, low quality content, shallow content, outdated content, or some other bait-n-switch approach)...that's an outlier. Preventing those sorts of sites from ranking well would be another way of protecting the value of authoritative & relevant links.

Best Practices Vary Across Time & By Market + Category

Along the way, concurrent with the above sorts of updates, Google also improved their spelling auto-correct features, auto-completed search queries for many years through a featured called Google Instant (though they later undid forced query auto-completion while retaining automated search suggestions), and then they rolled out a few other algorithms that further allowed them to model language & user behavior.

Today it would be much harder to get paid above median wages explicitly for sucking at basic spelling or scaling some other individual shortcut to the moon, like pouring millions of low quality articles into a (formerly!) trusted domain.

Nearly a decade after Panda, eHow's rankings still haven't recovered.

Back when I got started with SEO the phrase Indian SEO company was associated with cut-rate work where people were buying exclusively based on price. Sort of like a "I got a $500 budget for link building, but can not under any circumstance invest more than $5 in any individual link." Part of how my wife met me was she hired a hack SEO from San Diego who outsourced all the work to India and marked the price up about 100-fold while claiming it was all done in the United States. He created reciprocal links pages that got her site penalized & it didn't rank until after she took her reciprocal links page down.

With that sort of behavior widespread (hack US firm teaching people working in an emerging market poor practices), it likely meant many SEO "best practices" which were learned in an emerging market (particularly where the web was also underdeveloped) would be more inclined to being spammy. Considering how far ahead many Western markets were on the early Internet & how India has so many languages & how most web usage in India is based on mobile devices where it is hard for users to create links, it only makes sense that Google would want to place more weight on end user data in such a market.

If you set your computer location to India Bing's search box lists 9 different languages to choose from.

The above is not to state anything derogatory about any emerging market, but rather that various signals are stronger in some markets than others. And competition is stronger in some markets than others.

Search engines can only rank what exists.

"In a lot of Eastern European - but not just Eastern European markets - I think it is an issue for the majority of the [bream? muffled] countries, for the Arabic-speaking world, there just isn't enough content as compared to the percentage of the Internet population that those regions represent. I don't have up to date data, I know that a couple years ago we looked at Arabic for example and then the disparity was enormous. so if I'm not mistaken the Arabic speaking population of the world is maybe 5 to 6%, maybe more, correct me if I am wrong. But very definitely the amount of Arabic content in our index is several orders below that. So that means we do not have enough Arabic content to give to our Arabic users even if we wanted to. And you can exploit that amazingly easily and if you create a bit of content in Arabic, whatever it looks like we're gonna go you know we don't have anything else to serve this and it ends up being horrible. and people will say you know this works. I keyword stuffed the hell out of this page, bought some links, and there it is number one. There is nothing else to show, so yeah you're number one. the moment somebody actually goes out and creates high quality content that's there for the long haul, you'll be out and that there will be one." - Andrey Lipattsev – Search Quality Senior Strategist at Google Ireland, on Mar 23, 2016

Impacting the Economics of Publishing

Now search engines can certainly influence the economics of various types of media. At one point some otherwise credible media outlets were pitching the Demand Media IPO narrative that Demand Media was the publisher of the future & what other media outlets will look like. Years later, after heavily squeezing on the partner network & promoting programmatic advertising that reduces CPMs by the day Google is funding partnerships with multiple news publishers like McClatchy & Gatehouse to try to revive the news dead zones even Facebook is struggling with.

"Facebook Inc. has been looking to boost its local-news offerings since a 2017 survey showed most of its users were clamoring for more. It has run into a problem: There simply isn’t enough local news in vast swaths of the country. ... more than one in five newspapers have closed in the past decade and a half, leaving half the counties in the nation with just one newspaper, and 200 counties with no newspaper at all."

As mainstream newspapers continue laying off journalists, Facebook's news efforts are likely to continue failing unless they include direct economic incentives, as Google's programmatic ad push broke the banner ad:

"Thanks to the convoluted machinery of Internet advertising, the advertising world went from being about content publishers and advertising context—The Times unilaterally declaring, via its ‘rate card’, that ads in the Times Style section cost $30 per thousand impressions—to the users themselves and the data that targets them—Zappo’s saying it wants to show this specific shoe ad to this specific user (or type of user), regardless of publisher context. Flipping the script from a historically publisher-controlled mediascape to an advertiser (and advertiser intermediary) controlled one was really Google’s doing. Facebook merely rode the now-cresting wave, borrowing outside media’s content via its own users’ sharing, while undermining media’s ability to monetize via Facebook’s own user-data-centric advertising machinery. Conventional media lost both distribution and monetization at once, a mortal blow."

Google is offering news publishers audience development & business development tools.

Heavy Investment in Emerging Markets Quickly Evolves the Markets

As the web grows rapidly in India, they'll have a thousand flowers bloom. In 5 years the competition in India & other emerging markets will be much tougher as those markets continue to grow rapidly. Media is much cheaper to produce in India than it is in the United States. Labor costs are lower & they never had the economic albatross that is the ACA adversely impact their economy. At some point the level of investment & increased competition will mean early techniques stop having as much efficacy. Chinese companies are aggressively investing in India.

“If you break India into a pyramid, the top 100 million (urban) consumers who think and behave more like Americans are well-served,” says Amit Jangir, who leads India investments at 01VC, a Chinese venture capital firm based in Shanghai. The early stage venture firm has invested in micro-lending firms FlashCash and SmartCoin based in India. The new target is the next 200 million to 600 million consumers, who do not have a go-to entertainment, payment or ecommerce platform yet— and there is gonna be a unicorn in each of these verticals, says Jangir, adding that it will be not be as easy for a player to win this market considering the diversity and low ticket sizes.

RankBrain

RankBrain appears to be based on using user clickpaths on head keywords to help bleed rankings across into related searches which are searched less frequently. A Googler didn't state this specifically, but it is how they would be able to use models of searcher behavior to refine search results for keywords which are rarely searched for.

In a recent interview in Scientific American a Google engineer stated: "By design, search engines have learned to associate short queries with the targets of those searches by tracking pages that are visited as a result of the query, making the results returned both faster and more accurate than they otherwise would have been."

Now a person might go out and try to search for something a bunch of times or pay other people to search for a topic and click a specific listing, but some of the related Google patents on using click data (which keep getting updated) mentioned how they can discount or turn off the signal if there is an unnatural spike of traffic on a specific keyword, or if there is an unnatural spike of traffic heading to a particular website or web page.

And, since Google is tracking the behavior of end users on their own website, anomalous behavior is easier to track than it is tracking something across the broader web where signals are more indirect. Google can take advantage of their wide distribution of Chrome & Android where users are regularly logged into Google & pervasively tracked to place more weight on users where they had credit card data, a long account history with regular normal search behavior, heavy Gmail users, etc.

Plus there is a huge gap between the cost of traffic & the ability to monetize it. You might have to pay someone a dime or a quarter to search for something & there is no guarantee it will work on a sustainable basis even if you paid hundreds or thousands of people to do it. Any of those experimental searchers will have no lasting value unless they influence rank, but even if they do influence rankings it might only last temporarily. If you bought a bunch of traffic into something genuine Google searchers didn't like then even if it started to rank better temporarily the rankings would quickly fall back if the real end user searchers disliked the site relative to other sites which already rank.

This is part of the reason why so many SEO blogs mention brand, brand, brand. If people are specifically looking for you in volume & Google can see that thousands or millions of people specifically want to access your site then that can impact how you rank elsewhere.

Even looking at something inside the search results for a while (dwell time) or quickly skipping over it to have a deeper scroll depth can be a ranking signal. Some Google patents mention how they can use mouse pointer location on desktop or scroll data from the viewport on mobile devices as a quality signal.

Neural Matching

Last year Danny Sullivan mentioned how Google rolled out neural matching to better understand the intent behind a search query.

This is a look back at a big change in search but which continues to be important: understanding synonyms. How people search is often different from information that people write solutions about. pic.twitter.com/sBcR4tR4eT— Danny Sullivan (@dannysullivan) September 24, 2018

Last few months, Google has been using neural matching, --AI method to better connect words to concepts. Super synonyms, in a way, and impacting 30% of queries. Don't know what "soapopera effect" is to search for it? We can better figure it out. pic.twitter.com/Qrwp5hKFNz— Danny Sullivan (@dannysullivan) September 24, 2018

The above Tweets capture what the neural matching technology intends to do. Google also stated:

we’ve now reached the point where neural networks can help us take a major leap forward from understanding words to understanding concepts. Neural embeddings, an approach developed in the field of neural networks, allow us to transform words to fuzzier representations of the underlying concepts, and then match the concepts in the query with the concepts in the document. We call this technique neural matching.

To help people understand the difference between neural matching & RankBrain, Google told SEL: "RankBrain helps Google better relate pages to concepts. Neural matching helps Google better relate words to searches."

There are a couple research papers on neural matching.

The first one was titled A Deep Relevance Matching Model for Ad-hoc Retrieval. It mentioned using Word2vec & here are a few quotes from the research paper

• "Successful relevance matching requires proper handling of the exact matching signals, query term importance, and diverse matching requirements."
• "the interaction-focused model, which first builds local level interactions (i.e., local matching signals) between two pieces of text, and then uses deep neural networks to learn hierarchical interaction patterns for matching."
• "according to the diverse matching requirement, relevance matching is not position related since it could happen in any position in a long document."
• "Most NLP tasks concern semantic matching, i.e., identifying the semantic meaning and infer"ring the semantic relations between two pieces of text, while the ad-hoc retrieval task is mainly about relevance matching, i.e., identifying whether a document is relevant to a given query."
• "Since the ad-hoc retrieval task is fundamentally a ranking problem, we employ a pairwise ranking loss such as hinge loss to train our deep relevance matching model."

The paper mentions how semantic matching falls down when compared against relevancy matching because:

• semantic matching relies on similarity matching signals (some words or phrases with the same meaning might be semantically distant), compositional meanings (matching sentences more than meaning) & a global matching requirement (comparing things in their entirety instead of looking at the best matching part of a longer document); whereas,
• relevance matching can put significant weight on exact matching signals (weighting an exact match higher than a near match), adjust weighting on query term importance (one word might or phrase in a search query might have a far higher discrimination value & might deserve far more weight than the next) & leverage diverse matching requirements (allowing relevancy matching to happen in any part of a longer document)

Here are a couple images from the above research paper

And then the second research paper is

Deep Relevancy Ranking Using Enhanced Dcoument-Query Interactions
"interaction-based models are less efficient, since one cannot index a document representation independently of the query. This is less important, though, when relevancy ranking methods rerank the top documents returned by a conventional IR engine, which is the scenario we consider here."

That same sort of re-ranking concept is being better understood across the industry. There are ranking signals that earn some base level ranking, and then results get re-ranked based on other factors like how well a result matches the user intent.

Here are a couple images from the above research paper.

For those who hate the idea of reading research papers or patent applications, Martinibuster also wrote about the technology here. About the only part of his post I would debate is this one:

"Does this mean publishers should use more synonyms? Adding synonyms has always seemed to me to be a variation of keyword spamming. I have always considered it a naive suggestion. The purpose of Google understanding synonyms is simply to understand the context and meaning of a page. Communicating clearly and consistently is, in my opinion, more important than spamming a page with keywords and synonyms."

I think one should always consider user experience over other factors, however a person could still use variations throughout the copy & pick up a bit more traffic without coming across as spammy. Danny Sullivan mentioned the super synonym concept was impacting 30% of search queries, so there are still a lot which may only be available to those who use a specific phrase on their page.

Martinibuster also wrote another blog post tying more research papers & patents to the above. You could probably spend a month reading all the related patents & research papers.

The above sort of language modeling & end user click feedback compliment links-based ranking signals in a way that makes it much harder to luck one's way into any form of success by being a terrible speller or just bombing away at link manipulation without much concern toward any other aspect of the user experience or market you operate in.

Pre-penalized Shortcuts

Google was even issued a patent for predicting site quality based upon the N-grams used on the site & comparing those against the N-grams used on other established site where quality has already been scored via other methods: "The phrase model can be used to predict a site quality score for a new site; in particular, this can be done in the absence of other information. The goal is to predict a score that is comparable to the baseline site quality scores of the previously-scored sites."

Have you considered using a PLR package to generate the shell of your site's content? Good luck with that as some sites trying that shortcut might be pre-penalized from birth.

Navigating the Maze

When I started in SEO one of my friends had a dad who is vastly smarter than I am. He advised me that Google engineers were smarter, had more capital, had more exposure, had more data, etc etc etc ... and thus SEO was ultimately going to be a malinvestment.

Back then he was at least partially wrong because influencing search was so easy.

But in the current market, 16 years later, we are near the infection point where he would finally be right.

At some point the shortcuts stop working & it makes sense to try a different approach.

The flip side of all the above changes is as the algorithms have become more complex they have went from being a headwind to people ignorant about SEO to being a tailwind to those who do not focus excessively on SEO in isolation.

If one is a dominant voice in a particular market, if they break industry news, if they have key exclusives, if they spot & name the industry trends, if their site becomes a must read & is what amounts to a habit ... then they perhaps become viewed as an entity. Entity-related signals help them & those signals that are working against the people who might have lucked into a bit of success become a tailwind rather than a headwind.

If your work defines your industry, then any efforts to model entities, user behavior or the language of your industry are going to boost your work on a relative basis.

This requires sites to publish frequently enough to be a habit, or publish highly differentiated content which is strong enough that it is worth the wait.

Those which publish frequently without being particularly differentiated are almost guaranteed to eventually walk into a penalty of some sort. And each additional person who reads marginal, undifferentiated content (particularly if it has an ad-heavy layout) is one additional visitor that site is closer to eventually getting whacked. Success becomes self regulating. Any short-term success becomes self defeating if one has a highly opportunistic short-term focus.

Those who write content that only they could write are more likely to have sustained success.

A mistake people often make is to look at someone successful, then try to do what they are doing, assuming it will lead to similar success.
This is backward.
Find something you enjoy doing & are curious about.
Get obsessed, & become one of the best at it.
It will monetize itself.— Neil Strauss (@neilstrauss) March 30, 2019

It has been a while since Google has had a major algorithm update.

They recently announced one which began on the 12th of March.

This week, we released a broad core algorithm update, as we do several times per year. Our guidance about such updates remains as we’ve covered before. Please see these tweets for more about that:https://t.co/uPlEdSLHoXhttps://t.co/tmfQkhdjPL— Google SearchLiaison (@searchliaison) March 13, 2019

What changed?

It appears multiple things did.

When Google rolled out the original version of Penguin on April 24, 2012 (primarily focused on link spam) they also rolled out an update to an on-page spam classifier for misdirection.

And, over time, it was quite common for Panda & Penguin updates to be sandwiched together.

If you were Google & had the ability to look under the hood to see why things changed, you would probably want to obfuscate any major update by changing multiple things at once to make reverse engineering the change much harder.

Anyone who operates a single website (& lacks the ability to look under the hood) will have almost no clue about what changed or how to adjust with the algorithms.

In the most recent algorithm update some sites which were penalized in prior "quality" updates have recovered.

Though many of those recoveries are only partial.

Many SEO blogs will publish articles about how they cracked the code on the latest update by publishing charts like the first one without publishing that second chart showing the broader context.

The first penalty any website receives might be the first of a series of penalties.

If Google smokes your site & it does not cause a PR incident & nobody really cares that you are gone, then there is a very good chance things will go from bad to worse to worser to worsterest, technically speaking.

“In this age, in this country, public sentiment is everything. With it, nothing can fail; against it, nothing can succeed. Whoever molds public sentiment goes deeper than he who enacts statutes, or pronounces judicial decisions.” - Abraham Lincoln

Absent effort & investment to evolve FASTER than the broader web, sites which are hit with one penalty will often further accumulate other penalties. It is like compound interest working in reverse - a pile of algorithmic debt which must be dug out of before the bleeding stops.

Further, many recoveries may be nothing more than a fleeting invitation to false hope. To pour more resources into a site that is struggling in an apparent death loop.

The above site which had its first positive algorithmic response in a couple years achieved that in part by heavily de-monetizing. After the algorithm updates already demonetized the website over 90%, what harm was there in removing 90% of what remained to see how it would react? So now it will get more traffic (at least for a while) but then what exactly is the traffic worth to a site that has no revenue engine tied to it?

That is ultimately the hard part. Obtaining a stable stream of traffic while monetizing at a decent yield, without the monetizing efforts leading to the traffic disappearing.

A buddy who owns the above site was working on link cleanup & content improvement on & off for about a half year with no results. Each month was a little worse than the prior month. It was only after I told him to remove the aggressive ads a few months back that he likely had any chance of seeing any sort of traffic recovery. Now he at least has a pulse of traffic & can look into lighter touch means of monetization.

If a site is consistently penalized then the problem might not be an algorithmic false positive, but rather the business model of the site.

The more something looks like eHow the more fickle Google's algorithmic with receive it.

Google does not like websites that sit at the end of the value chain & extract profits without having to bear far greater risk & expense earlier into the cycle.

Thin rewrites, largely speaking, don't add value to the ecosystem. Doorway pages don't either. And something that was propped up by a bunch of keyword-rich low-quality links is (in most cases) probably genuinely lacking in some other aspect.

Generally speaking, Google would like themselves to be the entity at the end of the value chain extracting excess profits from markets.

RIP Quora!!! Q&A On Google - Showing Questions That Need Answers In Search https://t.co/mejXUDwGhT pic.twitter.com/8Cv1iKjDh2— John Shehata (@JShehata) March 18, 2019

This is the purpose of the knowledge graph & featured snippets. To allow the results to answer the most basic queries without third party publishers getting anything. The knowledge graph serve as a floating vertical that eat an increasing share of the value chain & force publishers to move higher up the funnel & publish more differentiated content.

As Google adds features to the search results (flight price trends, a hotel booking service on the day AirBNB announced they acquired HotelTonight, ecommerce product purchase on Google, shoppable image ads just ahead of the Pinterest IPO, etc.) it forces other players in the value chain to consolidate (Expedia owns Orbitz, Travelocity, Hotwire & a bunch of other sites) or add greater value to remain a differentiated & sought after destination (travel review site TripAdvisor was crushed by the shift to mobile & the inability to monetize mobile traffic, so they eventually had to shift away from being exclusively a reviews site to offer event & hotel booking features to remain relevant).

It is never easy changing a successful & profitable business model, but it is even harder to intentionally reduce revenues further or spend aggressively to improve quality AFTER income has fallen 50% or more.

Some people do the opposite & make up for a revenue shortfall by publishing more lower end content at an ever faster rate and/or increasing ad load. Either of which typically makes their user engagement metrics worse while making their site less differentiated & more likely to receive additional bonus penalties to drive traffic even lower.

In some ways I think the ability for a site to survive & remain though a penalty is itself a quality signal for Google.

Some sites which are overly reliant on search & have no external sources of traffic are ultimately sites which tried to behave too similarly to the monopoly that ultimately displaced them. And over time the tech monopolies are growing more powerful as the ecosystem around them burns down:

If you had to choose a date for when the internet died, it would be in the year 2014. Before then, traffic to websites came from many sources, and the web was a lively ecosystem. But beginning in 2014, more than half of all traffic began coming from just two sources: Facebook and Google. Today, over 70 percent of traffic is dominated by those two platforms.

Businesses which have sustainable profit margins & slack (in terms of management time & resources to deploy) can better cope with algorithmic changes & change with the market.

Over the past half decade or so there have been multiple changes that drastically shifted the online publishing landscape:

• the shift to mobile, which both offers publishers lower ad yields while making the central ad networks more ad heavy in a way that reduces traffic to third party sites
• the rise of the knowledge graph & featured snippets which often mean publishers remain uncompensated for their work
• higher ad loads which also lower organic reach (on both search & social channels)
• the rise of programmatic advertising, which further gutted display ad CPMs
  
• the rise of ad blockers
• increasing algorithmic uncertainty & a higher barrier to entry

Each one of the above could take a double digit percent out of a site's revenues, particularly if a site was reliant on display ads. Add them together and a website which was not even algorithmically penalized could still see a 60%+ decline in revenues. Mix in a penalty and that decline can chop a zero or two off the total revenues.

Businesses with lower margins can try to offset declines with increased ad spending, but that only works if you are not in a market with 2 & 20 VC fueled competition:

Startups spend almost 40 cents of every VC dollar on Google, Facebook, and Amazon. We don’t necessarily know which channels they will choose or the particularities of how they will spend money on user acquisition, but we do know more or less what’s going to happen. Advertising spend in tech has become an arms race: fresh tactics go stale in months, and customer acquisition costs keep rising. In a world where only one company thinks this way, or where one business is executing at a level above everyone else - like Facebook in its time - this tactic is extremely effective. However, when everyone is acting this way, the industry collectively becomes an accelerating treadmill. Ad impressions and click-throughs get bid up to outrageous prices by startups flush with venture money, and prospective users demand more and more subsidized products to gain their initial attention. The dynamics we’ve entered is, in many ways, creating a dangerous, high stakes Ponzi scheme.

And sometimes the platform claws back a second or third bite of the apple. Amazon.com charges merchants for fulfillment, warehousing, transaction based fees, etc. And they've pushed hard into launching hundreds of private label brands which pollute the interface & force brands to buy ads even on their own branded keyword terms.

They've recently jumped the shark by adding a bonus feature where even when a brand paid Amazon to send traffic to their listing, Amazon would insert a spam popover offering a cheaper private label branded product:

Amazon.com tested a pop-up feature on its app that in some instances pitched its private-label goods on rivals’ product pages, an experiment that shows the e-commerce giant’s aggressiveness in hawking lower-priced products including its own house brands. The recent experiment, conducted in Amazon’s mobile app, went a step further than the display ads that commonly appear within search results and product pages. This test pushed pop-up windows that took over much of a product page, forcing customers to either click through to the lower-cost Amazon products or dismiss them before continuing to shop. ... When a customer using Amazon’s mobile app searched for “AAA batteries,” for example, the first link was a sponsored listing from Energizer Holdings Inc. After clicking on the listing, a pop-up window appeared, offering less expensive AmazonBasics AAA batteries."

Buying those Amazon ads was quite literally subsidizing a direct competitor pushing you into irrelevance.

And while Amazon is destroying brand equity, AWS is doing investor relations matchmaking for startups. Anything to keep the current bubble going ahead of the Uber IPO that will likely mark the top in the stock market.

Some thoughts on Silicon Valley's endgame. We have long said the biggest risk to the bull market is an Uber IPO. That is now upon us.— Jawad Mian (@jsmian) March 16, 2019

As the market caps of big tech companies climb they need to be more predatious to grow into the valuations & retain employees with stock options at an ever-increasing strike price.

They've created bubbles in their own backyards where each raise requires another. Teachers either drive hours to work or live in houses subsidized by loans from the tech monopolies that get a piece of the upside (provided they can keep their own bubbles inflated).

"It is an uncommon arrangement — employer as landlord — that is starting to catch on elsewhere as school employees say they cannot afford to live comfortably in regions awash in tech dollars. ... Holly Gonzalez, 34, a kindergarten teacher in East San Jose, and her husband, Daniel, a school district I.T. specialist, were able to buy a three-bedroom apartment for $610,000 this summer with help from their parents and from Landed. When they sell the home, they will owe Landed 25 percent of any gain in its value. The company is financed partly by the Chan Zuckerberg Initiative, Mark Zuckerberg’s charitable arm."

The above sort of dynamics have some claiming peak California:

The cycle further benefits from the Alchian-Allen effect: agglomerating industries have higher productivity, which raises the cost of living and prices out other industries, raising concentration over time. ... Since startups raise the variance within whatever industry they’re started in, the natural constituency for them is someone who doesn’t have capital deployed in the industry. If you’re an asset owner, you want low volatility. ... Historically, startups have created a constant supply of volatility for tech companies; the next generation is always cannibalizing the previous one. So chip companies in the 1970s created the PC companies of the 80s, but PC companies sourced cheaper and cheaper chips, commoditizing the product until Intel managed to fight back. Meanwhile, the OS turned PCs into a commodity, then search engines and social media turned the OS into a commodity, and presumably this process will continue indefinitely. ... As long as higher rents raise the cost of starting a pre-revenue company, fewer people will join them, so more people will join established companies, where they’ll earn market salaries and continue to push up rents. And one of the things they’ll do there is optimize ad loads, which places another tax on startups. More dangerously, this is an incremental tax on growth rather than a fixed tax on headcount, so it puts pressure on out-year valuations, not just upfront cash flow.

If you live hundreds of miles away the tech companies may have no impact on your rental or purchase price, but you can't really control the algorithms or the ecosystem.

All you can really control is your mindset & ensuring you have optionality baked into your business model.

• If you are debt-levered you have little to no optionality. Savings give you optionality. Savings allow you to run at a loss for a period of time while also investing in improving your site and perhaps having a few other sites in other markets.
• If you operate a single website that is heavily reliant on a third party for distribution then you have little to no optionality. If you have multiple projects that enables you to shift your attention toward working on whatever is going up and to the right while letting anything that is failing pass time without becoming overly reliant on something you can't change. This is why it often makes sense for a brand merchant to operate their own ecommerce website even if 90% of their sales come from Amazon. It gives you optionality should the tech monopoly become abusive or otherwise harm you (even if the intent was benign rather than outright misanthropic).

As the update ensues Google will collect more data with how users interact with the result set & determine how to weight different signals, along with re-scoring sites that recovered based on the new engagement data.

Recently a Bing engineer named Frédéric Dubut described how they score relevancy signals used in updates

As early as 2005, we used neural networks to power our search engine and you can still find rare pictures of Satya Nadella, VP of Search and Advertising at the time, showcasing our web ranking advances. ... The “training” process of a machine learning model is generally iterative (and all automated). At each step, the model is tweaking the weight of each feature in the direction where it expects to decrease the error the most. After each step, the algorithm remeasures the rating of all the SERPs (based on the known URL/query pair ratings) to evaluate how it’s doing. Rinse and repeat.

That same process is ongoing with Google now & in the coming weeks there'll be the next phase of the current update.

So far it looks like some quality-based re-scoring was done & some sites which were overly reliant on anchor text got clipped. On the back end of the update there'll be another quality-based re-scoring, but the sites that were hit for excessive manipulation of anchor text via link building efforts will likely remain penalized for a good chunk of time.

Update: It appears a major reverberation of this update occurred on April 7th. From early analysis, Google is mixing in showing results for related midtail concepts on a core industry search term & they are also in some cases pushing more aggressively on doing internal site-level searches to rank a more relevant internal page for a query where they homepage might have ranked in the past.

Probably the single best video to watch to understand the power of Google & Facebook (or even most of the major problems across society) is this following video about pleasure versus happiness.

In constantly seeking pleasure we forego happiness.

The "feed" based central aggregation networks are just like slot machines in your pocket: variable reward circuitry which self-optimizes around exploiting your flaws to eat as much attention as possible.

The above is not an accident. It is, rather, as intended:

"That means that we needed to sort of give you a little dopamine hit every once in a while because someone liked or commented on a photo or a post or whatever ... It's a social validation feedback loop ... You're exploiting a vulnerability in human psychology ... [The inventors] understood this, consciously, and we did it anyway."

• Happy? Good! Share posed photos to make your friends feel their lives are worse than your life is.
• Outraged? Good! Click an ad.
• Hopeless? Good. There is a product which can deliver you pleasure...if only you can...click an ad.

Using machine learning to drive rankings is ultimately an exercise in confirmation bias:

For “Should abortion be legal?” Google cited a South African news site saying, “It is not the place of government to legislate against woman’s choices.”

When asked, “Should abortion be illegal?” it promoted an answer from obscure clickbait site listland.com stating, “Abortion is murder.”

Excellent work Google in using your featured snippets to help make the world more absolutist, polarized & toxic.

The central network operators not only attempt to manipulate people at the emotional level, but the layout of the interface also sets default user patterns.

Most users tend to focus their attention on the left side of the page: "if we were to slice a maximized page down the middle, 80% of the fixations fell on the left half of the screen (even more than our previous finding of 69%). The remaining 20% of fixations were on the right half of the screen."

This behavior is even more prevalent on search results pages: "On SERPs, almost all fixations (94%) fell on the left side of the page, and 60% those fixations can be isolated to the leftmost 400px."

On mobile, obviously, the attention is focused on what is above the fold. That which is below the fold sort of doesn't even exist for a large subset of the population.

Outside of a few central monopoly attention merchant players, the ad-based web is dying.

Mashable has raised about $46 million in VC funding over the past 4 years. And they just sold for about $50 million.

Breaking even is about as good as it gets in a web controlled by the Google / Facebook duopoly. :D

Other hopeful unicorn media startups appear to have peaked as well. That BuzzFeed IPO is on hold: "Some BuzzFeed investors have become worried about the company’s performance and rising costs for expansions in areas like news and entertainment. Those frustrations were aired at a board meeting in recent weeks, in which directors took management to task, the people familiar with the situation said."

Google's Chrome web browser will soon have an ad blocker baked into it. Of course the central networks opt out of applying this feature to themselves. Facebook makes serious coin by blocking ad blockers. Google pays Adblock Plus to unblock ads on Google.com & boy are there a lot of ads there.

Format your pages like Google does their search results and they will tell you it is a piss poor user experience & a form of spam - whacking you with a penalty for it.

Of course Google isn't the only search engine doing this. Mix in ads with a double listing and sometimes there will only be 1 website listed above the fold.

I've even seen some Bing search results where organic results have a "Web" label on them - which is conveniently larger than the ad label that is on ads. That is in addition to other tricks like...

• lots of ad extensions that push organics below the fold on anything with the slightest commercial intent
• bolding throughout ads (title, description, URL) with much lighter bolding of organics
• only showing 6 organic results on commercial searches that are likely to generate ad clicks

As bad as either of the above looks in terms of ad load or result diversity on the desktop, it is only worse on mobile.

On mobile devices organic search results can be so hard to find that people ask questions like "Are there any search engines where you don't have to literally scroll to see a result that isn't an advertisement?"

The answer is yes.

DuckDuckGo.

But other than that, it is slim pickings.

In an online ecosystem where virtually every innovation is copied or deemed spam, sustainable publishing only works if your business model is different than the central network operators.

Not only is there the aggressive horizontal ad layer for anything with a hint of commercial intent, but now the scrape layer which was first applied to travel is being spread across other categories like ecommerce.

Ecommerce retailers beware. There is now a GIANT knowledge panel result on mobile that takes up the entire top half of the SERP -> Google updates mobile product knowledge panels to show even more info in one spot: https://t.co/3JMsMHuQmJ pic.twitter.com/5uD8zZiSrK— Glenn Gabe (@glenngabe) November 14, 2017

Here are 2 examples. And alarms are going off at Amazon now. Yes, Prime is killer, but organic search traffic is going to tank. Go ahead & scroll down to the organic listings (if you dare).

And if anyone clicks the module, they are taken away from the SERPs into G-Land. Wow. :) pic.twitter.com/SswOPj4iGd— Glenn Gabe (@glenngabe) November 14, 2017

The more of your content Google can scrape-n-displace in the search results the less reason there is to visit your website & the more ad-heavy Google can make their interface because they shagged the content from your site.

Simply look at the market caps of the big tech monopolies vs companies in adjacent markets. The aggregate trend is expressed in the stock price. And it is further expressed in the inability for the unicorn media companies to go public.

As big as Snapchat & Twitter are, nobody who invested in either IPO is sitting on a winner today.

Google is outraged anyone might question the numbers & if the current set up is reasonable:

Mr Harris described as “factually incorrect” suggestions that Google was “stealing” ad revenue from publishers, saying that two thirds of the revenues generated by online content went to its originators.

“I’ve heard lots of people say that Google and Facebook are “ruthlessly stealing” all the advertising revenue that publishers hoped to acquire through online editions,” he told the gathering.

“There is no advertising on Google News. Zero. Indeed you will rarely see advertising around news cycles in Google Search either.

Sure it is not the ad revenues they are stealing.

Rather it is the content.

Either by scraping, or by ranking proprietary formats (AMP) above other higher quality content which is not published using the proprietary format & then later attaching crappier & crappier deals to the (faux) "open source" proprietary content format.

As Google grabs the content & cuts the content creator off from the audience while attaching conditions, Google's PR hacks will tell you they want you to click through to the source:

Google spokeswoman Susan Cadrecha said the company’s goal isn’t to do the thinking for users but “to help you find relevant information quickly and easily.” She added, “We encourage users to understand the full context by clicking through to the source.”

except they are the ones adding extra duplicative layers which make it harder to do.

Google keeps extracting content from publishers & eating the value chain. Some publishers have tried to offset this by putting more ads on their own site while also getting further distribution by adopting the proprietary AMP format. Those who realized AMP was garbage in terms of monetization viewed it as a way to offer teasers to drive users to their websites.

The partial story approach is getting killed though. Either you give Google everything, or they want nothing.

That is, after all, how monopolies negotiate - ultimatums.

Those who don't give Google their full content will soon receive manual action penalty notifications

Important: Starting 2/1/18, Google is requiring that AMP urls be comparable to the canonical page content. If not, Google will direct users to the non-AMP urls.

And the urls won't be in the Top Stories carousel. Site owners will receive a manual action: https://t.co/ROhbI6TMVz pic.twitter.com/hb9FTluV0S— Glenn Gabe (@glenngabe) November 16, 2017

The value of news content is not zero.

Being the go-to resource for those sorts of "no money here" news topics also enables Google to be the go-to resource for searches for [auto insurance quote] and other highly commercial search terms where Google might make $50 or $100 per click.

Every month Google announces new ad features.

Economics drive everything in publishing. But you have to see how one market position enables another. Google & Facebook are not strong in China, so Toutiao - the top news app in China - is valued at about $20 billion.

Now that Yahoo! has been acquired by Verizon, they've decided to shut down their news app. Unprofitable segments are worth more as a write off than as an ongoing concern. Look for Verizon to further take AIM at shutting down additional parts of AOL & Yahoo.

Firefox recently updated to make its underlying rendering engine faster & more stable. As part of the upgrade they killed off many third party extensions, including ours. We plan to update them soon (a few days perhaps), but those who need the extensions working today may want to install something like (Comodo Ice Dragon (or another browser based on the prior Firefox core) & install our extensions in that web browser.

As another part of the most recent Firefox update, Firefox dumped Yahoo! Search for Google search as their default search engine in a new multiyear deal where financial terms were not disclosed.

Yahoo! certainly deserved to lose that deal.

First, they signed a contract with Mozilla containing a change-of-ownership poison pill where Mozilla would still make $375 million a year from them even if they dump Yahoo!. Given what Yahoo! sold for this amounts to about 10% of the company price for the next couple years.

Second, Yahoo! overpaid for the Firefox distribution deal to where they had to make their user experience even more awful to try to get the numbers to back out.

Here is a navigational search result on Yahoo! where the requested site only appears in the right rail knowledge graph.

The "organic" result set has been removed. There's a Yahoo! News insert, a Yahoo Local insert, an ad inviting you to download Firefox (bet that has since been removed!), other search suggestions, and then graphical ads to try to get you to find office furniture or other irrelevant stuff.

Here is how awful those sorts of search results are: Yahoo! was so embarrassed at the lack of quality of their result set that they put their logo at the upper right edge of the page.

So now they'll be losing a million a day for a few years based on Marissa Mayer's fantastic Firefox deal.

And search is just another vertical they made irrelevant.

When they outsourced many verticals & then finally shut down most of the remaining ones, they only left a few key ones:

On our recent earnings call, Yahoo outlined out a plan to simplify our business and focus our effort on our four most successful content areas  – News, Sports, Finance and Lifestyle. To that end, today we will begin phasing out the following Digital Magazines:  Yahoo Food, Yahoo Health, Yahoo Parenting, Yahoo Makers, Yahoo Travel, Yahoo Autos and Yahoo Real Estate.

And for the key verticals they kept, they have pages like the following, which look like a diet version of eHow

Every day they send users away to other sites with deeper content. And eventually people find one they like (like TheAthletic or Dunc'd On) & then Yahoo! stops being a habit.

Meanwhile many people get their broader general news from Facebook, Google shifted their search app to include news, Apple offers a great news app, the default new tab on Microsoft Edge browser lists a localize news feed. Any of those is a superior user experience to Yahoo!.

It is hard to see what Yahoo!'s role is going forward.

Other than the user email accounts (& whatever legal liabilities are associated with the chronic user account hacking incidents), it is hard to see what Verizon bought in Yahoo!.

Wrastlin With The News

The current presidential cabinet includes a WWE co-founder & this passes for modern political discourse:

#FraudNewsCNN #FNN pic.twitter.com/WYUnHjjUjg— Donald J. Trump (@realDonaldTrump) July 2, 2017

CNN promised vengeance.

Something To Believe In

The pretense of objectivity has been dropped:

These reporters aren't ideologues. They're just right-thinking people who lean left. Somewhere along the line, they stopped pretending to be objective about Trump. ... People don't just disagree with each other. They can't imagine how a decent caring human being could disagree with their own view of race or the minimum wage or immigration or Trump. Being a member of the virtuous tribe means not only carrying the correct card in your wallet to reassure yourself. You have to also believe that the people carrying any other card are irrational, or worse, evil. They're not people to engage in conversation with. They are barriers to be ignored or pushed aside on the virtuous path to paradise. This intolerance and inability to imagine the virtue of the other side is the road to tyranny and chaos. It dehumanizes a good chunk of humanity and that in turn justifies the worst atrocities human beings are capable of.

The WSJ, typically a right-leaning publication, is differentiating their coverage of the president from most other outlets by attempting to be somewhat neutral.

The news is fake. Even historically left-leaning people are disgusted with outlets like CNN.

• "I think the president is probably right to say, like, look you are witch-hunting me. You have no smoking gun, you have no real proof." - CNN supervising producer John Bonifield
• "When you do shitty reporting like CNN, the Washington Post, the New York Times & Rachel Maddow especially. When you do that & it is revealed to be bullshit, what you're doing is building up Trump. There's no greater way to build up Trump than to falsely report on him. There's no better way to build up Trump than to make him the victim." - Jimmy Dore
• "Rachel Maddow was given the facts, she ignored them, & she kept right on going. That's MSNBC, that's CNN, that's the New York Times, the Washington Post - they're all horrible. That's why we had the Iraq war. That's why we have the Syria war. That's why we are still in Afghanistan. That's why we had Libya. That's why we have the biggest income disparity since the gilded age. Meanwhile we are spending more on the military than the rest of the world." - Jimmy Dore

And, since people need something to believe in, there are new American Gods:

"A half hour of cable news delivers enough psychic trauma for a whole year. The newspapers are talking of nothing but treason, espionage, investigations, protests." ... "Stocks are rallying because of how little faith we have in the government. The Mega Blue Chip Corporation is the new Sovereign."

Current Headwinds for Online Publishing

I struggle to keep up with the accelerating rate of change. A number of common themes in the current ecosystem are:

• We are moving toward a world where more things are becoming fake (only accelerated by the demonetization of neutrality & the algorithmic boost associated with reliably delivering confirmation bias in an algorithmic or manual fashion)
• risk keeps being radiated outward to the individual while monopoly platforms capture the spoils (forced-place health insurance purchases where the insurance company arbitrarily drops the sick member on the policy even though that is supposed to be illegal, more temp jobs where people don't get enough hours to get health insurance through their employer, under-funded pensions, outsourcing of core business functions to sweatshops where part-time workers don't get paid for dozens to hundreds of hours of required training & get to watch beheading videos all day)
• the barrier to entry keeps increasing (increased marketing cost due to brand bias, heavy ad loads on dominant platforms, & central platforms making partners do "bet the farm" moves in how they adjust distribution to drive adoption of proprietary formats & temporarily over-promote select content formats)
• the increasing chunk size of competition is making it much harder for individuals to build sustainable businesses. (Yes the tools of the trade are improving quickly, BUT the central platforms are demonetizing the adjacent fields faster than publishing tools & business options improve.)
• In Europe publishers are aggressively leaning on regulators to try to rebalance power.

Some of this stuff is cyclical. About a decade ago the European Commission went after Microsoft for bundling Internet Explorer. Google complained about the monopolistic practices to ensure Microsoft was fined. And we've went from that to a web where Google syndicates native ads that blend into page content while directly funding robot journalism. And then Google is ranking the robot-generated crap too.

Speaking of robot journalists, check out the top 3 results for this query. All 3 are auto-written by automated insights (AI software). Yikes pic.twitter.com/ltFGFXHNiF— Glenn Gabe (@glenngabe) July 8, 2017

But to keep the ecosystem clean & spam free, Google is also planning to leverage their web browser to further dictate the terms of online publishing. Chrome will block autoplay audio & will automatically reroute .dev domains to https. Cutting edge developers suggest using a web browser other than Google Chrome to prevent proprietary lock in across the web.

Is this a test, @Google? pic.twitter.com/V9FZ2hL2cA— TNW (@TheNextWeb) September 5, 2017

While Google distributes their Chrome browser as unwanted bundleware, other web browsers must display uninstall links front & center when trying to gain awareness of their product using Google AdWords. Microsoft Edge is coming to Android, but without a BrowserChoice.eu screen it is unlikely most users will change their web browser as most are unaware of what a web browser even is, let alone the limitations of any of them or the tracking embedded in them.

If you go back several years, there was celebration of the fact that the cost of doing a startup was so low. You didn't have to pay Oracle a million dollars for a server license any more. You didn't even have to rack your own hardware. Now you can just dial it up on Amazon. But there are now these gatekeepers and toll-takers. Back in 2004, you had the wide-open internet. - Jeremy Stoppelman

The Mobile Revolution

If you are an anti-social work at home webmaster who has dual monitors it is easy to dismiss cell phones as inefficient and chalk most mobile usage up to the following.

Russian man visited Chinese click farm.They make fake ratings for mobile apps and things like this.He said they have 10,000 more phones pic.twitter.com/qE96vgCCsi— English Russia (@EnglishRussia1) May 11, 2017

The reality is cell phones are more powerful than they seem if you are strictly consuming rather than working.

Deflationary impact of technology: everything in this Radio Shack flyer from 1991 adds up to $3,285.12 and can be done today on a smartphone pic.twitter.com/ONh3waWVgq— Jeffrey Kleintop (@JeffreyKleintop) June 23, 2017

And that is how the unstoppable quickly becomes the extinct!

10 years ago. pic.twitter.com/ZCWfHfpedi— Harry Tucker (@harrytuckerr) September 11, 2017

Many people the world over are addicted to their cell phones to where viral game makers are self-regulating before regulators step in: "From Tuesday, users below 12 years of age will be limited to one hour of play time each day, while those aged between 12 years and 18 years will be limited to two hours a day, Tencent said."

While China is using their various tools to clamp down on Honour of Kings, Tencent is bringing the game to the west, which makes blocking VPN services (with Apple's help - they must play along or have the phones reduced to bricks) & requiring local data storage & technology transfer more important. Anything stored locally can be easily disappeared: "China's already formidable internet censors have demonstrated a new strength-the ability to delete images in one-on-one chats as they are being transmitted, making them disappear before receivers see them."

China has banned live streaming, threatened their largest domestic social networks, shut down chat bots, require extensive multimedia review: "an industry association circulated new regulations that at least two "auditors" will, with immediate effect, be required to check all audiovisual content posted online" AND they force users to install spyware on their devices.

In spite of all those restrictions, last year "Chinese consumers spent $5.5 trillion through mobile payment platforms, about 50 times more than their American counterparts." In the last quarter Baidu had ¥20.87 billion in revenues, with 72% of their revenues driven by mobile.

People can not miss that which they've never seen, thus platform socialism works. Those who doubt it will be tracked & scored accordingly.

History, as well, can be scrubbed. And insurance companies watch everything in real-time - careful what you post. The watchful eye of the Chinese pre-crime team is also looking over every move.

Last quarter Facebook had revenues of $9.164 billion, with 87% coming from mobile devices.

pic.twitter.com/JlPBSlmKlw— banksy (@thereaIbanksy) September 16, 2017

Simulacrum has ALMOST been perfected:

"We didn't have a choice to know any life without iPads or iPhones. I think we like our phones more than we like actual people." ... "Rates of teen depression and suicide have skyrocketed since 2011. It's not an exaggeration to describe iGen as being on the brink of the worst mental-health crisis in decades. Much of this deterioration can be traced to their phones." ... "Teens who spend more time than average on screen activities are more likely to be unhappy, and those who spend more time than average on nonscreen activities are more likely to be happy."

pic.twitter.com/QBLBXIDDLK— banksy (@thereaIbanksy) August 24, 2017

The web is becoming easier to get addicted to due to personalization algorithms that reinforce our worldviews even as they make us feel more isolated and left out. And the barrier to entry for consumers into one of the few central gatekeeper ecosystems is dropping like a rock due to the falling cost of mobile devices, coupled with with images & video displacing text making literacy optional. As we become more "connected" we feel more isolated:

"Social isolation, loneliness or living alone was each a significant factor contributing to premature death. And each one of these factors was a more significant risk factor for dying than obesity. ... No one knows precisely why loneliness is surging, threatening the lives of many millions of people, but it does seem that the burgeoning use of technology may have something to do with it. Personally, I would contend that technology may be the chief factor fueling it."

The primary role of the big data mining companies is leveraging surveillance for social engineering

Unsettling that according to Mark Zuckerberg purpose of Facebook is forced social engineering. From "World Without Mind" by Franklin Foer: pic.twitter.com/CHRnefg9m2— Murtaza Hussain (@MazMHussain) October 8, 2017

App Annie expects the global app economy to be worth $6.3 trillion by 2021.

The reason those numbers can easily sound fake & mobile can seem overblown is how highly concentrated usage has become: "over 80 percent of consumer time on mobile devices is now spent on the apps, websites and properties" of just five companies: Facebook, Google, Apple, Yelp and Bing.

Maslow 2.0 pic.twitter.com/X1OguQG8Gq— ian bremmer (@ianbremmer) October 7, 2017

eMarketer stated Google will have more mobile ad revenue than desktop ad revenue in the US this year. They also predicted Google & Facebook will consume over 2/3 of US online ad spend within 2 years.

The central network operators not only maintain an outsized share of revenues, but also maintain an outsized share of profits. When the home team gets a 30% rake of any sale it is hard for anyone else to compete. Even after buying and gutting Motorola Google bought part of HTC for $1.1 billion. The game plan has never changed: commoditize the compliment to ensure user data & most of the profits flow to Google. Put up arbitrary roadblocks for competing services while launching free parallel offerings to drive lock-in.

For the last YEAR I've been battling App Store rejections - we made an app called Animoji with animated emojis...now I know why. https://t.co/jKJXfLMGj2— Ryan Jones (@rjonesy) September 9, 2017

Central data aggregators can keep collecting more user data & offer more granular ad distribution features. They can tell you that this micro moment RIGHT NOW is make or break:

it's intended to create a bizarre sense of panic among marketers - "OMG, we have to be present at every possible instant someone might be looking at their phone!" - which doesn't help them think strategically or make the best use of their marketing or ad spend.

The reality is that if you don't have a relationship with a person on their desktop computer they probably don't want your mobile app either.

If you have the relationship then mobile only increases profits.

Is iOS 11 specifically designed to make your older iPhones unusable and drain your battery so you have to upgrade to the newest phones?— Eric Jackson (@ericjackson) October 7, 2017

Many people attempting to build "the next mobile" will go bust, but wherever the attention flows the ads will follow.

Those with a broad & dominant tech platform can copy features from single-category devices and keep integrating them into their core products to increase user lock-in. And they can build accessories for those core devices while prohibiting the flow of data to third party devices to keep users locked into their ecosystem.

Smaller Screens, Shallower Attention

People often multi-task while using mobile devices.

When multi-tasking it is easier to accidentally click an ad. This happens 10s of billions of times a year:

This year, in-app mobile ad spend will reach $45.3 billion, up $11 billion from last year, according to eMarketer. And apps are where the money is at for mobile advertising, comprising 80 percent of all U.S. media dollars spent on mobile.

But multi-tasking means doing almost everything else worse. The "always on" mode not only increases isolation, but also lowers our ability to focus:

"while our phones offer convenience and diversion, they also breed anxiety. Their extraordinary usefulness gives them an unprecedented hold on our attention and vast influence over our thinking and behavior. ... Not only do our phones shape our thoughts in deep and complicated ways, but the effects persist even when we aren't using the devices. As the brain grows dependent on the technology, the research suggests, the intellect weakens. ... when people hear their phone ring but are unable to answer it, their blood pressure spikes, their pulse quickens, and their problem-solving skills decline. ... As the phone's proximity increased, brainpower decreased. ... Anticipating that information would be readily available in digital form seemed to reduce the mental effort that people made to remember it. ... people are all too quick to credit lies and half-truths spread through social media by Russian agents and other bad actors. If your phone has sapped your powers of discernment, you'll believe anything it tells you."

Further, the shallow attention stream makes it easy to displace content with ads:

4 Ads
3 map carrousel results
5 organic results
4 Ads

Then "see more results"

4 more Ads
5 organic results
4 more Ads

On desktop devices people don't accidentally misclick on ads at anywhere near the rate they fat thumb ads on mobile devices.

Desktop ad clicks convert to purchases. Mobile ad clicks convert to ad budget burned: "marketers are still seeing few shoppers purchasing on mobile. The 52% of share in traffic only has 26% share of revenue."

For traditional publishers mobile users drastically under-monetize desktop users due to

• drastically lower conversion rates (true for almost everyone in ecommerce outside of Amazon perhaps)
• limited cross-device tracking (how do you track people who don't even hit your site but hit a cached page hosted via Google AMP or Facebook Instant Articles?)
• lower ad load allowed on publisher sites due to limited screen size
• aggressive filtering of fat thumb ad clicks on partner sites from central ad networks

For the central network operators almost all the above are precisely the exact opposite.

• higher ad CTR by making entire interface ads (& perhaps even disappearing the concept of non-ads in the result set)
• great cross-device user tracking
• higher ad load allowed by the small screen size pushing content below the fold
• more lenient filtering of fat thumb accidental ad clicks

If you look at raw stats without understanding the underlying impact, it is easy to believe the ecosystem is healthy.

Assumption: Google's ads are more prominent, so organic must be dying.
Reality: As of Oct. 2016, 20X more organic clicks than paid ones. pic.twitter.com/FaEBpBZWSw— Rand Fishkin (@randfish) July 1, 2017

However the huge number of "no click" results are demonetizing easy publisher revenues, which have traditionally helped to fund more in-depth investigative reporting. Further, much of the direct navigation which happened in the past is now being passed through brand-related search result pages. You can argue that is an increase in search traffic, or you can argue it is shifting the roll of the address bar from navigation to search.

The first page is nothing but ads

Yep, and here they are in Philly. Home service ads, then AdWords traditional ads, then the local pack (way down below). :) pic.twitter.com/VOVZPWWHsg— Glenn Gabe (@glenngabe) July 17, 2017

On mobile so is the second, and most of the third

Hey, only 2.5 screens before you get to the 10 blue links. A full 4 screens if you cannot crack the top 2 organics. pic.twitter.com/bbm1pz8hyF— Jeremy Bochenek (@J_Bochenek) July 17, 2017

If a search query has lots of easy to structure crap around it, a user might need 6 or 7 scrolls to get to an organic result

Very interesting Google SERP for GoT. One barely visible organic result after >70% scroll depth. Oh my... #seo #GameOfThones pic.twitter.com/Z6j7VvJMI4— Bastian Grimm (@basgr) August 24, 2017

Then if third parties go "well Google does this, so I should too" they are considered a low quality user experience and get a penalty.

Emailed a client one month ago when I picked up ultra-aggressive ads (especially on mobile). They just received an ad experiences warning. pic.twitter.com/QLLZci1xKW— Glenn Gabe (@glenngabe) September 25, 2017

31% ad coverage on mobile website is excessive / spam / low quality user experience for a publisher, while 301% coverage is reasonable for the central network operators.

Google not only displaces the result set, but also leverages their search suggestion features & algorithmic influence to alter how people search & what they search for.

Ads are getting integrated into mobile keyboards.

The standard keyboard on the HTC 10 has begun showing ads [X-Post from r/mildlyinfuriating] https://t.co/FuXDJzilZ6 #blog pic.twitter.com/VriK54dBHb— Android Facts (@manatweets) July 16, 2017

And when a user finally reaches the publisher's website (provided they scroll past the ads, the AMP listings, and all the other scrape-n-displace trash) then when they finally land on a publication Google will overlay other recommended articles from other sites.

Whoa -> While you're reading a page on the Google app for iOS, you'll now see suggestions for related content https://t.co/n6FjkNqx82 pic.twitter.com/DZYTt8T7fI— Glenn Gabe (@glenngabe) September 19, 2017

That feature will eventually end up including ads in it, where publishers will get 0.00% of the revenue generated.

Remember how Google suggested publishers should make their websites faster, remove ads, remove clutter, etc. What was the point of all that? To create free real estate for Google to insert a spam unit into your website?

Continuing coverage of Google's new content recos. I'm sure Best Buy is thrilled to see Amazon show up while someone is on their page. Ouch. pic.twitter.com/qpDyGKPyYh— Glenn Gabe (@glenngabe) September 24, 2017

This wouldn't be bad if mobile were a small, fringe slice of traffic, but it is becoming the majority of traffic. And as mobile increases desktop traffic is shrinking.

Even politically biased outlets that appear to be nearly perfectly optimized for a filter bubble that promotes identity politics struggle to make the numbers work: "As a result of continued decline in direct advertising, [Salon's] total revenue in the fiscal year 2017 decreased by 34% to $4.6 million. Following the market trend, 84% of our advertising revenue in fiscal year 2017 was generated by programmatic selling. ... [Monthly unique visitors to our website saw] a decrease of 23%. We attribute the decline primarily to the changes in the algorithms used by Facebook."

I knew the last year was bad for online publishing, but the Salon 10K shows *just how bad* pic.twitter.com/oyH7pdCDNI— josh laurito (@joshlaurito) June 26, 2017

The above sorts of numbers are the logical outcome to this:

we've heard complaints from users that if they click on a result and it's difficult to find the actual content, they aren't happy with the experience. Rather than scrolling down the page past a slew of ads, users want to see content right away. So sites that don't have much content"above-the-fold" can be affected by this change. If you click on a website and the part of the website you see first either doesn't have a lot of visible content above-the-fold or dedicates a large fraction of the site's initial screen real estate to ads, that's not a very good user experience. Such sites may not rank as highly going forward.

Especially when combined with this:

As you scroll through it, you are then given travel ads for flight options through Google Flight search, hotels through Google Hotel search and restaurants through Google Local results. Then towards the bottom of the knowledge graph card, all the way at the end in a small grayish font, you have a link to "see web results."

Bad news for TripAdvisor.

Google has squeezed out SEO for travel. PCLN/EXPE SEM $ spend and higher conversion is a massive competitive advantage. Bad news for TRIP. pic.twitter.com/39QkxuN780— modest proposal (@modestproposal1) April 17, 2017

And amongst the good news for Expedia, there's also a bit of bad news for Expedia. The hotels are fighting Airbnb & OTAs. In travel Google is twice as big as the biggest OTA players. They keep eating more SERP real estate and adding more content behind tabs. On mobile they'll even disappear the concept of organic results.

Room previews in the search results not only means that second tier players are worth a song, but even the new growth players propped up by aggressive ad buying eventually hit a wall and see their stock crash.

As the entire ecosystem gets squeezed by middlemen and the market gets obfuscated with an incomplete selection it is ultimately consumers who lose: "Reservations made through Internet discount sites are almost always slated for our worst rooms."

The New York Times pitched Yelp as a pesky player holding a grudge:

"For six years, his company has been locked in a campaign on three continents to get antitrust regulators to punish Google, Yelp's larger, richer and more politically connected competitor. ... Yelp concluded that there was no better way to get Google's attention than to raise the specter of regulation. ... something [Mark Mahaney] calls the Death of Free Google. As the internet has migrated to mobile phones, Google has compensated for the smaller screen space by filling it with so many ads that users can have a hard time finding a result that hasn't been paid for."

In spite of how quick The New York Times was to dismiss Yelp, the monopoly platforms are stiffing competition & creativity while bundling fake reviews & junk features into their core platforms.

People can literally switch their name to "Loop dee Loop"

and leave you terrible, fake reviews.

Google's lack of effort & investment to clean up trash in their local services department highlights that they don't feel they need to compete on quality. Pay for core search distribution, throw an inferior service front & center, and win by default placement.

As AI advancements make fake reviews look more legit Google's lack of investment in creating a quality ecosystem will increasingly harm both consumers and businesses. Many low margin businesses will go under simply because their Google reviews are full of inaccurate trash or a competitor decided to hijack their business listing or list their business as closed.

To this day Google is still egregiously stealing content from Yelp:

Yelp said it investigated and found that over one hour, Google pulled images from Yelp's servers nearly 386,000 times for business listings in Google Maps, which Google exempted from its promise to not scrape content. Yelp then searched Google for 150 of the businesses from those map listings and found that for 110 of them, Google used a Yelp photo as the lead image in the businesses' listings in search results.

Stealing content & wrapping it in fake reviews is NOT putting the user first.

Facebook has their own matching parallel shifts.

The aggregate quality of mobile ad clicks is quite low. So as mobile becomes a much higher percent of total ad clicks, those who don't have scale and narrative control are reduced to taking whatever they can get. And mainstream media outlets are reduced to writing puff pieces so the brands they cover will pay to promote the stories on the main channels.

As programmatic advertising, ad blockers, unpatched Android-powered botnets & malware spread each day gets a little uglier for everyone but the central market operators. It is so bad that some of the central market operators offer surveillance apps which claim to protect user privacy! Other app makers not connected to monopoly profit streams monetize any way they can.

The narrative of growth can be sold (we are launching a new food channel, we are investing in our internal video team, we have exclusive real estate listings, and, um, we acquired a food channel) but the competition is a zero sum game with Google & Facebook eating off the publisher's plates.

That's why Time is trying to shave $400 million off their expenses & wants to sell their magazine division. Newspaper companies are selling for $1. It is also why Business Insider is no longer chasing growth & the New York Times is setting up a charitable trust.

The rise of ad blocking only accelerates the underlying desperation.

I have some thoughts about why news orgs are finding that people won't read long articles: pic.twitter.com/G8Zh6GTA6w— Ben Chase (@bbchase) July 4, 2017

I feel terrible for journalists who invest time and effort into doing a hard job well only to have it presented like this. pic.twitter.com/jIZxuJqVAq— Jeff Long (@banterability) October 5, 2017

As long as news websites make their own customer experience worse than what can be found as a cached copy on the monopoly platforms there is no reason to visit the end publisher website. That is why the proprietary formats promoted by the monopoly platforms are so dangerous. They force lighter monetization & offset the lack of revenue by given preferential placement:

click through rate from Google search went from 5.9% (Regular) to 10.3% (AMP), and average search position went from 5.9 (Regular) to 1.7 (AMP). Since then, we have deployed AMP across fifteen of our brands and we have been very pleased with the results. Today, AMP accounts for 79% of our mobile search traffic and 36% of our total mobile visits.

As long as almost nobody is using the new proprietary, ghetto lock-in format the math may work out there, but once many people adopt it then it becomes another recurring sunk cost with no actual benefit:

the only voices promoting AMP's performance benefits are coming from inside Google. ... given how AMP pages are privileged in Google's search results, the net effect of the team's hard, earnest work comes across as a corporate-backed attempt to rewrite HTML in Google's image.

Even if you get a slight uptick in traffic from AMP, it will lead to lower quality user engagement as users are browsing across websites rather than within websites. Getting a bit more traffic but 59% fewer leads is a fail.

No amount of collaborative publisher partnerships, begging for anti-trust exemptions, or whining about Google is going to fix the problem.

"The only way publishers can address this inexorable threat is by banding together. If they open a unified front to negotiate with Google and Facebook-pushing for stronger intellectual-property protections, better support for subscription models and a fair share of revenue and data-they could build a more sustainable future for the news business. But antitrust laws make such coordination perilous. These laws, intended to prevent monopolies, are having the unintended effect of preserving and protecting Google and Facebook's dominant position."

Wait a minute.

Wasn't it the New York Times which claimed Yelp was holding an arbitrary grudge against Google?

The following sounds a lot more desperate:

newspapers that once delivered their journalism with their own trucks increasingly have to rely on these big online platforms to get their articles in front of people, fighting for attention alongside fake news, websites that lift their content, and cat videos.

Well maybe that is just smaller publications & not the gray lady herself

"the temperature is rising in terms of concern, and in some cases anger, about what seems like a very asymmetric, disadvantageous relationship between the publishers and the very big digital platforms." - NYT CEO Mark Thompson

In unrelated news, there's another round of layoffs coming at the New York Times.

And the New York Times is also setting up a nonprofit division to expand journalism while their core company focuses on something else.

Apparently Yelp does not qualify as a publisher in this instance.

Or does it?

The Times is backing the move for what is called an anticompetitive safe haven, in part, Mr. Thompson said, "because we care about the whole of journalism as well as about The New York Times."

Ah, whole of journalism, which, apparently, no longer includes local business coverage.

You know the slogan: "news isn't news, unless it isn't local."

The struggles are all across the media landscape. The new Boston Globe CEO lasted a half-year. The San Diego Union-Tribune resorted to using GoFundMe. The Chicago Sun-Times sold for $1. Moody's issued a negative outlook for the US newspaper sector. As the industry declines the biggest players view consolidation as the only solution.

These struggles existed even before the largest brand advertisers like P&G cut back on low & no value ad venues like YouTube:

In the fourth quarter, the reduction in marketing that occurred was almost all in the digital space. And what it reflected was a choice to cut spending from a digital standpoint where it was ineffective: where either we were serving bots as opposed to human beings, or where the placement of ads was not facilitating the equity of our brands.

Google & Facebook are extending their grip on the industry with Google launching topical feeds & Facebook wanting to control subscription management.

Best of luck to journalists on the employment front:

The initiative, dubbed Reporters and Data and Robots (RADAR), will see a team of five journalists work with Natural Language Generation software to produce over 30,000 pieces of content for local media each month.

Hopefully editors catch the subtle errors the bots make, because most of them will not be this obvious & stupid.

The Guardian does not seem to know what a 40 is https://t.co/m7Gm1YrbXC pic.twitter.com/Y0sK9r0ltJ— Shuja Haider (@shujaxhaider) July 26, 2017

The cost of parasitic content recycling is coming down quickly:

In a show of strength last year, Microsoft used thousands of these chips at once to translate all of English Wikipedia into Spanish-3 billion words across five million articles-in less than a tenth of a second. Next Microsoft will let its cloud customers use these chips to speed up their own AI tasks.

Voice search makes it even easier to extract the rewards without paying publishers. Throwing pennies at journalists does nothing to change this.

And that voice shift is happening fast: "By 2020 half of search will be via voice"

If Google is subsidizing robotic journalism they are thus legitimizing robotic journalism. As big publishers employ the tactic, Google ranks it.

Checking some Heliograf articles (AI-written) reveals once again they do rank well. Google is in a tough position here. It's inevitable... pic.twitter.com/g0Etcx3rFj— Glenn Gabe (@glenngabe) September 16, 2017

It is almost impossible to compete economically with an entity that rewrites your work & has zero marginal cost of production.

YouTube has perhaps the worst comments on the web. Some mainstream news sites got rid of comments because they couldn't justify the cost of policing them. That in turn shifts the audience & attention stream to sites like Facebook & Twitter. Some news sites which are still leaving comments enabled rely on a Google filter, a technology Google can use on YouTube as they see fit.

Any plugins publishers use to lower their costs can later disappear. It looked like FindTheBest was doing well financially, but when it was acquired many news sites quickly found out the cost of free as they now have thousands of broken articles in their archives: "Last month, Graphiq announced that features for news publishers would no longer be available after Friday."

Driving costs toward zero by piling on external dependencies is no way to build a sustainable business. Especially when the central network operators are eating the playing field:

"Between fast-loading AMP articles from major news brands hosted in its domain, full pages of information scraped from outside sites that don't require you to visit them, basic shopping functions built into ads, YouTube, and a host of other features, the Google-verse is more of a digital walled garden than ever. ... If Google continues to choke these sites out, what incentive will there be for new ones to come along?"

Unprofitable partners which were buying growth with artificially cheap pricing eventually find out investors want profits more than growth & either reprice or go away. The longer you use something & the more locked in you are to it the more aggressively it can afford to reprice. Symbiotic relationships devolve into abusive ones:

• "for every pound an advertiser spends programmatically on the Guardian only 30 pence actually goes to the publisher." - Mediatel
• "Google wants to cut out the middlemen, which it turns out, are URLs." - MobileMoxie
• "[AMP is] a way for Google to obfuscate your website, usurp your content & remove any personal credibility from the web" - TheRegister
• "Though the stated initiative of ads.txt is to stop inventory resale, it achieves this by establishing 'preferred' channels, which naturally favors the industry's most influential companies" - Ad Exchanger

That Apple does extra work to undo AMP says a lot.

Those who think the central network operators are naive to the power structure being promoted by the faux solutions are either chasing short-term goals or are incredibly masochistic.

Arbitraging brand is the core business model of the attention merchant monopoly.

we've found out that 98% of our business was coming from 22 words. So, wait, we're buying 3,200 words and 98% of the business is coming from 22 words. What are the 22 words? And they said, well, it's the word Restoration Hardware and the 21 ways to spell it wrong, okay?

Publishers buying the "speed" narrative are failing themselves. The Guardian has 11 people working on AMP integration. And what is Google doing about speed? Google shut down Google Instant search results, often displays a screen or two full of ads which mobile users have to scroll past to find the organic search results AND is testing auto-playing videos in the search results.

Facebook is also promoting fast loading & mobile-friendly pages.

To keep bleeding clicks out of the "organic" ecosystem they don't even need to have explicit malicious intent. They can run a thousand different tests every month (new vertical sitelink formats, swipable sitelinks, showing 8 sitelinks on tiny mobile devices, MOAR sitelinks, message extensions, extensions on call-only ads, price discount labels, frame 3rd party content inline, dramatically ramp up featured snippets +QnA listings, more related searches, more features in ad units, larger ad units, ad units that replace websites & charge advertisers for sending clicks from Google to Google, launch a meta-search service where they over-promote select listings, test dropping URLs from listings, put ads in the local pack, change color of source links or other elements, pop ups of search results inside search results, etc.) & keep moving toward whatever layout drives more ad clicks, keeps users on Google longer & forces businesses to buy ads for exposure, claiming they are optimizing the user experience the whole time.

They can hard-code any data type or feature, price it at free to de-fund adjacent businesses, consolidate market power, then increase rents after they have a monopoly position in the adjacent market.

And they can fund research on how to remove watermarks from images.

Why not make hosting free, get people to publish into a proprietary format & try to shift news reading onto the Google app. With enough attention & market coverage they can further extort publishers into accepting perpetually worse deals. And free analytics & business plugins which are widely adopted can have key features get pushed into the paid version. Just look at Google Analytics - its free or $150,000+/yr.

The above sorts of moves can be done in isolation, or in a combinatorial approach. Publishers aloof of the ecosystem shifts may use microformats to structure their content. They'll then find it is integrated in Google's new image search layout, where Google copies the content wholesale &
shows it near other third party images framed by Google.

How about some visually striking, yet irrelevant listings for competing brands on branded searches to force the brand ad buy. And, of course rounded card corners to eat a few more pixels, along with faint ad labeling on ads coupled with vibrant colored dots on the organic results to confuse end users into thinking the organic results are the ads.

While Google turns their search results into an adspam farm, they invite you to test showing fewer ads on your site to improve user experience. Google knows best - let them automate your ad load & ad placement.

What is the real risk of AI? Bias.

"It's important that we be transparent about the training data that we are using, and are looking for hidden biases in it, otherwise we are building biased systems," Giannandrea added. "If someone is trying to sell you a black box system for medical decision support, and you don't know how it works or what data was used to train it, then I wouldn't trust it."

And how does Google justify their AI investments? Through driving incremental ad clicks: "The DeepMind founders understand that their power within [Alphabet], and their ability to get their way with [Alphabet CEO] Larry Page, depends on how many eyeballs and clicks and ad dollars they can claim to be driving"

No bias at all there!

Truth: Google killed publishing in 2015. What you're reading now is detritus + new junk posted by crazies walking around empty offices— 11 (@searchsleuth998) August 1, 2017

And if publishing was killed in 2015, things have only got worse since then:

Looking at 2015 vs 2017 data for all keywords ranking organically on the first page, we’ve seen a dramatic change in CTR. Below we’ve normalized our actual CTR on a 1–10 scale, representing a total drop of 25% of click share on desktop and 55% on mobile.

SEOs who were overly reliant on the search channel were the first to notice all the above sorts of change, as it is their job to be hyper-aware of ecosystem shifts. But publishers far removed from SEO who never focused on SEO are now writing about the trends SEOs were writing about nearly a decade ago. Josh Marshall recently covered Google's awesome monopoly powers.

few publishers really want to talk about the depths or mechanics of Google's role in news publishing. Some of this is secrecy about proprietary information; most of it is that Google could destroy or profoundly damage most publications if it wanted to. So why rock the boat? ... Google's monopoly control is almost comically great. It's a monopoly at every conceivable turn and consistently uses that market power to deepen its hold and increase its profits. Just the interplay between DoubleClick and Adexchange is textbook anti-competitive practices. ... Is your favorite website laying off staff or 'pivoting to video'. In most cases, the root cause is not entirely but to a significant degree driven by the platform monopolies

His article details how Google owns many points of the supply chain

So let's go down the list: 1) The system for running ads, 2) the top purchaser of ads, 3) the most pervasive audience data service, 4) all search, 5) our email. ... But wait, there's more! Google also owns Chrome, the most used browser for visiting TPM.

He also covers the price dumping technique that is used to maintain control

In many cases, alternatives don't exist because no business can get a footing with a product Google lets people use for free.

And he shared an example of Google algorithms gone astray crippling his business, even though it was not related to search & unintentional:

Because we were forwarding to ourselves spam that other people sent to us, Google decided that the owner of the TPM url was a major spammer and blocked emails from TPM from being sent to anyone.

If the above comes across as depressing, don't worry. The search results now contain a depression diagnostic testing tool.

Last August I wrote a blog post about how attention merchants were sucking the value out of online publishing. In it I noted how the Yahoo! Directory disappeared & how even DMOZ saw a sharp drop in traffic & rankings over the past few years.

The concept of a neutral web is dead. In its place is agenda-driven media.

• Politically charged misinformed snippets.
• Ads cloaked as content.
• Public relations propaganda.
• Mostly correct (but politically insensitive) articles being "fact checked" where a minor detail is disputed to label the entire piece as not credible.

As the tech oligarchs broadly defund publishing, the publishers still need to eat. Aggregate information quality declines to make the numbers work. Companies which see their ad revenues slide 20%, 30% or 40% year after year can't justify maintaining the labor-intensive yet unmonetized side projects.

There is Wikipedia, but it is not without bias & beyond the value expressed in the hidden bias most of the remaining value from it flows on through to the attention merchant / audience aggregation / content scraper platforms.

Last month DMOZ announced they were closing on March 14th without much fanfare. And on March 17th the directory went offline.

A number of people have pushed to preserve & archive the DMOZ data. Some existing DMOZ editors are planning on launching a new directory under a different name but as of the 17th DMOZ editors put up a copy at dmoztools.net. Jim Boykin scraped DMOZ & uploaded a copy here. A couple other versions of DMOZ have been published at OpenDirectoryProject.org & Freemoz.org.

DMOZ was not without criticism or controversy,

Although site policies suggest that an individual site should be submitted to only one category, as of October 2007, Topix.com, a news aggregation site operated by DMOZ founder Rich Skrenta, has more than 17,000 listings.

Early in the history of DMOZ, its staff gave representatives of selected companies, such as Rolling Stone or CNN, editing access in order to list individual pages from their websites. Links to individual CNN articles were added until 2004, but were entirely removed from the directory in January 2008 due to the content being outdated and not considered worth the effort to maintain.

but by-and-large it added value to the structure of the web.

As search has advanced (algorithmic evolution, economic power, influence over publishers, enhanced bundling of distribution & user tracking) general web directories haven't been able to keep pace. Ultimately the web is a web of links & pages rather than a web of sites. Many great sites span multiple categories. Every large quality site has some misinformation on it. Every well-known interactive site has some great user contributions & user generated spam on it. Search engines have better signals about what pages are important & which pages have maintained importance over time. As search engines have improved link filtering algorithms & better incorporated user tracking in rankings, broad-based manual web directories had no chance.

The web of pages vs web of sites concept can be easily observed in how some of the early successful content platforms have broken down their broad-based content portals into a variety of niche sites.

When links were (roughly) all that mattered, leveraging a website's link authority meant it was far more profitable for a large entity to keep publishing more content on the one main site. That is how eHow became the core of a multi-billion Dollar company.

Demand Media showed other publishers the way. And if the other existing sites were to stay competitive, they also had to water down content quality to make the numbers back out. The problem with this was the glut of content was lower ad rates. And the decline in ad rates was coupled with a shift away from a links-only view of search relevancy to a model based on weighting link profiles against user engagement metrics.

Websites with lots of links, lots of thin content & terrible engagement metrics were hit.

Kristen Moore, vp of marketing for Demand Media, explained what drove the most egregious aspects of eHow's editorial strategy: “There’s some not very bright people out there.”

eHow improved their site design, drastically reduced their ad density, removed millions of articles from their site, and waited. However nothing they did on that domain name was ever going to work. They dug too deep of a hole selling the growth story to pump a multi-billion Dollar valuation. And they generated so much animosity from journalists who felt overwork & underpaid that even when they did rank journalists would typically prefer to link to anything but them.

The flip side of that story is the newspaper chains, which rushed to partner with Demand Media to build eHow-inspired sections on their sites.

• traveltips.usatoday.com
• homeguides.sfgate.com
• smallbusiness.chron.com
• the Arizona Republic
• Even the bastion of left-wing thinking Salon couldn't ignore the easy money opportunity.

Brands which enjoy the Google brand subsidy are also quite hip to work with Demand Media, which breathes new life into once retired content: "Sometimes Demand will even dust off old content that’s been published but is no longer live and repurpose it for a brand."

As Facebook & Google grew more dominant in the online ad ecosystem they aggressively moved to suck in publisher content & shift advertiser spend onto their core properties. The rise of time spent on social sites only made it harder for websites to be sought out destination. Google also effectively cut off direct distribution by consolidating & de-monetizing the RSS reader space then shutting down a project they easily could have left run.

As the web got more competitive, bloggers & niche publications which were deeply specialized were able to steal marketshare in key verticals by leveraging a differentiated editorial opinion.

Even if they couldn't necessarily afford to build strong brands via advertising, they were worthy of a follow on some social media channels & perhaps an email subscription. And the best niche editorial remains worthy of a direct visit:

Everything about Techmeme and its lingering success seems to defy the contemporary wisdom of building a popular website. It publishes zero original reporting and is not a social network. It doesn’t have a mobile app or a newsletter or even much of a social presence beyond its Twitter account, which posts dry commodity news with zero flair for clickability.

As a work around to the Panda hits, sites like eHow are now becoming collections of niche-focused sites (Cuteness.com, Techwalla.com, Sapling.com, Leaf.tv, etc will join Livestrong.com & eHow.com). It appears to be working so far...

...but they may only be 1 Panda update away from finding out the new model isn't sustainable either.

About.com has done the same thing (TheSpruce.com, Verywell.com, Lifewire.com, TheBalance.com). Hundreds of millions of Dollars are riding on the hope that as the algorithms keep getting more granular they won't discover moving the content to niche brands wasn't enough.

As content moves around search engines with billions of Dollars in revenue can recalibrate rankings for each page & adjust rankings based on user experience. Did an influential "how to" guide become irrelevant after a software or hardware update? If so, they can see it didn't solve the user's problem and rank a more recent document which reflects the current software or hardware. Is a problem easy to solve with a short snippet of content? If so, that can get scraped into the search results.

Web directories which are built around sites rather than pages have no chance of competing against the billions of Dollars of monthly search ads & the full cycle user tracking search companies like Google & Bing can do with their integrated search engines, ad networks, web browsers & operating systems.

Arguably in most cases the idea of neutral-based publishing no longer works on the modern web. The shill gets exclusive stories. The political polemic gets automatic retweets from those who identify. The content which lacks agenda probably lacks the economics to pay for ads & buy distribution unless people can tell the creator loves what they do so much it influences them enough to repeatedly visit & perhaps pay for access.