The Google Search Distortion

Consider this question:

Is it morally right for Google to rank all vendors by popularity?

If the answer is ‘yes’, what moral basis does Google have for pushing those highly deserving popular vendors further down the results page, to be displaced by the AdWords results of less deserving, less popular vendors (simply because those less popular vendors pay Google)?

If the answer is ‘no’, what right does Google have to homogenize a large proportion of the economy by forcing a popularity heuristic on it? To appoint some arbitrary vendors to be kings (simply because they are ‘popular’ or, more likely, to have a good content and SEO team) and force the remainder to become an underclass of vendors who must pay Google to be found and participate in trade?

The answer, of course, is ‘none of the above’. Google is using the wrong kind of search results for products and services.

The wrong kind of search results

Google Search is designed for finding facts. That is, it identifies the web resource which most reliably and accurately represents those facts, from a host of web resources which contain part or all of those facts.

This is why popularity is a useful heuristic for fact-finding – people will link to and visit sites which they find most useful (by representing facts most clearly).

It also why Knowledge Graph results (on the right of the SERP) are useful to people, because people looking for facts don’t even want the web resource, they just want the facts:

[INSERT IMAGE]

Now consider what an appropriate search result would be for the phrase ‘plumber NYC’. Google says it’s a list ranked by popularity (and, in recent years, modified the SERP to display information from other Google properties such as Maps or Places).

If you consider what ‘plumber NYC’ actually represents, however, you would arrive at a Venn diagram in which the set of people who are plumbers intersects with the set of people who are NYC. That is the only true result for that search.

[INSERT VENN DIAGRAM]

Of course, that data set is not particularly useful to people.

To make it useful to each buyer, Google would provide:

  1. faceted search which allows people to drill down by many different factors – service type, availability, location, maximum price, guarantees, credentials and so on, and then
  2. a choice of ranking heuristics which represent the preference of each individual buyers – the fastest, cheapest, most trustworthy, longest in business, newest in business and so on.

This kind of search would be incredibly useful to buyers, allowing them to find a small set of plumbers, any of which they would be happy to do business with. And it would create a diverse and distributed market for plumbing in NYC, with plumbers having a niche to operate within and being found by buyers to trade with them.

That kind of search, however, would not make billions of dollars for Google.

How Google makes money from Search

It’s also a something that would make Google no money because it has no scarcity (at the top of search results) to exploit.

I put it to you, then, that Google stumbled across a way to extract money from our transactions through their Search product and, justifies continuing to do so by perpetuating the myth that ‘web search’ is a sacrosanct public service (rather than changing it so that it actually helps buyers find the products and services that they actually want and helps all vendors to be found for trade).

  1. GOOGLE IS PRIORITIZING CONVENIENCE AND PROFIT OVER USEFULNESS (WHICH IS ITS MISSION).
  2. GOOGLE A/B TESTS FOR USAGE AND PROFIT, NOT PUBLIC BENEFIT.
  3. HARDER TO DO
  4. ARBITRARY ELITE
  5. KNOWING THE VALUE OF SEARCH (TO REDUCE UNCERTAINTY IN BIDDING -> VALUE TO GOOGLE, WHERE WE NEED UNCERTAINTY IN VALUE TO HAVE OPPORTUNITY)

The search engines of the early 1990s were distorted by advertising – companies could pay the search engines to have their website highly on the Search Engine Results Page (SERP). Google’s founders recognized this problem and created a pure algorithm for web search, called PageRank.

As Google’s founders foretold, however, advertising creates mixed incentives and Google’s SERP is not immune from these.

In my view the problems with Google Search are as follows:

1.the homogenization of billions of people’s search (to allocate trade to a tiny proportion of vendors and, therefore, fostering economic inequality),
2. the heuristic used is popularity (in which positions become entrenched),
3. the representation of offers are compressed to a title and couple of lines of text (such that users cannot understand the difference between and diversity of offers, favoring the people at the top),
4. the misapplication of a ranked list results paradigm to offers of products and services (because search is optimized for finding universal facts and not for exploring the diversity of products and services – I can explain further if you wish),
5. Google’s selfish addition of a second heuristic of ‘ability to pay Google’ to drive AdWords results (in which the underclass of unpopular vendors must compete with each other to give their margins to Google),
6. The distortion of our information through a popularity-based feedback loop (in which only popular information is discoverable), and
7. Google’s abuse of our personal data.

DuckDuckGo only addresses the last point. Otherwise it’s a carbon copy of Google (with less traction, so the ads will be cheaper … for now).

We’re so convinced that the biggest problem with search is privacy that we’ve completely ignored the underlying economic an informational problems. The public is voluntarily giving away our privacy because that’s the only form of aggregation made available to us. If we had a better form of aggregation (offerbots) we would not need to.

That said, I use DDG because I’m thankful that my privacy is not abused. I will continue to use DDG until I have an offerbot (and have real search available to me)

Google’s SERP is a distorted map which captures our attention to Search results (unpaid offers) and sells it by redirecting our attention to AdWords results (paid offers).

To make sense of how Google is able to manipulate us in this way, we need to dive into some detail on the differences between ‘fact search’ (which is how Google captures our attention) and ‘product search’ (which is how Google sells our attention)[1].

We use Google Search (i.e. we give Google our attention) because it helps us to reliably find facts – how fast a cheetah runs[2], what the weather will be like today or where the nearest Thai restaurant is. Web publishers offer these facts to us in their web documents, Google receives (crawls) those offers and processes them to generate their Search results.

We also use Google Search because it’s cheap. It costs no money to the people who offer facts and it costs little attention to the people receiving facts because they only have to spend attention on:

  1. reading the first result in Search, and
  2. clicking through and reading the web page (that was the first result in Search):

In other words, Google Search conserves our attention by eliminating the need for us to personally read and click through many sites trying to find a fact (a task which has a high attention cost):

There are two reasons that Google is able to make their Search results so cheap for receiving facts. The first reason, the one that we’ve been told over and over again, is that Google’s founders are very smart and came up with a better algorithm than everyone else[3].

The second reason, however, is more fundamental than that – it’s that facts have a correct answer. Cheetahs run at a certain speed[4], the weather bureau releases only one forecast for your city and there is only one Thai restaurant nearest to your current location. It is the nature of facts (more than the nature of Google’s algorithm) which allows websites to be ranked according to their relevance to a fact (or, possibly, the truthfulness of their fact claims[5]) such that we can simply go to the first search result and obtain our fact.

This inherent nature of facts has allowed Google to go one step further in conserving attention on their SERP by ‘Knowledge Graph’ results (a summary of the fact we’re looking for) next to Search results[6]. This means that we can get the fact directly from Google rather than ‘wasting’ attention by visiting the website (e.g. Wikipedia) which offered the fact in the first place:

Google has dedicated itself to being on the cutting edge of search technology and, therefore, on the cutting edge of conserving our attention. We never stop giving Google Search our attention because we know that Google will never stop making Search better and better. Except, of course, where they don’t.

This is where we need to pay attention to the difference between facts and products, because they are not the same kinds of offers, and, therefore, we need a different kind of search.

Searching for products is fundamentally different from searching for facts, because offers of products have no correct answer. If you were, for instance, to search Google for a ‘notebook computer’, there is no way that Google could deliver the very best offer of a notebook computer to the top of search results because there is no such thing. There are only many different notebook computers with many different specifications, offered by many different vendors on many different terms which may, or may not, meet the needs of many different people in the many different situations they find themselves in.

When we’re searching for products, therefore, we’re not looking to find a correct answer, we’re looking to build a consideration set of offers[7] that we can compare to each other and to our needs. That is, we want to find a number of different offers[8] so that we can understand the range of offers available, determine the important differences and, ultimately, select an offer that suits our needs best.

And, it turns out, Google Search is a costly method of finding and comparing these offers, because it forces us to use our attention (to spend our processing power rather than Google’s) to find, receive and consider them. That is, Search results gives us a list of links to websites that we must click on in order to visit each offering party’s site, navigate their web document and receive the offers from each of those web documents (and then rinse and repeat with the next Search result):

WORSE, IT SHOULD BE A VENN DIAGRAM NOT A RANKED LIST. IT SHOULD BE FACETED SEARCH.

Google Search wastes our attention when looking for products , which is why most people don’t go past the first page of Search results and, therefore, why it’s valuable to appear on that first page – because.

What Google did, therefore, was also include another kind of results on its SERP –  AdWords results – above and to the right of Search results:

One form of AdWords results are Google’s Product Listing Ads which appear above Search results[9]:

Like Google’s Knowledge Graph results, Product Listing Ads conserve our attention by avoiding the need to visit websites to gather those offers – offers can be collected directly from the SERP.

More than this, however, clicking on the ‘Shop for …’ link above these Product Listing Ads opens Google Shopping – a map in which Google conserves our attention by providing:

  1. offer information (pictures, product names, descriptions and prices),
  2. processing power (by grouping offers from multiple vendors against each product and providing faceted search and multiple sorting/display options), and
  3. shortlist storage (so that we don’t need to remember our consideration set):

In other words, Google offers two kinds of product search results on their SERP – one which is very costly in terms of our attention and another which is incredibly cheap:

We could consider AdWords to be yet another helpful Google service designed to conserve our attention, except that we must remember that Google operates a two-sided market[10] – it has designed its map so that our behaviour as receiving parties will result in payments from offering parties.

It’s no surprise, then, to learn that every result in AdWords and Shopping is a ‘sponsored’ offer – the offering party must pay Google to have their offers appear on that map. Nor is it a surprise to see that what’s cheap for the receiving party is costly for the offering party:

Google is, therefore, deliberately manipulating our attention costs as we navigate their map so that we ‘choose’ AdWords and ignore Search results, forcing offering parties onto the costly path which requires paying Google[11]:

The ever-increasing convenience and much-touted relevance of Google advertising[12], therefore, is not a benefit to receiving parties (Google’s users) but a weapon wielded against offering parties (who are forced to pay for advertising).

The same applies to the ease with which offering parties can bid on AdWords – this is not a benefit for small business[13], but a means of “overselling” advertising space (increasing the number of people bidding for each keyword) so that the price of bids can be maximised[14].

The success of this business model relies on Google doing two things:

  1. capturing our attention by giving us more and more ‘free’ services (fact search, Maps, Android, etc.) so that we use Google for everything (including product searches), and
  2. selling our attention by constantly innovating their AdWords results[15] (and not innovating Search results[16]) so that there is a significant convenience gap which drives receiving parties toward AdWords results.

In other words, everything else that Google ‘gives’ us is merely a tool for:

  1. capturing our attention so that it can be sold in this way, and
  2. capturing all of our data so that Google’s founders can use it (and the profits collected from selling our attention) to fulfil their personal goal of building an artificial intelligence which does everything[17].

This come as a surprise to us as Google users, given that they took the high moral ground in the purity of Search results, at a time when other search engines were taking secret payments to rank certain websites higher than others.

What we can see here, however, is a different kind of distortion in which Search results might be unsullied by payments, but that we are discouraged from using Search results at all.

This is predicted by Google’s founders in the paper they to describe their search engine (Brin & Page, 1998) which appendix entitled ‘Advertising and mixed motives’. There they write a highly prophetic statement: “We expect that advertising funded search engines will be inherently biased towards the advertisers and away from the needs of the consumers”.

[1] “The claim that we’re making big profits on the back of newspapers

[i.e. fact search]

also misrepresents the reality. In search, we make our money primarily from advertisements for products.” Eric Schmidt, Executive Chairman of Google http://www.wsj.com/news/articles/SB10001424052748704107104574569570797550520

[2] [Cite Matt Cutts video on search relevance]

[3] http://infolab.stanford.edu/~backrub/google.html

[4] [Cite Matt Cutts video with cheetah running speed]

[5] http://www.newscientist.com/article/mg22530102.600-google-wants-to-rank-websites-based-on-facts-not-links.html#.VWKGM0Y5M8I

[6] http://googleblog.blogspot.com.au/2012/08/building-search-engine-of-future-one.html

[7] See (Hauser, 2013) for an explanation of consideration sets.

[8] See (Stigler, 1961) for insight on how many offers we need to receive and the nature of our search costs as we receive them.

[9] http://www.google.com/ads/shopping/getstarted.html

[10] Hal Varian, Google’s Chief Economist: “… what does Google do? The answer, I claim, is that Google is a ‘yenta’ — a traditional Yiddish word for ‘match-maker’. On the search side, it matches people who are seeking information to people who provide information. On the ad side, it matches people who want to buy things to those who want to sell things. From an economics perspective, Google runs a ‘two sided matching’ mechanism.” (Varian, 2007)

[11] That is, Google’s SERP has a self-serving choice architecture (Thaler, Sunstein, & Balz, 2010) which strongly nudges us to choose paid offers in preference to unpaid offers. Receiving parties choose something which lowers their cost, forcing others to increase theirs (just like credit companies in which the buyer chooses the convenience of their credit card, forcing the vendor to share their revenue with the credit card provider).

[12]  Cite ‘ten things we know to be true’ on Google website

[13]

[14] See (Varian, 2007), in which Google’s Chief Economist explains how Google’s auction works and the benefits (to Google) of ‘oversold’ AdWords space.

[15] See the long list of AdWords innovations at Google’s official blog: http://adwords.blogspot.com.au/

[16] The structure and display of Search results has changed remarkably little since Google launched their Search product. They’ve certainly made many changes to their algorithms and incorporated many different types of results (Local, Places, Knowledge Graph, etc.) alongside the web search results on the SERP, but Google has made few innovations which might conserve our attention as we gather product offers through Search results.

[17] [Say something along the lines of ‘This is an important topic in and of itself, but this isn’t the place’. Cite Google and the world brain, and the Stanford video if I mention ‘does everything’]