The future of online search engines. Part 1 – The status quo and its shortcomings.

Photo by Dino Reichmuth on Unsplash

Introduction

Search engines, Google in particular, have had a tremendous impact on our society. Every year, people search online or “google” at least 1 trillion times. In a matter of seconds, people can access a plethora of content. The Internet and the way that information can be found there are a powerful force that has disrupted many different industries.

In a series of articles, I will summarize and explain the online search engine industry. In this article, I will focus on the basic facts about the industry and its biggest challenges. In later articles, I will review the newer entrants to the industry and discuss what they can offer in order to explain where the industry may go in the future.

How the industry looks today

Market structure

The online search engine industry is currently dominated by Google, the market leader in almost every country with the important exception of China where Baidu leads instead. Besides Google, Bing and Yahoo! also offer search engines, however, they possess a much less impressive share of the market. The current market split is the following:

• Google - 86.28%
• Bing - 6.45%
• Yahoo! - 3.41%
• Baidu - 0.93% ¹

Google, Bing, and Yahoo! are all US-based companies with a similar business model. The most important difference between Baidu and the rest of the industry is their level of censorship. Baidu, a Chinese company, is strictly regulated by the Chinese government, which is not known for allowing freedom of speech. ²

In the rest of this article, I will focus mainly on Google due to its dominant position and the fact that the other US-based companies are not highly differentiated. Baidu is not discussed in this article because, in many aspects, Google’s weaknesses are Baidu’s strengths, which will be explained in more detail later on. Baidu’s market share around the world is also negligible in comparison to Google’s.

Business model

Google’s business model is based on advertising. In 2017, its ad revenue amounted to 95.38 billion dollars compared to Google’s total revenue, which amounted to 110.9 billion US dollars. Google Search is not the only location where Google displays ads. Many free services available on Google, such as YouTube, are also monetized in a similar manner by generating revenue from ads. ³

Bing, owned by Microsoft, generates a much smaller revenue percentage from its paid search advertising than Google AdWords does for Google. In 2017, Bing’s ad revenue amounted to 1.8 billion compared to the company’s total revenue of 89.95 billion US dollars. Microsoft also makes money from advertising on its service, LinkedIn, which generated 1.3 billion US dollars in the same year. ⁴ Similar to Google, Microsoft offers various products and services that also generate revenue from advertising.

Yahoo!’s reliance on advertising is also smaller than Google’s but much bigger than Microsoft’s. In 2015, its ad revenue amounted to 3.32 billion compared to its total revenue of 5.2 billion US dollars. ⁵ Similar to Google and Microsoft, Yahoo! provides a plethora of free services that rely on advertising as their main source of revenue.

In all cases, their search engines are only one of the building blocks for a larger business model. Each company experiments with different products and revenue models. Even though advertising is the main revenue source for Google and Yahoo!, both companies are exploring different profit streams. For example, Google hopes to generate other sources of income through its parent company, Alphabet.

Challenges

Google Search provides an easy-to-use and convenient service for its users that is completely free-of-charge. Over the years, this service has evolved significantly. Google Search delivers relevant results even when people are not sure exactly what they are looking for or how to precisely formulate their questions. Together with its email service, Gmail, and its mobile operating system, Android, Google can serve as a personal assistant in our pockets. Google tries to predict what we might need and at what time to present relevant information even before we search for it.

However, Google’s services are not free of shortcomings. There are, at least, a few concerns, which are mostly related to the political and economic power that Google possess. I will try to briefly explain each of these concerns below. The list is definitely not comprehensive. Also, the following discussion focuses on the broader implications for our society rather than on the technical or functional challenges.

Censorship and discrimination

The Internet and its protocols were designed to work on top of various physical networks that belong to or are maintained by many different parties. Its architecture enables it to be a relatively open medium, where anyone can join and share their opinions. However, the architecture itself is not the only important building block of the community’s openness. The concept of net neutrality is the second ingredient that ensures an open community. Wikipedia defines it as follows:

Net neutrality is the principle that Internet service providers treat all data on the Internet equally, and not discriminate or charge differently by user, content, website, platform, application, type of attached equipment, or method of communication. For instance, under these principles, internet service providers are unable to intentionally block, slow down or charge money for specific websites and online content. This is sometimes enforced through government mandate. These regulations can be referred to as “common carrier” regulations. This does not block all abilities that Internet service providers have to impact their customer’s services. Opt-in/opt-out services exist on the end user side, and filtering can be done on a local basis, as in the filtration of sensitive material for minors. Net neutrality regulations exist only to protect against misuse.

Here is a great example from the physical world that explains the importance of net neutrality:

As a simple example, if a man operates the only ferry over to town, that simple boatman is in a position of great power over other sectors of the economy, even the sovereign authorities. If, for example, he decided to charge one butcher more than another to carry his goods, this operator could bankrupt the one who didn’t enjoy his favor. The boatman is thus deemed to bear responsibilities beyond those of most ordinary businesses. ⁶

In other words, whoever runs the infrastructure should be equally open to all users of its service. Otherwise, the business would possess to much power over everyone relying on it.

As of June 2018, the US government, the Federal Communications Commission to be precise, decided to repeal the existing net neutrality rules. The FCC’s Chairman, Ajit Pai, explained:

The United States is simply making a shift from pre-emptive regulation, which foolishly presumes that every last wireless company is an anti-competitive monopolist, to targeted enforcement based on actual market failure or anti-competitive conduct. ⁷

An anti-competitive monopolist is precisely what AT&T has been acting as for many years. They’ve mastered a practice of delaying the enforcement of laws in court, which has enabled them to gain a competitive advantage. Tim Wu’s book, The Master Switch, provides plenty of examples of the information industry’s dirty practices. The industry, on the other hand, claims that they will maintain the openness of market practices even without oversight.

Several internet providers made public pledges that they would not block or throttle sites once the rules were repealed. The companies argued that Title II gave the F.C.C. too much control over their business, and that the regulations made it hard to expand their networks.

Yet, Wu argued:

The effects at first would be small. But it doesn’t make a genius to realize that if AT&T and the cable companies exercised broad discretion to speed up the business of some firms and slowdown that of others, they would gain the power of life and death over the Internet.

Simultaneously, AT&T is merging with Time Warner. Whose content do you think will be given preferential treatment on their network? Andrew Hwang from Roosevelt’s Institute explained some potential negative consequences of the merger.

These same concerns apply to other companies like Google and Facebook. Both of these companies, collectively, are a major source of incoming traffic for the rest of the web. ⁸ Unfortunately, they are not known for always maintaining neutrality. For instance, Google was accused of giving unfair priority to its own services when customers performed local searches. ⁹ Even if they were unprejudiced, we shouldn’t be expected to rely on their self-enforced neutrality. Civilization hasn’t been built upon the blind acceptance of others’ benevolence. We have developed rules and regulations to protect the greater good rather than the interests of a few; we have acknowledged that people often will act in their own self-interest if left unchecked. It’s yet to be seen if Google, Facebook, and others will be subject to more regulation. ¹⁰ For now, they allege that they are “doing the right thing,” which we hope doesn’t hide more selfish motives or an abuse of their position.

Privacy

Search engines and Google Search, in particular, are great at providing you with information that exactly matches what you are looking for. There is an important element of their platform that makes this possible - tracking. The more the company can learn about your interests, relationships, job, habits, location, etc., the more accurate the results of your search will be. The company’s goal is to predict what you might need and deliver it precisely when you might need it. ¹¹ Convenient? Absolutely! However, let’s first examine the tradeoffs that we, knowingly or unknowingly, accept in exchange for this convenience.

Many democracies around the world rely on a concept of “separation of powers”. Typically, democratic governments are divided into three branches: a legislature, an executive, and a judiciary. This model ensures a system of checks and balances to avoid autocracy. By this principle, the same balance should be maintained between the government and people. Privacy is one of the fundamental building blocks of the people’s defense against government interference or overreach. ¹²

There is a common misconception about privacy. People believe that they don’t need it because they have nothing to hide. Edward Snowden countered this argument beautifully with the following statement:

Some might say “I don’t care if they violate my privacy; I’ve got nothing to hide.” Help them understand that they are misunderstanding the fundamental nature of human rights. Nobody needs to justify why they “need” a right: the burden of justification falls on the one seeking to infringe upon the right. But even if they did, you can’t give away the rights of others because they’re not useful to you. More simply, the majority cannot vote away the natural rights of the minority.

But even if they could, help them think for a moment about what they’re saying. Arguing that you don’t care about the right to privacy because you have nothing to hide is no different than saying you don’t care about free speech because you have nothing to say. ¹³

Now, what does this have to do with Google? Revealing your personal data to a company makes you much more vulnerable to their control and influence. Moreover, these companies and the information that they possess are within reach of the government. ¹⁴ The companies and the government could use your personal information to influence your behaviors to their own advantage. Convenience comes at a cost, and it’s important to understand the potential consequences that might arise.

Transparency

Transparency has been agreed upon as an important factor that ultimately improves our society. It’s been touted as an important principle and practiced in varying degrees within many different environments, including businesses and governments. Since Google Search plays such an important role in our society, some people believe they should be more transparent about how they rank all results. The answer to that question lies in their algorithms and how they work. Therefore, there has been a greater call for transparency about those very algorithms. ¹⁵ Google maintains great secrecy about its algorithms. They offer these two concerns as their primary reasons for not being more transparent:

1. It’s their proprietary recipe to maintain their competitive lead within the market
2. It helps them to protect the system from being gamed by outside sources ¹⁶

The first point is certainly valid for Google. For the rest of us, it’s less important who the market leader is. It’s hard to measure the significance and impact of the company’s lead within the market. What should matter to us as consumers are the benefits that the company can offer for the society as a whole.

The argument from the second point could be compared to what has been happening in the open-source movement. Some people believed that open-source software would be more vulnerable to attacks since it was by definition more “open.” In fact, studies have shown that open-source software is not susceptible to higher security risks. ¹⁷ Admittedly, Google is a complex system, which shouldn’t be analyzed only from a technical perspective. Too narrow an analysis may lead to terrible second- and third-order consequences. Carole Cadwalladr, in her article “Google, democracy and the truth about internet search”, shed some light on how broad and negative the implications might be of a closed system with a lot of power. Fake news, racism, and manipulation of the electorate all made the list.

There is plenty of reverse engineering, analysis, speculation and even some tidbits of information released by Google itself that allow us to begin to understand how Google Search might work. With Google becoming an “AI-first” company, it will only become harder to grasp rather than easier. Even their own engineers occasionally find it difficult to assess why certain search results rank higher than others. ¹⁸ The quality of the content, which is often touted as the most important metric, is not a silver bullet. Otherwise, the entire SEO industry would be out of business. If we, as a society, don’t know the rules on which we have to rely, how do we know they are good for us?

Whether or not we need more transparency and to what degree we need it are debatable topics. Within that debate, Google’s opinion, although certainly valuable as the leader of the industry, shouldn’t be the dominant voice. They are too biased because of their market position and the potential profits that they could lose. It also means that an alternative approach is more likely to come from an outsider. It would be easier for an outsider to challenge the status quo since they are not in a position to lose so much.

Advertising

Google’s founders, Larry Page and Sergey Brin, have been aware of the bias that advertising introduces to search engines since Google’s early days. In their paper, “The Anatomy of a Large-Scale Hypertextual Web Search Engine”, they wrote:

Currently, the predominant business model for commercial search engines is advertising. The goals of the advertising business model do not always correspond to providing quality search to users. For example, in our prototype search engine one of the top results for cellular phone is “The Effect of Cellular Phone Use Upon Driver Attention”, a study which explains in great detail the distractions and risk associated with conversing on a cell phone while driving. This search result came up first because of its high importance as judged by the PageRank algorithm, an approximation of citation importance on the web [Page, 98]. It is clear that a search engine which was taking money for showing cellular phone ads would have difficulty justifying the page that our system returned to its paying advertisers. For this type of reason and historical experience with other media [Bagdikian 83], we expect that advertising funded search engines will be inherently biased towards the advertisers and away from the needs of the consumers.

In short, what advertisers want is not necessarily aligned with the best interests of the users. If a search engine monetizes advertising, it is then incentivized to act in the best interest of the advertisers and not the consumers. In such case, the search results are not the product, the consumers are. For years, Google has been able to balance both forces, the advertisers and the consumers, successfully. All the while, they have been extending their level of influence and collecting a handsome paycheck.

In the meantime, online advertising could be becoming a less and less effective means of improving sales, ¹⁹ Google and Facebook both leverage their massive data to improve targeting, ²⁰ ad blockers are on the raise, and publishers still struggle to monetize their content. There is room for innovative business models to emerge. Kevin Werbach, a Wharton legal studies and business ethics professor, made a great point:

It’s not written in stone that advertising has to be the dominant monetization model for online and mobile services. ²¹

Conclusion

Google’s free services and convenience come at a cost. The long-term consequences and tradeoffs are still difficult to accurately measure and properly assess. The list of challenges presented here is definitely not exhaustive. My goal here was to identify the most important problems and lay the groundwork for further analysis.

In the next article, I will discuss the landscape of new entrants to the industry and how they might address the issues presented here. In the meantime, feel free to share any questions, suggestions, or comments below. Follow me on Twitter.

Notes