Web Search Results: Biases, Inaccuracies and Their Consequences

This paper is available on arxiv under CC 4.0 license.

Authors:

(1) Aleksandra Urman, She is a corresponding author from Department of Informatics, University of Zurich, Switzerland;

(2) Mykola Makhortykh, Institute of Communication and Media Studies, University of Bern, Switzerland;

(3) Roberto Ulloa, GESIS - Leibniz-Institut für Sozialwissenschaften, Germany;

(4) Juhi Kulshrestha, Department of Politics and Public Administration, University of Konstanz, Germany.

Table of Links

Web search results: biases, inaccuracies and their consequences

In recent years a growing number of studies, primarily from the fields of computer science and communication science, have examined the presence of various biases in search results. To do so, researchers usually rely on a methodology called algorithm impact auditing - an investigation of the outputs of an algorithmic system - such as SE’s information retrieval and ranking algorithms - with the aim to assess whether they contain biases or inaccuracies (Mittelstadt, 2016; see Bandy, 2021 for an overview of audits across different domains).

Algorithm audits in the context of web search have been largely focused on how specific biases lead to distortions in search results (Bandy, 2021). Scholars have found evidence of gender bias in image search (Kay et al., 2015; Otterbacher et al., 2017; Makhortykh et al., 2021a) and in Google’s Knowledge Graph Carousel (Lipani et al., 2021), political bias in text search (Kravets and Toepfl, 2021; Kulshrestha et al., 2019), biases related to historical content in image search (Makhortykh et al., 2021b), and various forms of source-related biases (Diakopoulos et al., 2018; Haim et al., 2018; Kravets and Toepfl, 2021; Makhortykh et al., 2020; Puschmann, 2019; Trielli and Diakopoulos, 2019; Unkel and Haim, 2021; Urman et al., 2021a, 2021b). Further, several studies have documented information inequalities that can arise not only between users of different SEs but also between the users of the same SE due to search personalization or randomization (Hannak et al., 2013; Kliman-Silver et al., 2015; Paramita et al., 2021). Importantly, these inequalities can pertain to information crucial for individual and societal well-being, for instance concerning suicide helplines (Haim et al., 2017) or COVID-19 (Makhortykh et al., 2020).

While bias in web search received a lot of attention from scholars, there are only a few studies looking specifically at factually inaccurate information in search results (Bernstam et al., 2008; Bradshaw, 2019; Cooper and Feder, 2004). However, the possibility of web search promoting inaccurate information is concerning for a number of reasons. First, people highly trust the information they encounter through SEs, which are viewed as more reliable sources of information than legacy media (2021 Edelman Trust Barometer | Edelman, n.d.); consequently, the probability of individuals trusting inaccurate information acquired via SEs is high. Second, the mere act of using SEs increases individuals' confidence about their knowledge on the topic and motivates them to conflate external knowledge - that is the one available online - with their own (Ward, 2021). Third, people’s search behaviour and interpretation of results are subject to selective exposure and confirmation bias that leads to them perceiving incorrect information that aligns with their existing attitudes as reliable and dismissing the results that do not align with their attitudes (Knobloch-Westerwick et al., 2015; Nichols, 2017; Suzuki and Yamamoto, 2020). Fourth, search results can sway people’s opinions including on issues as important as voting preferences (Epstein and Robertson, 2015; Zweig, 2017) or commercial brand choices (Jansen et al., 2011). Fifth, inaccuracies in search results can proliferate to other socio-technical and informational systems that are built with reliance on SE outputs such as fake news detection algorithms relying on search outputs for the cues as to which information is false (Shim et al., 2021; Varshney and Vishwakarma, 2021).

Considering the potential harm which can be caused by search bias and factually inaccurate information, we suggest that comprehensive auditing of web search is of no less importance than the critical examinations of content appearing on legacy or social media. We strive to add to the body of literature on web search malperformance through an analysis of presence of conspiratorial information in search results. As we highlight below, this is a highly relevant subject due to the role the internet plays in the proliferation of conspiracy theories.