A Comparative Algorithm Audit of Conspiracies on the Net: Conclusion and Bibliography

Table of Links

• Abstract and Introduction
• Web search results: biases, inaccuracies and their consequences
• Conspiracy theories in the online environments
• Research questions
• Methodology
• Results
• Discussion
• Limitations
• Conclusion and Bibliography

Conclusion

We found that most of the search engines display conspiracy-promoting results, though the share of such results varies across specific conspiracy-related queries. In our sample most conspiracy-promoting results came from social media platforms and dedicated conspiracy websites, while debunking information was found predominantly on scientific websites and, to a smaller extent, on legacy media. Our observations are robust across several locations and two time periods. The good news is that Google - the search engine with the biggest market share - has managed to mitigate the problem to a few isolated instances. We suggest that the example of Google shows that conspiratorial results can be effectively handled to become less prevalent in top SE outputs, and that other SEs should follow suit and put the results they provide under higher scrutiny not only with regard to conspiracy theories but other types of inaccurate and/or biased information. This is especially relevant and timely as of now, when radical groups are attempting to create an alternative tech ecosystem and, among other, migrate from Google to DuckDuckGo, accusing the former of censorship and reinforcing the prioritization of far-right and/or conspiratorial sources on the latter (Diggit Magazine, 2020). It shows the potential for ideologically charged hijacking of smaller SEs that can influence their outputs. Against this backdrop, it is crucial to assure the quality of information SEs provide to users through both SEs’ own internal monitoring and external audits such as the one conducted in the present study.

Bibliography

2021 Edelman Trust Barometer | Edelman (n.d.). Available at: https://www.edelman.com/trust/2021-trust-barometer (accessed 4 March 2021).

Aupers S (2012) ‘Trust no one’: Modernization, paranoia and conspiracy culture. European Journal of Communication 27(1). SAGE Publications Ltd: 22–34. DOI: 10.1177/0267323111433566.

Bandy J (2021) Problematic Machine Behavior: A Systematic Literature Review of Algorithm Audits. arXiv:2102.04256 [cs]. Available at: http://arxiv.org/abs/2102.04256 (accessed 18 November 2021).

Bernstam EV, Walji MF, Sagaram S, et al. (2008) Commonly cited website quality criteria are not effective at identifying inaccurate online information about breast cancer. Cancer 112(6): 1206–1213. DOI: 10.1002/cncr.23308.

Bessi A, Zollo F, Vicario MD, et al. (2015) Trend of Narratives in the Age of Misinformation. PLOS ONE 10(8). Public Library of Science: e0134641. DOI: 10.1371/journal.pone.0134641.

Bradshaw S (2019) Disinformation optimised: gaming search engine algorithms to amplify junk news. Internet Policy Review 8(4). Available at: https://policyreview.info/articles/analysis/disinformation-optimised-gaming-search-engi ne-algorithms-amplify-junk-news (accessed 1 November 2021).

Cooper JD and Feder HMJ (2004) Inaccurate Information About Lyme Disease on the Internet. The Pediatric Infectious Disease Journal 23(12): 1105–1108. DOI: 10.1097/01.inf.0000145411.57449.f3.

Diakopoulos N, Trielli D, Stark J, et al. (2018) I Vote For—How Search Informs Our Choice of Candidate. al Digital Dominance: The Power of Google, Amazon, Facebook, and Apple, M. Moore and D. Tambini (Eds.): 22.

Diggit Magazine (2020) ‘Dems Fraud’: Far Right and Data Voids on DuckDuckGo.com. Available at: https://www.diggitmagazine.com/articles/dems-fraud-data-voids (accessed 2 November 2021).

Douglas KM, Uscinski JE, Sutton RM, et al. (2019) Understanding Conspiracy Theories. Political Psychology 40(S1): 3–35. DOI: 10.1111/pops.12568.

Epstein R and Robertson RE (2015) The search engine manipulation effect (SEME) and its possible impact on the outcomes of elections. Proceedings of the National Academy of Sciences 112(33): E4512–E4521. DOI: 10.1073/pnas.1419828112.

European Commission E (2021) Identifying conspiracy theories. Available at: https://ec.europa.eu/info/live-work-travel-eu/coronavirus-response/fighting-disinformati on/identifying-conspiracy-theories_en (accessed 30 November 2021).

Fisher M, Goddu MK and Keil FC (2015) Searching for explanations: How the Internet inflates estimates of internal knowledge. Journal of Experimental Psychology: General 144(3): 674–687. DOI: 10.1037/xge0000070.

Google (2021) Google Search Quality Rater Guidelines. Google. Available at: https://static.googleusercontent.com/media/guidelines.raterhub.com/en//searchqualityev aluatorguidelines.pdf (accessed 29 October 2021).

Haim M (2020) Agent-based Testing: An Automated Approach toward Artificial Reactions to Human Behavior. Journalism Studies 21(7). Routledge: 895–911. DOI: 10.1080/1461670X.2019.1702892.

Haim M, Arendt F and Scherr S (2017) Abyss or Shelter? On the Relevance of Web Search Engines’ Search Results When People Google for Suicide. Health Communication 32(2). Routledge: 253–258. DOI: 10.1080/10410236.2015.1113484.

Haim M, Graefe A and Brosius H-B (2018) Burst of the Filter Bubble? Digital Journalism 6(3). Routledge: 330–343. DOI: 10.1080/21670811.2017.1338145.

Hannak A, Sapiezynski P, Molavi Kakhki A, et al. (2013) Measuring personalization of web search. In: Proceedings of the 22nd international conference on World Wide Web, New York, NY, USA, 13 May 2013, pp. 527–538. WWW ’13. Association for Computing Machinery. DOI: 10.1145/2488388.2488435.

Harambam J (2021) Contemporary Conspiracy Culture: Truth and Knowledge in an Era of Epistemic Instability. S.l.: ROUTLEDGE.

Houli D, Radford ML and Singh VK (2021) “COVID19 is_”: The Perpetuation of Coronavirus Conspiracy Theories via Google Autocomplete. Proceedings of the Association for Information Science and Technology 58(1): 218–229. DOI: 10.1002/pra2.450.

Jansen BJ, Sobel K and Zhang M (2011) The Brand Effect of Key Phrases and Advertisements in Sponsored Search. International Journal of Electronic Commerce 16(1). Routledge: 77–106. DOI: 10.2753/JEC1086-4415160103.

Kay M, Matuszek C and Munson SA (2015) Unequal Representation and Gender Stereotypes in Image Search Results for Occupations. In: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. New York, NY, USA: Association for Computing Machinery, pp. 3819–3828. Available at: https://doi.org/10.1145/2702123.2702520 (accessed 2 February 2021).

Kayser-Bril N (2020) Ten years on, search auto-complete still suggests slander and disinformation. In: AlgorithmWatch. Available at: https://algorithmwatch.org/en/auto-completion-disinformation/ (accessed 27 October 2021).

Kliman-Silver C, Hannak A, Lazer D, et al. (2015) Location, Location, Location: The Impact of Geolocation on Web Search Personalization. In: Proceedings of the 2015 Internet Measurement Conference, New York, NY, USA, 28 October 2015, pp. 121–127. IMC ’15. Association for Computing Machinery. DOI: 10.1145/2815675.2815714.

Knobloch-Westerwick S, Johnson BK and Westerwick A (2015) Confirmation Bias in Online Searches: Impacts of Selective Exposure Before an Election on Political Attitude Strength and Shifts. Journal of Computer-Mediated Communication 20(2): 171–187. DOI: 10.1111/jcc4.12105.

Kravets D and Toepfl F (2021) Gauging reference and source bias over time: how Russia’s partially state-controlled search engine Yandex mediated an anti-regime protest event. Information, Communication & Society 0(0). Routledge: 1–17. DOI: 10.1080/1369118X.2021.1933563.

Kulshrestha J, Eslami M, Messias J, et al. (2017) Quantifying Search Bias: Investigating Sources of Bias for Political Searches in Social Media. In: Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing, Portland, Oregon, USA, 25 February 2017, pp. 417–432. CSCW ’17. Association for Computing Machinery. DOI: 10.1145/2998181.2998321.

Kulshrestha J, Eslami M, Messias J, et al. (2019) Search bias quantification: investigating political bias in social media and web search. Information Retrieval Journal 22(1): 188–227. DOI: 10.1007/s10791-018-9341-2.

Lewandowsky S, Oberauer K and Gignac GE (2013) NASA Faked the Moon Landing—Therefore, (Climate) Science Is a Hoax: An Anatomy of the Motivated Rejection of Science. Psychological Science 24(5). SAGE Publications Inc: 622–633. DOI: 10.1177/0956797612457686.

Lipani A, Piroi F and Yilmaz E (2021) Towards More Accountable Search Engines: Online Evaluation of Representation Bias. arXiv:2110.08835 [cs]. Available at: http://arxiv.org/abs/2110.08835 (accessed 26 October 2021).

Mahl D, Zeng J and Schäfer MS (2021) From “Nasa Lies” to “Reptilian Eyes”: Mapping

Communication About 10 Conspiracy Theories, Their Communities, and Main Propagators on Twitter. Social Media + Society 7(2). SAGE Publications Ltd: 20563051211017480. DOI: 10.1177/20563051211017482.

Makhortykh M, Urman A and Ulloa R (2020) How search engines disseminate information about COVID-19 and why they should do better. Harvard Kennedy School Misinformation Review 1(COVID-19 and Misinformation). DOI: 10.37016/mr-2020-017.

Makhortykh M, Urman A and Ulloa R (2021a) Detecting Race and Gender Bias in Visual Representation of AI on Web Search Engines. In: Advances in Bias and Fairness in Information Retrieval (eds L Boratto, S Faralli, M Marras, et al.), Cham, 2021, pp. 36–50. Communications in Computer and Information Science. Springer International Publishing. DOI: 10.1007/978-3-030-78818-6_5.

Makhortykh M, Urman A and Ulloa R (2021b) Hey, Google, is it what the Holocaust looked like? : Auditing algorithmic curation of visual historical content on Web search engines. First Monday. DOI: 10.5210/fm.v26i10.11562.

Merriam Webster Dictionary (n.d.) Definition of CONSPIRACY THEORY. Available at: https://www.merriam-webster.com/dictionary/conspiracy+theory (accessed 20 October 2021).

Metaxas P and Finn S (n.d.) The infamous #Pizzagate conspiracy theory: Insight from a TwitterTrails investigation.: 5.

Mittelstadt B (2016) Automation, Algorithms, and Politics| Auditing for Transparency in Content Personalization Systems. International Journal of Communication 10(0). 0: 12.

Mohammed SN (2019) Conspiracy Theories and Flat-Earth Videos on YouTube. The Journal of Social Media in Society 8(2). 2: 84–102.

Nichols T (2017) The Death of Expertise: The Campaign against Established Knowledge and Why It Matters. Oxford University Press.

Noble SU (2018) Algorithms of Oppression: How Search Engines Reinforce Racism. NYU Press.

Otterbacher J, Bates J and Clough P (2017) Competent Men and Warm Women: Gender Stereotypes and Backlash in Image Search Results. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, New York, NY, USA, 2 May 2017, pp. 6620–6631. CHI ’17. Association for Computing Machinery. DOI: 10.1145/3025453.3025727.

Pan B, Hembrooke H, Joachims T, et al. (2007) In Google We Trust: Users’ Decisions on Rank, Position, and Relevance. Journal of Computer-Mediated Communication 12(3): 801–823. DOI: 10.1111/j.1083-6101.2007.00351.x.

Paramita ML, Orphanou K, Christoforou E, et al. (2021) Do you see what I see? Images of the COVID-19 pandemic through the lens of Google. Information Processing & Management 58(5): 102654. DOI: 10.1016/j.ipm.2021.102654.

Puschmann C (2019) Beyond the Bubble: Assessing the Diversity of Political Search Results. Digital Journalism 7(6). Routledge: 824–843. DOI: 10.1080/21670811.2018.1539626.

Röchert D, Neubaum G, Ross B, et al. (2022) Caught in a networked collusion? Homogeneity in conspiracy-related discussion networks on YouTube. Information Systems 103: 101866. DOI: 10.1016/j.is.2021.101866.

Sallam M, Dababseh D, Yaseen A, et al. (2020) Conspiracy Beliefs Are Associated with Lower Knowledge and Higher Anxiety Levels Regarding COVID-19 among Students at the University of Jordan. International Journal of Environmental Research and Public Health 17(14). 14. Multidisciplinary Digital Publishing Institute: 4915. DOI: 10.3390/ijerph17144915.

Samory M and Mitra T (2018) Conspiracies Online: User Discussions in a Conspiracy Community Following Dramatic Events. Proceedings of the International AAAI Conference on Web and Social Media 12(1). 1. Available at: https://ojs.aaai.org/index.php/ICWSM/article/view/15039 (accessed 20 October 2021).

Schultheiß S, Sünkler S and Lewandowski D (2018) We still trust in Google, but less than 10 years ago: an eye-tracking study. University of Borås. Available at: http://informationr.net/ir/23-3/paper799.html (accessed 26 August 2020).

Shim J-S, Lee Y and Ahn H (2021) A link2vec-based fake news detection model using web search results. Expert Systems with Applications 184: 115491. DOI: 10.1016/j.eswa.2021.115491.

Slaughter A-M (2012) The Real New World Order. 18 January. Available at: https://www.foreignaffairs.com/articles/1997-09-01/real-new-world-order (accessed 29 October 2021).

Stano S (2020) The Internet and the Spread of Conspiracy Content. In: Routledge Handbook of Conspiracy Theories. Routledge.

StatCounter Global Stats (n.d.) Desktop Search Engine Market Share Worldwide. Available at: https://gs.statcounter.com/search-engine-market-share/desktop/worldwide (accessed 29 October 2021).

Suzuki M and Yamamoto Y (2020) Analysis of Relationship between Confirmation Bias and Web Search Behavior. In: Proceedings of the 22nd International Conference on Information Integration and Web-based Applications & Services, New York, NY, USA, 30 November 2020, pp. 184–191. iiWAS ’20. Association for Computing Machinery. DOI: 10.1145/3428757.3429086.

Trielli D and Diakopoulos N (2019) Search as News Curator: The Role of Google in Shaping Attention to News Information. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. New York, NY, USA: Association for Computing Machinery, pp. 1–15. Available at: https://doi.org/10.1145/3290605.3300683 (accessed 26 October 2021).

Ulloa R, Makhortykh M and Urman A (2021) Algorithm Auditing at a Large-Scale: Insights from Search Engine Audits. arXiv:2106.05831 [cs]. Available at: http://arxiv.org/abs/2106.05831 (accessed 26 October 2021).

Unkel J and Haas A (2017) The effects of credibility cues on the selection of search engine results. Journal of the Association for Information Science and Technology 68(8): 1850–1862. DOI: 10.1002/asi.23820.

Unkel J and Haim M (2021) Googling Politics: Parties, Sources, and Issue Ownerships on Google in the 2017 German Federal Election Campaign. Social Science Computer Review 39(5). SAGE Publications Inc: 844–861. DOI: 10.1177/0894439319881634.

Urman A and Makhortykh M (2021) You Are How (and Where) You Search? Comparative Analysis of Web Search Behaviour Using Web Tracking Data. arXiv:2105.04961 [cs]. Available at: http://arxiv.org/abs/2105.04961 (accessed 11 October 2021).

Urman A, Makhortykh M and Ulloa R (2021a) Auditing Source Diversity Bias in Video Search Results Using Virtual Agents. In: Companion Proceedings of the Web Conference 2021, New York, NY, USA, 19 April 2021, pp. 232–236. WWW ’21. Association for Computing Machinery. DOI: 10.1145/3442442.3452306.

Urman A, Makhortykh M and Ulloa R (2021b) The Matter of Chance: Auditing Web Search Results Related to the 2020 U.S. Presidential Primary Elections Across Six Search Engines. Social Science Computer Review. SAGE Publications Inc: 08944393211006863. DOI: 10.1177/08944393211006863.

Uscinski JE and Parent JM (2014) American Conspiracy Theories. Illustrated edition. Oxford ; New York: Oxford University Press.

Uscinski JE, Klofstad C and Atkinson MD (2016) What Drives Conspiratorial Beliefs? The Role of Informational Cues and Predispositions. Political Research Quarterly 69(1). SAGE Publications Inc: 57–71. DOI: 10.1177/1065912915621621.

Van Aelst P, Strömbäck J, Aalberg T, et al. (2017) Political communication in a high-choice media environment: a challenge for democracy? Annals of the International Communication Association 41(1): 3–27. DOI: 10.1080/23808985.2017.1288551.

van der Linden S (2015) The conspiracy-effect: Exposure to conspiracy theories (about global warming) decreases pro-social behavior and science acceptance. Personality and Individual Differences 87: 171–173. DOI: 10.1016/j.paid.2015.07.045.

Varshney D and Vishwakarma DK (2021) Hoax news-inspector: a real-time prediction of

fake news using content resemblance over web search results for authenticating the credibility of news articles. Journal of Ambient Intelligence and Humanized Computing 12(9): 8961–8974. DOI: 10.1007/s12652-020-02698-1.

Vosoughi S, Roy D and Aral S (2018) The spread of true and false news online. Science 359(6380). American Association for the Advancement of Science: 1146–1151. DOI: 10.1126/science.aap9559.

Ward AF (2021) People mistake the internet’s knowledge for their own. Proceedings of the National Academy of Sciences 118(43). National Academy of Sciences. DOI: 10.1073/pnas.2105061118.

Wood M and Douglas K (2013) “What about building 7?” A social psychological study of online discussion of 9/11 conspiracy theories. Frontiers in Psychology 4: 409. DOI: 10.3389/fpsyg.2013.00409.

Zweig K (2017) Watching the watchers: Epstein and Robertson’s „Search Engine Manipulation Effect“. In: AlgorithmWatch. Available at: https://algorithmwatch.org/en/watching-the-watchers-epstein-and-robertsons-search-en gine-manipulation-effect/ (accessed 1 November 2021).