Get the most recent info and news about InFarm on HackerNoon, where 10k+ technologists publish stories for 4M+ monthly readers.

InFarm is an agricultural AI company focused on delivering ground breaking technology to farmers. We use state of the art AI and advanced technology to develop tools that help producers become more profitable and efficient.

InFarm

infarm

The Indian Farmers Movement (Infarm) will organise a two-week long agricultural and educational fest at Kodenchery from December 26. A reception committee has also been constituted to make the ...

Infarm fest begins on December 26

Source: Clayton County Register INFARM, once Europe's largest vertical farming company, has filed for bankruptcy in the Netherlands as of September 19. The company ceased operations in key ...

Agrowatch Weekly #3 - InFarm Declares Bankruptcy, Syngenta & Corteva Announces Agtech Partnerships & Much More

Vertical farming company Infarm has been “declared bankrupt” in the Netherlands, according to official documents. The news comes after the company closed operations across its key markets in ...

Infarm’s Dutch branch declared bankrupt

Elsewhere, Tikehau Capital invested in a biologicals startup, vertical farming company Infarm was declared bankrupt, and agrifoodtech layoffs continued.

The Week in AgriFoodTech: Paine Schwartz launches $1.7bn agrifood fund, Cricket One unveils new plant, Infarm ‘declared bankrupt’

An Adventure Amidst the Industrial Revolution

249 Stories To Learn About Web Development

An Inquiry into the Nature and Causes of the Wealth of Nations: Book III - Chapter III

The Monster Hunter series has found massive success in recent years. The latest release from Capcom, Monster Hunter Rise, has become part of the company’s . This feat is only more impressive when considering the game only released on one platform so far, the Nintendo Switch. With MH Rise set to release on PC by next year, the future of the series is all but guaranteed to be bright. top 10 best-selling games At least, that’s what the optimists would tell you. On the other hand, Monster Hunter Rise has garnered its fair share of criticisms from players, especially long-time fans of the series. The main problem, for them, isn’t that something is wrong with the game itself - it’s about how it compares to other Monster Hunter games. As someone who’s been following and playing Monster Hunter Rise since its release, I’ve noted more than a few of these complaints. The most common ones I’ve seen involve the game lacking content or being too easy - these complaints even made their way into official reviews of the game, such as the one made by . They might not be the most popular opinions of the game, but they were widespread enough to reach major outlets; I wouldn’t be surprised if at least one person took those opinions to heart. IGN France I consider myself a fan of the series, so I delved into Monster Hunter in order to see just how well-founded those complaints were. Rise’s very development is different from most other Monster Hunter games, and there’s definitely a fair share of differences between it and previous titles. For the sake of this article, I’ll be focusing mainly on Rise, but I’ll mention previous titles in the series as I go along, such as Monster Hunter 4, Monster Hunter Generations Ultimate, and Monster Hunter: World. To start with, I’ll be discussing Rise’s content, and whether it really is as lacking as some describe. Monster Variety Across the Series Image via official press release As the name implies, the biggest focus of the Monster Hunter series is...well, the monsters. They provide the challenge of the games, and beating them down is the only way you can really progress. Having a wide variety of monsters can provide a lot of substantial time within these games, but the numbers might not be as impressive if you look closely enough. Monster Hunter 4 Monster Hunter 4 has a roster of over 50 monsters. In comparison, Rise only has 46, including the special ‘Apex’ monsters featured in rampages. This seems to be a spot where 4 has Rise beat...at least, at first. Many of the different monsters in the older title are actually just alternate forms of existing monsters - a monster like a Yian Kut-Ku is considered different from a Blue Yian Kut-Ku, for example. If you cut down each game’s monster list to only include monsters that don’t share names, the roster is only one or two monsters away from being the same size. Monster Hunter World To add to this, Monster Hunter World, the most recent game before Rise, only has 36 monsters. It was the first game to redesign how Monster AI worked, but the roster pales in comparison to Rise - which carries over those changes and makes even more improvements. The Iceborne DLC available for Monster Hunter: World does bump up the roster considerably, increasing it to around 70, but that brings up an issue reflected in Monster Hunter Generations Ultimate. Monster Hunter Generations Ultimate Being the only other Monster Hunter title available on the Switch, Generations Ultimate boasts an impressive amount of monsters. Almost 100 monsters make up the count in that game, and even when discounting similar monsters, it still has the largest roster of any game in the series. There’s a good reason for this, however; it’s not actually the base game. Generations Ultimate is the “expanded” version of Generations, a title released on the 3DS. Many mainline games get their own expanded versions, bumping up not just the amount of monsters but also how much there is to do in the game. This is also why Iceborne provides such a large increase of monsters; that acts as Monster Hunter: World’s expansion. To put it simply, Monster Hunter Rise provides a number of unique monsters comparable to most normal games in the series. Following the recent pattern of releases, it seems likely that Rise will gain an expansion for itself - and that may very well increase its roster of monsters to heights greater than previous titles. As a side note, Monster Hunter Generations was conceived as a celebration of the series up to that point. With that in mind, it’s no wonder it has so many monsters! Content Beyond Monsters Image via unofficial Monster Hunter wiki Of course, while monsters might be a major focus of Monster Hunter, there is more to the series than meets the eye. Each game provides its own unique quests and features, with some of them changing the games pretty heavily. Generations Ultimate provides the most of these features, as one might expect, with a massive amount of quests and fights that are either new or brought over from previous titles. Without even regarding expansion titles, however, Monster Hunter: World and Monster Hunter 4 do boast a fair share of their own special features. In Monster Hunter 4, normal monsters could be ‘Frenzied’, a state they take on that increases their power and changes their behavior. This provided a further challenge to already existing monsters, as they were able to wield greater power and sometimes even different moves. World had a similar feature for monsters, where they would reach ‘Tempered’ states. Fighting these monsters would lead to increased rewards and even rare materials you couldn’t find otherwise. Rise doesn’t have anything similar to that...but in the end, this may not actually matter. Rise has its own special features, with one heavily-advertised feature being Rampage Quests. In these quests, you play through a defense mission, repelling monsters with weapons and installations before they can break into your village. At higher levels, this will give you rare materials and rewards needed to farm for charms, which provide some of the best gear in the game. It can get repetitive after a while, and you’ll be facing a lot of the same monsters in order to obtain that gear...but that’s not very different from how it worked in World, where only high-level monsters gave players the best materials. In fact, it’s even more freeform in Rise, as players have the option of keeping those materials from the rampage; instead, they can use lower-value materials from regular monsters to create charms. The reason why Rise seems to lack content is because of presentation. Instead of going after the same monsters in new states, hunters in Rise are encouraged to simply go after what they’ve fought before. This gives players a more ‘set your own goals’ mindset, rather than being tempted with special modifiers and unique materials that accomplish the same thing in the end. While others might find this to be a subjective opinion, I find that Rise’s content is perfectly comparable to - if not greater than - the content found in older games. There is more to this, however. The amount of content and incentives to play is different than how many hours one might put into a game. To understand why some players find themselves putting less hours into Rise than previous titles, it’s important to understand the differences between the two. From that, a point can be made about the final complaint against Monster Hunter Rise - difficulty. How Rise Differs from Other Monster Hunter Titles Image via GamesRadar Monster Hunter Rise features the same overall formula as other games in the series. You hunt monsters, you make gear from those monsters, and you use that gear to fight stronger monsters. Rise happens to bring over some extra changes from Monster Hunter: World, such as the ability to use items while moving and the lack of loading screens. With these changes, Rise plays almost like an alternative version of World - with some alterations, that is. Where Rise differs most is the addition of two features. The first is the Wirebug, which acts as an addition to the player’s normal moveset. Hunters are able to vault through the air and climb up walls using the Wirebug, and certain weapons use its abilities to perform counterattacks and powerful moves. The second addition is the Palamute, a dog-like companion you can ride to traverse areas with greater speed. These additions are where the complaints regarding difficulty stem from, I believe. Wirebugs can allow players to more quickly recover and react to monster blows, and Palamutes provide players an extremely fast way to travel within each locale. Much of the difficulty from older games came from how much slower players were; it could take a very long time to recover from attacks, use items, or even just walk across a map. Rise gives players strength and speed almost unheard of in the series at the time. As a result, monsters are beaten much sooner, danger is a lot less apparent, and the overall challenge simply isn’t as apparent. That is a sentiment I disagree with, however. After all, it wasn’t just the players that have gotten stronger and faster. How “Difficulty” Has Changed Throughout the Monster Hunter Series Image via Reno Gazette Journal Older Monster Hunter titles, especially those before Monster Hunter: World, did not have very great enemy AI. Monsters would struggle to turn their bodies, different species would share animations and entire movesets, and even the toughest monsters were only really difficult due to some very strange hitboxes in their attacks. World completely redesigned how monsters acted during fights, but many of them still had very limited - and similar - movesets. Monster Hunter Rise, on the other hand, takes this redesign even further. There are still obvious bits of similarity, but even the most familiar fights are taken in brand-new directions. A good example of this can be found in the monsters Rathian and Rathalos - they’re almost the exact same species, and they’ve shared most of their moves with each other throughout the series. Rise keeps the general moveset for Rathian the same, but Rathalos gains a whole plethora of attacks. It will use its flight to combine a plethora of attacks, from fireballs to claw swipes, in manners that previous Monster Hunter titles wouldn’t even dream of doing. This is where Rise’s difficulty comes from - not just how players are meant to fight against monsters, but the monsters themselves. Instead of learning animation transitions and specific hitbox locations, players are meant to learn how a monster actually fights. The actual challenge between each monster becomes more substantial, even if the game provides a lot of tools to make fights easier. You might not struggle as much as in the rest of the series, but if you do end up failing, it’s not because of any faulty mechanics or broken hitboxes; it’s because you simply need to improve your skills. The Shortcomings of Monster Hunter Rise Image via Official Monster Hunter Twitter My statements so far have been in defense of Rise, confronting each complaint against it and offering my own arguments as to why those complaints might be short-sighted. However, there is a strong case to be made in favor of those complaints when taking two things into account: time, and pricing. Rise is still a very recent title in the Monster Hunter series, and it only reached its current state after numerous game updates. Upon release, the game featured fewer monsters and didn’t even allow players to increase their rank. Such an issue is irrelevant now, but it brings the game’s update cycle into consideration, especially when referencing other titles. Older games feature Event Quests that were slowly added into each title across a manner of weeks or months. Rise features Event Quests, as well, but the number pales in comparison to those featured in World, Generations Ultimate, or even 4. Because of these games being older, their price is much lower, as well. As of right now, without any sales, Monster Hunter: World can be purchased on the Steam store alongside its expansion for about the same price as Monster Hunter Rise. Because of the included expansion, a newcomer to the series would be getting way more out of the purchase of World and Iceborne, including much more content and much greater challenge. Of course, this issue will eventually be resolved in the same way it was introduced - time. Rise continues to get new event quests every week, and it wouldn’t be a stretch to assume that the eventual release of an expansion for Rise would lower the base game’s price. Players who decide to wait for that time to come will get quite a lot of bang for their buck, and players who don’t wait still have some great things to look forward to. Final Verdict: What Rise Provides for Newcomers and Veterans Image via Official Monster Hunter Twitter I would argue that Monster Hunter Rise is most definitely worth grabbing, even when considering older games in the series. Newcomers to the franchise will find a less frustrating experience with Rise overall, with the numerous tools at their disposal sure to keep them interested throughout the whole game. And while the search for higher difficulty might not be completely answered for veterans, the new monsters - and alterations to existing ones - will provide a satisfying challenge for many long-time hunters. But don’t think you should consider Rise in place of older games. Each title offers its own unique experience, after all. If you have similar complaints against Rise even after playing through it, you may very well find a lot of merit in games like World, Generations Ultimate, or 4. In the end, it’s all still Monster Hunter - as long as you enjoy that, there will be plenty of hunts for you! Relevant Sources: Monster Hunter Rise Is About To Become Capcom's Best-Selling Single-Platform Game - Nintendo Life Monster Hunter Rise Test: The Price of Accessibility (ign.com) Monster Hunter: World Iceborne 15.10 Removes Event Quest Time Limits (gameskinny.com) MONSTER HUNTER RISE[Update Information] Monster Hunter: World on Steam (steampowered.com) MONSTER HUNTER RISE for Nintendo Switch - Nintendo Game Details

Is Monster Hunter Rise Better or Worse than Capcom’s Previous Installments?

High APY won’t last long, Defi + upgraded DAO is the true future The farming mania Have you enjoyed the yield farming in the last couple of days? The price of COMP has jumped from $30 to $340 in just a week. And by supplying token assets on Compound, the lenders(of some tokens) could earn 20%+ APY (not counting the incentives COMP yet). On Saturday, June 20, Compound has overtaken MakerDAO to become the largest decentralized finance protocol by total value locked according to data from DeFi Pulse. There is over $580M in cryptocurrency locked on Compound compared to $470M on MakerDAO. Source: The Block Behind the halo It happened so quickly that people on Twitter kept comparing it with the ICO scam. Not to judge for that yet, let’s understand the mechanism of how Compound works first. In a nutshell, it’s called “Liquidity Mining.” The more people borrow or lend, the more rewards(COMP token) they can “farm,” which increases the liquidity of the financing and the size of the balance sheet. It’s not different from that you save money to the bank and earn the interest rate (it’s very low). However, on Compound, borrowers are willing to pay a lot more, because they think they can make more money with what they borrow from you than what they have to pay you. How does it work? They are getting paid to borrow. When they borrow from you on Compound, they are getting COMP Token as a sort of cashback. The more they borrow, the more COMP Token they get. At this time, the cashback(big juicy subsidies) is actually worth more than the cost of the fees to borrow, which means people are incentivized to borrow just to “farm” the cashback. As liquidity miners are compensated for both lending and borrowing, the optimal strategy is to lend the highest interest rate asset, borrow as much as they can against their tokens and then add the remaining assets back into the lending pool. So, People are borrowing funds to “farm” COMP, then using the same funds out to lend, increasing their yield (COMP reward) even further. As a result, now they can earn 100% APY instead of the 0.01%-1.00% your banks are offering you, which is a 100–10,000X increase. What’s going to happen next? While almost everyone agrees that when something becomes too good to be true, it either won’t be sustainable or won’t exist at all. Since you can lend, borrow, then lend and borrow again, just like using leverages to do the arbitrage, it has the material risks of liquidation, especially the APY rate is fluctuating severely. And actually, it has already happened in some users’ accounts. Do you remember the Trans-Fee Mining(the more you trade, the more FT you can farm) introduced by Fcoin? Within a few weeks after launching the incentive program, FCoin’s 24-hour trading volume reached a whopping $17B on June 26, 2018 (3 times the total trading volume of other top 10 exchanges). However, the surge did not last long. As reported on CoinMarketCap, FCoin’s 24-hour trading volume dropped to $0.16B on August 12, 2018. It collapsed when people started to dump FT, and it was a speeding-up vicious cycle when people saw the FT price was dropping. So, same here, does Compound give its users a reason to hold COMP instead of selling it? There would be a tricky point that when people dump COMP token to secure their profit, the price of the COMP token keeps dropping and becomes unattractive or hard to compensate for the difference between the borrowing and lending cost, then people start to withdraw their token assets from the platform, and the tragedy begins. As an “insider” of the blockchain and crypto field, few of my friends borrow or lend from these Defi platforms, or holding COMP or MKR tokens. It might be some selection bias, but it reminds me that the size of this Defi market is actually limited to the level of speculative activities in the crypto industry. If there are no breakthrough use scenarios to bring in new users, the imagination that the existing user base will support a new giant will also be limited. It’s just another zero-sum game. Is Defi a Pseudo-proposition？ Definitely no. But I don’t think Defi will replace the Cefi. On the contrary, they will co-exist in the long term. Like the day and night, Defi and Cefi will complement each other to serve different business/people, and they will have interactions, like the stable coin/Fiat and KYC. Defi is still at its early stage. If it is only borrowing or lending to gain arbitrage, it will just become another alternative investment. How could it change the world as it intends to be? In the physical world, companies create value, and that’s what the stocks, bonds, or alternative investment products are pegging. In the decentralized world, distributed collaborations constitute the Decentralized Autonomous Organization (DAO). As the counterpart of the traditional company and as a reasonable logic, Defi should peg to the value creation entity- DAO. Different DAOs are set to accomplish various goals, so borrowing and lending is only a small part of the whole landscape. As an organization lives on the blockchain and internet, all the activities that the distributed community members carry out (like finish a task, conduct a transaction or share knowledge) will be recorded and tracked in almost real-time, which offers the edge for Defi to be different from traditional centralized finance. So, not as in the traditional finance scenario that the information asymmetry forced investors to gauge the past business data and bet on the future almost blindly, or invest in a project to try their luck, Defi investors or Defi service providers could scrutinize the “atomic” layer (in theory), with their investment or alternative investment pegged to the progress of the collaboration or the project in the DAO. Too complicated or not real? No worries, DAO has the capability to backbone that, and now almost all the Defi projects are leveraging the DAO’s governance structure. However, the DAO’s governance protocols need to be upgraded, voting and budget allocating (which most governance protocols are focusing on) are not enough, because that is only the beginning of the whole collaboration process in the DAO. A systematic framework should be built up to regulate the distributed collaborative relationship, to manage the collaboration, and to protect the interests of all the collaborators. With the framework, Defi investors or service providers can peg to the project’s real value, controlling their own destiny while also providing the liquidity for the project, which is a win-win. So, to conclude, Defi + DAO (with updated governance framework) will become the real organic field to farm. Contributed by Kevin Liu

Yield Farming Cannot Be The Organic Future of Blockchain

Authors:
(1) Yagci, Nurce, HAW Hamburg, Germany & nurce.yagci@haw-hamburg.de;
(2) Sünkler, Sebastian, HAW Hamburg, Germany & sebastian.suenkler@haw-hamburg.de;
(3) Häußler, Helena, HAW Hamburg, Germany & helena.haeuessler@haw-hamburg.de;
(4) Lewandowski, Dirk, HAW Hamburg, Germany & dirk.lewandowski@haw-hamburg.de. Table of Links Abstract and Introduction Literature Review Objectives and Research Questions Methods Results Discussion Conclusion, Research Data, Acknowledgments, and References OBJECTIVES AND RESEARCH QUESTIONS Our study addresses the following research questions: 1. Do top results from alternative search engines differ from Google's in regard to the number of unique sources? 2. Do top results from alternative search engines differ from Google's in regard to top sources? 3. Do top results from alternative search engines differ from Google's in regard to source concentration, i.e., are results distributed over more or fewer sources in different search engines? To answer these questions, we selected three alternative search engines to compare to Google. Aside from Bing, which is the biggest competitor to Google, we chose DuckDuckGo and Metager. DuckDuckGo uses results from Bing, Yahoo, and Yandex, and has the added benefit of advanced privacy settings such as non-personalized results. Metager, a German meta-search engine, aggregates results from several search engines, including Bing and Yandex. As such, either search engine might provide a more varied set of sources. Since the alternative search engines each have an index different from Google's, the comparison will give insights into what domains are favored by Google and which domains are excluded. Also, we will gain insights into the differences in ranking between them. This paper is available on arxiv under CC 4.0 license. Authors: (1) Yagci, Nurce, HAW Hamburg, Germany & nurce.yagci@haw-hamburg.de; (2) Sünkler, Sebastian, HAW Hamburg, Germany & sebastian.suenkler@haw-hamburg.de; (3) Häußler, Helena, HAW Hamburg, Germany & helena.haeuessler@haw-hamburg.de; (4) Lewandowski, Dirk, HAW Hamburg, Germany & dirk.lewandowski@haw-hamburg.de. Authors: (1) Yagci, Nurce, HAW Hamburg, Germany & nurce.yagci@haw-hamburg.de; (2) Sünkler, Sebastian, HAW Hamburg, Germany & sebastian.suenkler@haw-hamburg.de; (3) Häußler, Helena, HAW Hamburg, Germany & helena.haeuessler@haw-hamburg.de; (4) Lewandowski, Dirk, HAW Hamburg, Germany & dirk.lewandowski@haw-hamburg.de. Table of Links Abstract and Introduction Abstract and Introduction Literature Review Literature Review Objectives and Research Questions Objectives and Research Questions Methods Methods Results Results Discussion Discussion Conclusion, Research Data, Acknowledgments, and References Conclusion, Research Data, Acknowledgments, and References OBJECTIVES AND RESEARCH QUESTIONS Our study addresses the following research questions: 1. Do top results from alternative search engines differ from Google's in regard to the number of unique sources? 2. Do top results from alternative search engines differ from Google's in regard to top sources? 3. Do top results from alternative search engines differ from Google's in regard to source concentration, i.e., are results distributed over more or fewer sources in different search engines? To answer these questions, we selected three alternative search engines to compare to Google. Aside from Bing, which is the biggest competitor to Google, we chose DuckDuckGo and Metager. DuckDuckGo uses results from Bing, Yahoo, and Yandex, and has the added benefit of advanced privacy settings such as non-personalized results. Metager, a German meta-search engine, aggregates results from several search engines, including Bing and Yandex. As such, either search engine might provide a more varied set of sources. Since the alternative search engines each have an index different from Google's, the comparison will give insights into what domains are favored by Google and which domains are excluded. Also, we will gain insights into the differences in ranking between them. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

How Different Search Engines Display and Prioritize Information

Entrepreneurs, Journalists, Researchers: The Ethereum Influencers to Follow to Stay Informed!

This paper is available on arxiv under CC 4.0 license. Authors: (1) Pietro Ghezzi, Brighton & Sussex Medical School, Falmer, Brighton, UK; (2) Peter G Bannister, Brighton & Sussex Medical School, Falmer, Brighton, UK; (3) Gonzalo Casino, Communication Department, Pompeu Fabra University, Barcelona, Spain and Iberoamerican Cochrane Center, Barcelona, Spain; (4) Alessia Catalani, Department of Biomolecular Sciences, University of Urbino Carlo Bo, Urbino, PU, Italy; (5) Michel Goldman, Institute for Interdisciplinary Innovation in healthcare (I3h), Université libre de Bruxelles; (6) Jessica Morley, Oxford Internet Institute, University of Oxford, Oxford, UK; (7) Marie Neunez, Institute for Interdisciplinary Innovation in healthcare (I3h), Université libre de Bruxelles; (8) Andreu Prados, Communication Department, Pompeu Fabra University, Barcelona, Spain, Iberoamerican Cochrane Center, Barcelona, Spain, Department of Biomolecular Sciences, University of Urbino Carlo Bo, Urbino, PU, Italy, Institute for Interdisciplinary Innovation in healthcare (I3h), Université libre de Bruxelles, Oxford Internet Institute, University of Oxford, Oxford, UK, and Blanquerna School of Health Sciences, Universitat Ramon Llull, Barcelona, Spain; (9) Mariarosaria Taddeo, Oxford Internet Institute, University of Oxford, Oxford, UK, Blanquerna School of Health Sciences, Universitat Ramon Llull, Barcelona, Spain, and The Alan Turing Institute, London, UK; (10) Tania Vanzolini, Department of Biomolecular Sciences, University of Urbino Carlo Bo, Urbino, PU, Italy; (11) Luciano Floridi, Oxford Internet Institute, University of Oxford, Oxford, UK, Blanquerna School of Health Sciences, Universitat Ramon Llull, Barcelona, Spain, and The Alan Turing Institute, London, UK. Table of Links Abstract and Introduction Methods Web search and content analysis Results Discussion Conclusion and References Table 1. Example of classification of webpages Table 2. Percentage of vaccine-negative webpages in different SERPs Table 2. Percentage of vaccine-negative webpages in different SERPs Total of 30 webpages for each SERP. Cells are colour coded: median (20%), yellow; maximum (53%), red; minimum (0%), green. *Significantly different from Google.com by two-tailed Fisher’s test corrected for multiple comparison using the method of Benjamini, Krieger and Yekutieli at a false discovery rate of 5%. #significantly different from the localized version of Google.

Online Information of Vaccines: Table 2. Percentage of Vaccine-negative Webpages in Different SERPs

This paper is available on arxiv under CC 4.0 license. Authors: (1) Pietro Ghezzi, Brighton & Sussex Medical School, Falmer, Brighton, UK; (2) Peter G Bannister, Brighton & Sussex Medical School, Falmer, Brighton, UK; (3) Gonzalo Casino, Communication Department, Pompeu Fabra University, Barcelona, Spain and Iberoamerican Cochrane Center, Barcelona, Spain; (4) Alessia Catalani, Department of Biomolecular Sciences, University of Urbino Carlo Bo, Urbino, PU, Italy; (5) Michel Goldman, Institute for Interdisciplinary Innovation in healthcare (I3h), Université libre de Bruxelles; (6) Jessica Morley, Oxford Internet Institute, University of Oxford, Oxford, UK; (7) Marie Neunez, Institute for Interdisciplinary Innovation in healthcare (I3h), Université libre de Bruxelles; (8) Andreu Prados, Communication Department, Pompeu Fabra University, Barcelona, Spain, Iberoamerican Cochrane Center, Barcelona, Spain, Department of Biomolecular Sciences, University of Urbino Carlo Bo, Urbino, PU, Italy, Institute for Interdisciplinary Innovation in healthcare (I3h), Université libre de Bruxelles, Oxford Internet Institute, University of Oxford, Oxford, UK, and Blanquerna School of Health Sciences, Universitat Ramon Llull, Barcelona, Spain; (9) Mariarosaria Taddeo, Oxford Internet Institute, University of Oxford, Oxford, UK, Blanquerna School of Health Sciences, Universitat Ramon Llull, Barcelona, Spain, and The Alan Turing Institute, London, UK; (10) Tania Vanzolini, Department of Biomolecular Sciences, University of Urbino Carlo Bo, Urbino, PU, Italy; (11) Luciano Floridi, Oxford Internet Institute, University of Oxford, Oxford, UK, Blanquerna School of Health Sciences, Universitat Ramon Llull, Barcelona, Spain, and The Alan Turing Institute, London, UK. Table of Links Abstract and Introduction Methods Web search and content analysis Results Discussion Conclusion and References Table 1. Example of classification of webpages Table 2. Percentage of vaccine-negative webpages in different SERPs Table 1. Example of classification of webpages Positive stance on vaccines https://www.historyofvaccines.org/content/articles/do-vaccines-cause-autism https://www.nhs.uk/news/medication/no-link-between-mmr-and-autism-major-study-finds/ https://www.nhs.uk/conditions/vaccinations/mmr-vaccine/ https://www.autism.org.uk/get-involved/media-centre/position-statements/mmr-vaccine.aspx https://kidshealth.org/en/parents/autism-studies.html https://vaccine-safety-training.org/mmr-vaccine-increases.html https://www.nejm.org/doi/full/10.1056/NEJMoa021134 https://www.ncbi.nlm.nih.gov/pubmed/30986133 https://www.ncbi.nlm.nih.gov/pubmed/15366972 Negative stance on vaccines https://www.thelancet.com/journals/lancet/article/PIIS0140673605756968/fulltext www.whale.to/vaccine/vaccine_autism_proven.html www.vaccineriskawareness.com/Infant-Vaccines-Produce-Autism-Symptoms-In-Primates https://leftbrainrightbrain.co.uk/2014/07/17/more-of-that-vaccineautism-research-that-doesnt-exist/ edition.cnn.com/2011/HEALTH/01/05/autism.vaccines/index.html https://www.coasttocoastam.com/show/2018/02/21 nocompulsoryvaccination.com/2014/08/22/vaccine-autism-cover-up/ https://www.newswars.com/vaccine-autism-questioned-by-doctor-congressman-elect/ vaxtruth.org/2011/08/vaccines-do-not-cause-autism/ Neutral stance on vaccines https://www.physicsforums.com/threads/vaccine-and-autism-link.880852/ www.discovermagazine.com/2009/jun/06-why-does-vaccine-autism-controversy-live-on https://www.ecso.org/news/autism-charity-founder-anti-vaccination-campaigner/ https://www.theguardian.com/society/2019/jun/01/professor-who-links-vaccines-to-autismfunded-through-university-portal https://www.independent.co.uk/news/world/americas/trump-vaccines-autism-links-anti-vaxxerus-president-false-vaccine-a8331836.html https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2376879/

Online Information of Vaccines: Table 1. Example of Classification of Webpages

This paper is available on arxiv under CC 4.0 license. Authors: (1) Pietro Ghezzi, Brighton & Sussex Medical School, Falmer, Brighton, UK; (2) Peter G Bannister, Brighton & Sussex Medical School, Falmer, Brighton, UK; (3) Gonzalo Casino, Communication Department, Pompeu Fabra University, Barcelona, Spain and Iberoamerican Cochrane Center, Barcelona, Spain; (4) Alessia Catalani, Department of Biomolecular Sciences, University of Urbino Carlo Bo, Urbino, PU, Italy; (5) Michel Goldman, Institute for Interdisciplinary Innovation in healthcare (I3h), Université libre de Bruxelles; (6) Jessica Morley, Oxford Internet Institute, University of Oxford, Oxford, UK; (7) Marie Neunez, Institute for Interdisciplinary Innovation in healthcare (I3h), Université libre de Bruxelles; (8) Andreu Prados, Communication Department, Pompeu Fabra University, Barcelona, Spain, Iberoamerican Cochrane Center, Barcelona, Spain, Department of Biomolecular Sciences, University of Urbino Carlo Bo, Urbino, PU, Italy, Institute for Interdisciplinary Innovation in healthcare (I3h), Université libre de Bruxelles, Oxford Internet Institute, University of Oxford, Oxford, UK, and Blanquerna School of Health Sciences, Universitat Ramon Llull, Barcelona, Spain; (9) Mariarosaria Taddeo, Oxford Internet Institute, University of Oxford, Oxford, UK, Blanquerna School of Health Sciences, Universitat Ramon Llull, Barcelona, Spain, and The Alan Turing Institute, London, UK; (10) Tania Vanzolini, Department of Biomolecular Sciences, University of Urbino Carlo Bo, Urbino, PU, Italy; (11) Luciano Floridi, Oxford Internet Institute, University of Oxford, Oxford, UK, Blanquerna School of Health Sciences, Universitat Ramon Llull, Barcelona, Spain, and The Alan Turing Institute, London, UK. Table of Links Abstract and Introduction Methods Web search and content analysis Results Discussion Conclusion and References Table 1. Example of classification of webpages Table 2. Percentage of vaccine-negative webpages in different SERPs Corresponding author: Professor Pietro Ghezzi, Brighton and Sussex Medical School, Trafford Centre, University of Sussex, Falmer, Brighton, BN19RY, UK. Email: p.ghezzi@bsms.ac.uk; phone: +44-(0)1273-873112 Abstract The fact that Internet companies may record our personal data and track our online behaviour for commercial or political purpose has emphasized aspects related to online privacy. This has also led to the development of search engines that promise no tracking and privacy. Search engines also have a major role in spreading low-quality health information such as that of anti-vaccine websites. This study investigates the relationship between search engines’ approach to privacy and the scientific quality of the information they return. We analyzed the first 30 webpages returned searching “vaccines autism” in English, Spanish, Italian and French. The results show that “alternative” search engines (Duckduckgo, Ecosia, Qwant, Swisscows and Mojeek) may return more anti-vaccine pages (10-53%) than Google.com (0%). Some localized versions of Google, however, returned more anti-vaccine webpages (up to 10%) than Google.com. Our study suggests that designing a search engine that is privacy savvy and avoids issues with filter bubbles that can result from user-tracking is necessary but insufficient; instead mechanisms should be developed to test search engines from the perspective of information quality (particularly for health-related webpages), before they can be deemed trustworthy providers of public health information. Introduction The World Health Organization lists vaccine hesitancy as one of the top ten threats to global health in 2019 1, requiring ongoing global monitoring. Despite the fact that the 1998 study which incorrectly suggested that the MMR vaccine could cause autism in children and prompted antivaccine beliefs 2 has now been discredited, misinformation, and indeed, disinformation, about vaccines continues to be published on the Internet perpetuating such beliefs. It has been suggested that this misinformation plays a role in the current low uptake of vaccines in developed countries 3. Understanding whether this is actually the case, and how to address this issue, is crucial as web-based sources of health information may be instrumental to solve the sustainability challenge currently facing health systems across the globe. The accuracy of information provided by a website is a key indicator of its overall information quality (IQ). The broader aspects of IQ have been the subject of many studies 5, but health IQ and trustworthiness of the sources have only partially been characterized 6. Studies looking at the influence of variations in eHealth literacy levels 7 and trust in different sources of online health information2 indicate that the relationship is not linear in all cases, i.e. higher health IQ does not result automatically in higher perceived levels of trustworthiness. For example, in a study by Chen et al. 8, 618 people were recruited to complete a survey which tested their eHealth literacy level, asked them to identify which of 25 sources of health information they used, and how much they trusted each source. The study showed that people with lower eHealth literacy were less likely to trust medical websites (typically higher IQ) and more likely to trust social media, blogs, and celebrity webpages (typically lower IQ). This might seem a spurious result, were it not for the fact that research has shown that those with high eHealth literacy assess more accurately the credibility and relevance of online health information, whereas those with low eHealth literacy often struggle to locate and understand eHealth information 9. This difficulty lowers their self-efficacy 10, distorts their perception of source credibility, and impacts negatively perceived trustworthiness 9, ultimately creating a need for individuals with low eHealth literacy to find an alternative means of determining trustworthiness in online sources of information. One such alternative is social endorsement. Visible social endorsement, e.g. ‘likes,’ 11 enables those with low eHealth literacy to determine trust based on the bandwagon heuristic and assume that, if the source has already been deemed valid by others, then it is safe for them to trust it too 10,12. Traditional medical websites afford those with low eHealth literacy no such alternative means of determining credibility and trust. This suggests that those who are more vulnerable to the real-world effects of both disinformation and misinformation (e.g. declining to vaccinate their children) are more likely to rely on online sources with lower health IQ, which are more prone to spread such inflammatory and inaccurate information. Personalisation of online search results may favour this phenomenon and lead to a vicious cycle where the more one searches and reads dis- and mis-information about vaccines the less one finds and reads scientific information on the same topic. At the same time, there are increasing concerns about the privacy risks associated with Internet search engines storing potentially sensitive and private health information contained within users search histories, combining it with additional information collected for tracking purposes, and using these data for commercial or other purposes 13,14. This creates a public push back against the idea that search engines or public health providers should interfere in the results people see when searching for health information online. This makes it hard to address concerns about health disinformation/misinformation in a way that is at least socially acceptable if not ideally preferable 15. The UK’s National Health Service (NHS) discovered this when its announcement of Amazon Alexa responding with guaranteed high-IQ content from NHS.UK to user voice queries such as “Alexa, how do I treat a migraine,” resulted in a public outcry over privacy infringement 16. This raises the question whether, in the context of online vaccination information, it is possible at all to balance concerns about user privacy and IQ. The following pages seek to lay the foundations for answering this question by reporting on the methodology for, and results of, a study concerning the following research question: “What is the current relationship between search engines’ approach to privacy and the scientific quality of the information they return?”.

Online Information of Vaccines: Abstract and Introduction

This paper is available on arxiv under CC 4.0 license. Authors: (1) Pietro Ghezzi, Brighton & Sussex Medical School, Falmer, Brighton, UK; (2) Peter G Bannister, Brighton & Sussex Medical School, Falmer, Brighton, UK; (3) Gonzalo Casino, Communication Department, Pompeu Fabra University, Barcelona, Spain and Iberoamerican Cochrane Center, Barcelona, Spain; (4) Alessia Catalani, Department of Biomolecular Sciences, University of Urbino Carlo Bo, Urbino, PU, Italy; (5) Michel Goldman, Institute for Interdisciplinary Innovation in healthcare (I3h), Université libre de Bruxelles; (6) Jessica Morley, Oxford Internet Institute, University of Oxford, Oxford, UK; (7) Marie Neunez, Institute for Interdisciplinary Innovation in healthcare (I3h), Université libre de Bruxelles; (8) Andreu Prados, Communication Department, Pompeu Fabra University, Barcelona, Spain, Iberoamerican Cochrane Center, Barcelona, Spain, Department of Biomolecular Sciences, University of Urbino Carlo Bo, Urbino, PU, Italy, Institute for Interdisciplinary Innovation in healthcare (I3h), Université libre de Bruxelles, Oxford Internet Institute, University of Oxford, Oxford, UK, and Blanquerna School of Health Sciences, Universitat Ramon Llull, Barcelona, Spain; (9) Mariarosaria Taddeo, Oxford Internet Institute, University of Oxford, Oxford, UK, Blanquerna School of Health Sciences, Universitat Ramon Llull, Barcelona, Spain, and The Alan Turing Institute, London, UK; (10) Tania Vanzolini, Department of Biomolecular Sciences, University of Urbino Carlo Bo, Urbino, PU, Italy; (11) Luciano Floridi, Oxford Internet Institute, University of Oxford, Oxford, UK, Blanquerna School of Health Sciences, Universitat Ramon Llull, Barcelona, Spain, and The Alan Turing Institute, London, UK. Table of Links Abstract and Introduction Methods Web search and content analysis Results Discussion Conclusion and References Table 1. Example of classification of webpages Table 2. Percentage of vaccine-negative webpages in different SERPs Conclusion Our analysis shows that while it may well be technically possible to design a search engine that manages to balance privacy-preservation with the promotion of high IQ material, this is currently not the case. The current relationship between privacy-preserving design features of search engines and the IQ of the results they return is inverse (although not proportionally). In instances where this can have a negative impact on public health, as in the example we have provided of the promotion of anti-vaccine misinformation, not intervening to alter the design of the algorithm even if this means sacrificing some degree of user privacy - can lead to severe harm for a large population of user and is, therefore, unethical. Designing a search engine that is privacy savvy and avoids issues with filter bubbles that can result from user-tracking may be a good thing, like designing a car with an engine that does not pollute and is inexpensive to run, and designers should seek to balance the different aspects of search engine design highlighted in Figure 3. However, if the brakes in an environmentally-designed car do not work, the car is unsafe and this negates the positive ethical decisions made by the designers. In a car this is a highly unlikely design outcome, as a car has to undergo several rounds of testing by regulatory agencies before being allowed on the market. This is not yet the case for search engines, which are only regulated from the perspective of data protection – which is primarily interpreted as data security rather than data privacy. Our study suggests that this is necessary but insufficient, and instead mechanisms should be developed to test search engines from the perspective of IQ, (particularly for YMYL webpages) before they can be deemed trustworthy providers of public health information. References 1. World Health Organization. Ten threats to global health in 2019. 2019. https://www.who.int/emergencies/ten-threats-to-global-health-in-2019 (archived at https://web.archive.org/web/20190627111040/https://www.who.int/emergencies/ten-threats-toglobal-health-in-2019) (accessed 27/06/2019 2. Wakefield AJ, Murch SH, Anthony A, et al. Ileal-lymphoid-nodular hyperplasia, non-specific colitis, and pervasive developmental disorder in children. Lancet 1998; 351(9103): 637-41. 3. Stahl JP, Cohen R, Denis F, et al. The impact of the web and social networks on vaccination. New challenges and opportunities offered to fight against vaccine hesitancy. Med Mal Infect 2016; 46(3): 117-22. 4. Martin GP, Sutton E, Willars J, Dixon-Woods M. Frameworks for change in healthcare organisations: a formative evaluation of the NHS Change Model. Health Serv Manage Res 2013; 26(2-3): 65-75. 5. Illari P, Floridi L. Information quality, data and philosophy. The Philosophy of Information Quality: Springer; 2014: 5-23. 6. Al-Jefri M, Evans R, Uchyigit G, Ghezzi P. What is health information quality? Ethical dimension and perception by users. Frontiers in medicine 2018; 5. 7. Norman CD, Skinner HA. eHEALS: The eHealth Literacy Scale. J Med Internet Res 2006; 8(4): e27. 8. Chen X, Hay JL, Waters EA, et al. Health Literacy and Use and Trust in Health Information. J Health Commun 2018; 23(8): 724-34. 9. Paige SR, Krieger JL, Stellefson ML. The Influence of eHealth Literacy on Perceived Trust in Online Health Communication Channels and Sources. J Health Commun 2017; 22(1): 53-65. 10. Kim J, Park HA. Development of a health information technology acceptance model using consumers' health behavior intention. J Med Internet Res 2012; 14(5): e133. 11. Borah P, Xiao X. The Importance of 'Likes': The Interplay of Message Framing, Source, and Social Endorsement on Credibility Perceptions of Health Information on Facebook. J Health Commun 2018; 23(4): 399-411. 12. Huo C, Zhang M, Ma F. Factors influencing people’s health knowledge adoption in social media: the mediating effect of trust and the moderating effect of health threat. Library Hi Tech 2018; 36(1): 129-51. 13. Chiou L, Tucker C. Search engines and data retention: Implications for privacy and antitrust: National Bureau of Economic Research, 2017. 14. Esteve A. The business of personal data: Google, Facebook, and privacy issues in the EU and the USA. International Data Privacy Law 2017; 7(1): 36-47. 15. Floridi L, Taddeo M. What is data ethics? : The Royal Society; 2016. 16. Moskvitch K. Amazon’s Alexa deal with the NHS can help patients but at what cost? Wired, 2019. (accessed 8/11/2019). 17. Arif N, Al-Jefri M, Bizzi IH, et al. Fake News or Weak Science? Visibility and Characterization of Antivaccine Webpages Returned by Google in Different Languages and Countries. Front Immunol 2018; 9: 1215. 18. Maki A, Evans R, Ghezzi P. Bad news. Analysis of the quality of information on influenza prevention returned by Google in English and Italian. Frontiers in Immunology 2015; 6:616. doi: 10.3389/fimmu.2015.00616. 19. Fleiss JL, Levin B, Paik MC. Statistical methods for rates and proportions. Hoboken, NJ, USA: John Wiley & Sons; 2013. 20. Economist T. Seek and you shall find. Google rewards reputable reporting, not left-wing politics. The Economist 2019; 431(9146): 85. 21. Taddeo M, Floridi L. The Debate on the Moral Responsibilities of Online Service Providers. Science and engineering ethics 2015: 1-29. 22. Mittelstadt BD, Allo P, Taddeo M, Wachter S, Floridi L. The ethics of algorithms: Mapping the debate. Big Data & Society 2016; 3(2): 2053951716679679. 23. Association for Computing Machinery. ACM Code of Ethics and Professional Conduct. 2018. https://www.acm.org/code-of-ethics (archived at https://web.archive.org/web/20190627114921/https://www.acm.org/code-of-ethics) (accessed 27/06/2019 2019). 24. Tufekci Z. Algorithmic harms beyond Facebook and Google: Emergent challenges of computational agency. Colo Tech LJ 2015; 13: 203. 25. Hasnas J. Teaching business ethics: The principles approach. Journal of Business Ethics Education 2013; 10: 275-304. 26. Macdonald C, Gavura S. Alternative Medicine and the Ethics Of Commerce. Bioethics 2016; 30(2): 77-84. 27. Taddeo M, Floridi L. The Debate on the Moral Responsibilities of Online Service Providers. Sci Eng Ethics 2016; 22(6): 1575-603. 28. Chigwedere P, Seage GR, 3rd, Gruskin S, Lee TH, Essex M. Estimating the lost benefits of antiretroviral drug use in South Africa. J Acquir Immune Defic Syndr 2008; 49(4): 410-5. 29. Ouyang Y. Student's death highlights gaps in China's health regulations. Lancet Oncol 2016; 17(6): 709. 30. Browne M, Thomson P, Rockloff MJ, Pennycook G. Going against the Herd: Psychological and Cultural Factors Underlying the 'Vaccination Confidence Gap'. PLoS One 2015; 10(9): e0132562. 31. Bean SJ. Emerging and continuing trends in vaccine opposition website content. Vaccine 2011; 29(10): 1874-80.

Online Information of Vaccines: Conclusion and References

This paper is available on arxiv under CC 4.0 license. Authors: (1) Pietro Ghezzi, Brighton & Sussex Medical School, Falmer, Brighton, UK; (2) Peter G Bannister, Brighton & Sussex Medical School, Falmer, Brighton, UK; (3) Gonzalo Casino, Communication Department, Pompeu Fabra University, Barcelona, Spain and Iberoamerican Cochrane Center, Barcelona, Spain; (4) Alessia Catalani, Department of Biomolecular Sciences, University of Urbino Carlo Bo, Urbino, PU, Italy; (5) Michel Goldman, Institute for Interdisciplinary Innovation in healthcare (I3h), Université libre de Bruxelles; (6) Jessica Morley, Oxford Internet Institute, University of Oxford, Oxford, UK; (7) Marie Neunez, Institute for Interdisciplinary Innovation in healthcare (I3h), Université libre de Bruxelles; (8) Andreu Prados, Communication Department, Pompeu Fabra University, Barcelona, Spain, Iberoamerican Cochrane Center, Barcelona, Spain, Department of Biomolecular Sciences, University of Urbino Carlo Bo, Urbino, PU, Italy, Institute for Interdisciplinary Innovation in healthcare (I3h), Université libre de Bruxelles, Oxford Internet Institute, University of Oxford, Oxford, UK, and Blanquerna School of Health Sciences, Universitat Ramon Llull, Barcelona, Spain; (9) Mariarosaria Taddeo, Oxford Internet Institute, University of Oxford, Oxford, UK, Blanquerna School of Health Sciences, Universitat Ramon Llull, Barcelona, Spain, and The Alan Turing Institute, London, UK; (10) Tania Vanzolini, Department of Biomolecular Sciences, University of Urbino Carlo Bo, Urbino, PU, Italy; (11) Luciano Floridi, Oxford Internet Institute, University of Oxford, Oxford, UK, Blanquerna School of Health Sciences, Universitat Ramon Llull, Barcelona, Spain, and The Alan Turing Institute, London, UK. Table of Links Abstract and Introduction Methods Web search and content analysis Results Discussion Conclusion and References Table 1. Example of classification of webpages Table 2. Percentage of vaccine-negative webpages in different SERPs Discussion The aim of this research was to investigate whether search engines focused on privacy preservation ranked webpages with higher and/or lower levels of IQ (interpreted as EBM based content i.e. vaccine-positive rather than vaccine-negative) differently to the primary commercial search engines. In turn, this answered the research question “What is the current relationship between search engines’ approach to privacy and the scientific quality of the information they return?”. The results indicate that currently privacy-enhancing search engines often give more visibility to webpages promoting vaccine hesitancy or with a clear anti-vaccine position than Google. This is in agreement with the findings from a recent study by the Economist which analyzed 175,000 news links returned by Google to demonstrate that the search engine’s algorithm favorstrustworthy publications 20. Reputation and trustworthiness are key factors included in Google’s ranking algorithm. In 2019, Google published search quality evaluation guidelines, 4 which define webpages containing information that may affect the users’ health or financial stability, as “your money your life” (YMYL) pages. These guidelines reveal that when rating YMYL webpages, Google looks at the three criteria of Expertise-Authority-Trustworthiness (EA-T) and states “High E-A-T information pages on scientific topics should be produced by people or organizations with appropriate scientific expertise and represent well-established scientific consensus on issues where such consensus exists”. Thus, in the case of Google, the bandwagon heuristic mechanism of assessing medical information credibility is working in favour of promoting IQ. This is perhaps because those responsible for driving up the ‘reputation’ of specific websites by, for example, linking to them, have higher levels of eHealth literacy and therefore act as pseudo-gatekeepers protecting those with lower eHealth literacy from poor IQ results by ensuring that websites providing inaccurate online health information have a low ranking in the search results. In the case of privacy-preserving engines this appears not to be working – potentially because they do not track ‘reputation’ factors, which could be seen as proxies for IQ and credibility, e.g. clicks and bounce-rate, and so have no gatekeepers (pseudo or otherwise) working to limit the circulation of misinformation. To our knowledge, the algorithms used by these ‘alternative’ search engines are not public - Qwant states that they use their own algorithms5 - but despite this, we found a large overlap between the SERP of Qwant and those of Bing and Ecosia. Likewise, Ecosia and Swisscows showed a similar overlap (70%) with Bing. This means that it is not possible to check what factors determine the outcome of the decision making processes of the search engines algorithms, and hence identify those elements that contribute to circulate misinformation. Even without being able to check the exact mechanisms for the overall poorer results, the results suggest that, currently, decisions made by search engines to prioritize privacypreservation may have a negative impact on the IQ of results returned to users in health contexts. The pledge to provide independent and unbiased results or not to promote or hide websites based on political or moral interests can be seen as ethically ambiguous, in view of the potential consequences of pointing to scientifically unsound health information. Medical ethics requires that patients give informed consent before treatment and must, therefore, be informed accurately of the risks and benefits associated with treatment, something that is not possible if the search engine provider used by an individual is returning results with low IQ. From this consequentialist perspective, it is inherently unethical choosing not to interfere with a search engine’s ranking algorithm to ensure ‘manually’ that results of higher IQ are prioritized, whilst those of lower IQ are suppressed. This builds on arguments already made in the wider literature about algorithm ethics 21,22. For example, the Association for Computing Machinery (ACM) code of ethics and professional conduct includes, as a general principle, “avoid harm” along with that of “respect privacy” 23, which aligns with the Hippocratic oath. Specifically, the implication that the current design of privacy-preserving search engine algorithms underestimates the need for evaluation of IQ fits into the wider discussion about algorithmic design and how to ensure design decisions are made to protect and incorporate key values such as IQ. It is necessary, but insufficient, to design search engine algorithms that index purely on ‘relevance’, they must also be designed to index on quality. The challenge lies in the ability to do this in a way that balances the need to accept different perspectives (particularly those that are rooted in different cultural, religious or social ideals) whilst also filtering for IQ. Supporters of such arguments, including the authors, note that this requirement necessitates making the workings of search engine algorithms more transparent 24 to ensure their ethical compliance. Providing information on vaccines that is based on misinformation or disinformation (including studies whose data or conclusions have been shown to be wrong) is a deceptive practice, which goes against the basic tenets of medical and business ethics 25. This is in line with those who argue that the promotion of ‘alternative medicine’ is unethical because it lacks evidence and transparency of clinical efficacy and should be considered ‘false advertising’ 26. Online service providers have a moral responsibilities to ensure that users access health information that is scientifically validate 27. Misinformation and disinformation concerning healthcare circulating on the internet can have severe consequences and lead to widespread harm. Consider the example of South African president Mbeki, who delayed introducing anti-retroviral drugs in favor of alternative medicine based on information obtained from HIV-denialist internet websites, a decision estimated to have resulted in over 300,000 deaths 28. More recently, a cancer patient died in China after following an alternative, non-approved therapy they found using a search engine 29. In response Chinese authorities issued new rules that require search engines to provide “objective, fair and authoritative results”. Moral responsibility follows on the level of harm that misinformation and disinformation on healthcare may cause. It could be argued that there is a greater onus on Google – and other commercial search engine providers – than on alternative search engines to take these considerations into account when designing or interfering with algorithms for the purposes of promoting ethical compliance, given their larger market share. Google currently has over 90% of the worldwide market share6 and therefore has the potential to indirectly ‘cause harm’, through the potential promotion of low IQ webpages on vaccinations, to a great many more people than any of the alternative providers. However, studies have shown that those who hold anti-authoritarian views, openness to (potentially) controversial opinions, and an interest in alternative medicine are, in some cases, already more likely to hold vaccine-hesitant beliefs 30. These are also the individuals who are most likely to use alternative search providers given their sensitivity to privacy and tracking concerns. Therefore, while the alternative providers might reach a proportionally smaller audience – they are reaching an audience that is already more receptive to antivaccine information 31, and therefore more vulnerable to its effects. In other words, unless the alternative providers take steps to rank vaccine-related search results according to IQ, they may cause greater harm to those they do reach, meaning that the net negative impact is still greater even though the number of individuals they reach is smaller.

Online Information of Vaccines: Discussion

This paper is available on arxiv under CC 4.0 license. Authors: (1) Pietro Ghezzi, Brighton & Sussex Medical School, Falmer, Brighton, UK; (2) Peter G Bannister, Brighton & Sussex Medical School, Falmer, Brighton, UK; (3) Gonzalo Casino, Communication Department, Pompeu Fabra University, Barcelona, Spain and Iberoamerican Cochrane Center, Barcelona, Spain; (4) Alessia Catalani, Department of Biomolecular Sciences, University of Urbino Carlo Bo, Urbino, PU, Italy; (5) Michel Goldman, Institute for Interdisciplinary Innovation in healthcare (I3h), Université libre de Bruxelles; (6) Jessica Morley, Oxford Internet Institute, University of Oxford, Oxford, UK; (7) Marie Neunez, Institute for Interdisciplinary Innovation in healthcare (I3h), Université libre de Bruxelles; (8) Andreu Prados, Communication Department, Pompeu Fabra University, Barcelona, Spain, Iberoamerican Cochrane Center, Barcelona, Spain, Department of Biomolecular Sciences, University of Urbino Carlo Bo, Urbino, PU, Italy, Institute for Interdisciplinary Innovation in healthcare (I3h), Université libre de Bruxelles, Oxford Internet Institute, University of Oxford, Oxford, UK, and Blanquerna School of Health Sciences, Universitat Ramon Llull, Barcelona, Spain; (9) Mariarosaria Taddeo, Oxford Internet Institute, University of Oxford, Oxford, UK, Blanquerna School of Health Sciences, Universitat Ramon Llull, Barcelona, Spain, and The Alan Turing Institute, London, UK; (10) Tania Vanzolini, Department of Biomolecular Sciences, University of Urbino Carlo Bo, Urbino, PU, Italy; (11) Luciano Floridi, Oxford Internet Institute, University of Oxford, Oxford, UK, Blanquerna School of Health Sciences, Universitat Ramon Llull, Barcelona, Spain, and The Alan Turing Institute, London, UK. Table of Links Abstract and Introduction Methods Web search and content analysis Results Discussion Conclusion and References Table 1. Example of classification of webpages Table 2. Percentage of vaccine-negative webpages in different SERPs Web search and content analysis Searches in English were done from Falmer, Sussex, United Kingdom; in Italian from Urbino, Italy; in French from Bruxelles, Belgium; in Spanish from Barcelona, Spain. Each search was done using a logged-out Chrome browser cleared of cookies and previous search history so that the only identification was the IP address and its geolocalization. Additionally, when available, the local version of each search engine was used (e.g. Google.co.uk and Google.it). For searching Google.com, automatic redirection to google.co.uk was avoided by using the URL Google.com/ncr (no-country-redirect). The first 30 URL results from each search engine result page (SERP), excluding those marked as advertisements, and were transferred to a spreadsheet. When a duplicate URL was encountered, it was excluded. Websites were then visited and the content of each page was coded as vaccine-positive, negative or neutral, depending on the stance taken on the connection between vaccines and autism. Webpages recommending vaccination and/or negating the link with autism were coded as “vaccine-positive”. Those promoting vaccine hesitancy, cautioning about the risk of autism or openly anti-vaccine, were coded as “vaccine-negative”. Additionally, webpages that claimed further studies needed to be conducted to clarify the link between vaccines and autism were also coded as “vaccine-negative”, as previous research by {Browne, 2015 #421} has shown that users perceive this as confirmation of the fact that vaccine safety has not been proven. Webpages simply reporting the history of the anti-vaccine movement or a related legal debate were coded as “vaccine-neutral”. Examples of positive, negative and neutral webpages are provided in Table 1. Coding was completed by two raters for each language and inter-rater agreement was calculated with GraphPad, which uses equations 18.16 to 18.20 from Fleiss 19. On a sample of 59 webpages in English, agreement was 85%, with a Kappa of 0.669 (standard error, 0.077) and a 95% confidence interval from 0.518 to 0.820, a strength of agreement is considered to be 'good' 19. In Italian, agreement was 90%, with a Kappa of 0.818 (standard error, 0.067) and a 95% confidence interval from 0.687 to 0.950, a strength of agreement is considered to be 'very good'. In Spanish, agreement was 83%, with a Kappa of 0.609 (standard error, 0.098) and a 95% confidence interval from 0.418 to 0.801, a strength of agreement is considered to be 'very good'. In French, agreement was 89% with a Kappa of 0.746 (standard error, 0.091) and a 95% confidence interval from 0.568 to 0.924. When frequencies of vaccine-negative webpages was compared across different search engines, we used a two-tailed Fisher’s test corrected for multiple comparison using the method of Benjamini, Krieger and Yekutieli and a false discovery rate (FDR) of 5%.

Online Information of Vaccines: Web Search and Content Analysis

This paper is available on arxiv under CC 4.0 license. Authors: (1) Pietro Ghezzi, Brighton & Sussex Medical School, Falmer, Brighton, UK; (2) Peter G Bannister, Brighton & Sussex Medical School, Falmer, Brighton, UK; (3) Gonzalo Casino, Communication Department, Pompeu Fabra University, Barcelona, Spain and Iberoamerican Cochrane Center, Barcelona, Spain; (4) Alessia Catalani, Department of Biomolecular Sciences, University of Urbino Carlo Bo, Urbino, PU, Italy; (5) Michel Goldman, Institute for Interdisciplinary Innovation in healthcare (I3h), Université libre de Bruxelles; (6) Jessica Morley, Oxford Internet Institute, University of Oxford, Oxford, UK; (7) Marie Neunez, Institute for Interdisciplinary Innovation in healthcare (I3h), Université libre de Bruxelles; (8) Andreu Prados, Communication Department, Pompeu Fabra University, Barcelona, Spain, Iberoamerican Cochrane Center, Barcelona, Spain, Department of Biomolecular Sciences, University of Urbino Carlo Bo, Urbino, PU, Italy, Institute for Interdisciplinary Innovation in healthcare (I3h), Université libre de Bruxelles, Oxford Internet Institute, University of Oxford, Oxford, UK, and Blanquerna School of Health Sciences, Universitat Ramon Llull, Barcelona, Spain; (9) Mariarosaria Taddeo, Oxford Internet Institute, University of Oxford, Oxford, UK, Blanquerna School of Health Sciences, Universitat Ramon Llull, Barcelona, Spain, and The Alan Turing Institute, London, UK; (10) Tania Vanzolini, Department of Biomolecular Sciences, University of Urbino Carlo Bo, Urbino, PU, Italy; (11) Luciano Floridi, Oxford Internet Institute, University of Oxford, Oxford, UK, Blanquerna School of Health Sciences, Universitat Ramon Llull, Barcelona, Spain, and The Alan Turing Institute, London, UK. Table of Links Abstract and Introduction Methods Web search and content analysis Results Discussion Conclusion and References Table 1. Example of classification of webpages Table 2. Percentage of vaccine-negative webpages in different SERPs Results Figure 1 shows the ranking of positive (green), neutral (yellow), and negative (red) websites returned by the different search engines in English, French, Italian and Spanish. A comparison of the percentage of vaccine-negative webpages in different SERPs (Table 2) shows that Google is consistently returning less misinformation, although some of its local version show more vaccine-negative webpages than the English-US version (Google.com). Other search engines return some vaccine-negative webpages with some, like Mojeek in English-UK and Arianna and Virgilio in Italian are the most likely to rank higher webpages with misinformation. In English, the frequency of vaccine-negative webpages in Yahoo, Bing and Mojeek was significantly higher than in Google.com. In Italian, all SERPs had a significantly higher proportion of vaccine-negative webpages compared with Google.com. In French, all search engines returned a significantly higher number of vaccine-negative webpages than Google.com or the local version Google.fr. There were no significant differences for search engines in Spanish. The SERPs of all search engines providers contained a higher proportion of vaccine-negative results than those obtained from Google.com. Even the localized versions of Google (Google.co.uk, Google.it and Google.es) returned more negative pages than the US/international English Google.com. The two Italian-only search engines (Virgilio and Arianna) returned the highest number of negative pages in Italian. In several instances, the same webpages were returned by different search engines, with large overlaps between these results as shown in Figure 2.

Online Information of Vaccines: Results

This paper is available on arxiv under CC 4.0 license. Authors: (1) Pietro Ghezzi, Brighton & Sussex Medical School, Falmer, Brighton, UK; (2) Peter G Bannister, Brighton & Sussex Medical School, Falmer, Brighton, UK; (3) Gonzalo Casino, Communication Department, Pompeu Fabra University, Barcelona, Spain and Iberoamerican Cochrane Center, Barcelona, Spain; (4) Alessia Catalani, Department of Biomolecular Sciences, University of Urbino Carlo Bo, Urbino, PU, Italy; (5) Michel Goldman, Institute for Interdisciplinary Innovation in healthcare (I3h), Université libre de Bruxelles; (6) Jessica Morley, Oxford Internet Institute, University of Oxford, Oxford, UK; (7) Marie Neunez, Institute for Interdisciplinary Innovation in healthcare (I3h), Université libre de Bruxelles; (8) Andreu Prados, Communication Department, Pompeu Fabra University, Barcelona, Spain, Iberoamerican Cochrane Center, Barcelona, Spain, Department of Biomolecular Sciences, University of Urbino Carlo Bo, Urbino, PU, Italy, Institute for Interdisciplinary Innovation in healthcare (I3h), Université libre de Bruxelles, Oxford Internet Institute, University of Oxford, Oxford, UK, and Blanquerna School of Health Sciences, Universitat Ramon Llull, Barcelona, Spain; (9) Mariarosaria Taddeo, Oxford Internet Institute, University of Oxford, Oxford, UK, Blanquerna School of Health Sciences, Universitat Ramon Llull, Barcelona, Spain, and The Alan Turing Institute, London, UK; (10) Tania Vanzolini, Department of Biomolecular Sciences, University of Urbino Carlo Bo, Urbino, PU, Italy; (11) Luciano Floridi, Oxford Internet Institute, University of Oxford, Oxford, UK, Blanquerna School of Health Sciences, Universitat Ramon Llull, Barcelona, Spain, and The Alan Turing Institute, London, UK. Table of Links Abstract and Introduction Methods Web search and content analysis Results Discussion Conclusion and References Table 1. Example of classification of webpages Table 2. Percentage of vaccine-negative webpages in different SERPs Methods In seeking to answer the above research question, this study builds on previous work which: (a) focuses on the results returned by a search of “vaccines autism” in different countries and languages, and found a great variability in the composition of the webpages returned and their ranking 17; and (b) suggests that whether a webpage promotes a medical treatment based on the practices of evidence-based medicine (EBM; e.g. based on the reporting by Cochrane reviews) can act as a proxy for health IQ. The phrase “vaccines autism” was searched using the main search engines (Google, Bing and Yahoo) and alternative providers which focus on respecting privacy and personal data (Duckduckgo, Mojeek, Qwant and Ecosia3) in four languages/countries (English-UK, Italian, Spanish and French) and the international, US-English version of Google (Google.com). We also used some country- or language-specific search engines. Each webpage returned was classified as vaccine-positive, -negative or –neutral as described in the Table 1

Online Information of Vaccines: Methods

This paper is available on arxiv under CC 4.0 license.
Authors:
(1) Kiriaki Frangias;
(2) Andrew Lin;
(3) Ellen Vitercik;
(4) Manolis Zampetakis. Table of Links Abstract and Introduction Warm-up: Agents with known equal disutility Agents with unknown disutilites Experiments Conclusions and future directions, References A Summary of notation B Omitted proofs from Section 2 C Omitted proofs from Section 3 D Additional Information about Experiments D Additional Information about Experiments D.1 Post-processing algorithm Algorithm 4 takes as input “comparisons”, which is a list of s sublists, each containing the returned results of the binary comparisons performed by an agent. Each result is represented with a string of the form “i < j”, representing that according to the agent, the true score of item Ti is smaller than that of item Tj. The input “identified” is a set containing the indeces of all agents who returned at least one verified comparison incorrectly. Algorithm 4 iterates through the comparisons returned by agents not in the set “identified” and adds them to the set “condensed”, which will be returned at the end. If the comparison is already in the set “condensed”, then it is not added again. If there has been another agent whose returned comparison disagrees with it, then the comparison is added to the set “conflicts” (if not already added), and is removed from the set “condensed”. D.2 Principal’s utility vs. parameter π The plot shows a monotonic increase in the principal’s utility as π increases. The shaded region corresponds to a 90% confidence interval across 15 iterations. We notice that for any value of π that is greater than 0.35, it is in the principal’s best interest to implement our contract instead of performing all pairwise comparisons herself. D.3 Additional Hyper-parameter settings from Section 4 For the remainder of this section, we fix the hyper-parameter settings found in Section 4, unless otherwise stated. Namely, we assume that π = 0.8, δ = 0.01, n = 100, s = 100, ψ¯ = 2, and ψ = 0.01. Discussion of Figure 5. In Figure 5 we provide additional hyper-parameter settings for Figure 1a in Section 4. First, we select the probability of an effortful agent being “good” to be a value in {0.3, 0.6, 0.9} and the number of agents and items to be in the set {50, 200}. In plots 5a-5c results are averaged across 15 trials, while in plots 5d- 5f, results are averaged across 10 trials. The blue shading corresponds to a 90% confidence interval. We notice that the confidence intervals decrease with the number of agents, even for a small number of trials. The confidence intervals also decrease, as the agents’ work becomes more predictable (π increases). We find that as π increases, the principal’s utility also increases along with the threshold for ψ, below which it is in the principal’s best interest to run CrowdSort than perform all comparisons herself. This threshold also decreases with the number of agents, s. This implies that there is a sweet spot, in which the expected number of “good” agents is just enough that the ground-truth ordering of the items is retrieved and it is sufficiently small so that the principal can pay all of them. Discussion of Figure 6. In Figure 6 we provide additional hyper-parameter settings for Figure 1a in Section 4. Specifically, we select the per-comparison disutility of the principal, ψ¯, along with the per-comparison utility of the principal, λ, to take values in {4, 6, 8}. The number of agents and items is in the set {200, 500} and our results are averaged across 2 trials with the blue shading corresponding to a 90% confidence interval. We notice that the utility of the principal dramatically increases with the value she assigns to the correct ordering. Interestingly, by comparing Figure 5f to Figures 6a-6a, we find that even when the reliability of the agents is smaller (π = 0.8 in 5f and π = 0.9 in 6a6c), the principal’s point of indifference is higher when her disutility is larger. As before, we notice that when a large number of agents is incentivized, their labor needs to be sufficiently inexpensive (smaller threshold on ψ) for their work to be more profitable to the principal than performing all pairwise comparisons herself. Lastly, our results show, that for a broad range of hyperparameters, our theoretical analysis accurately predicts the value of ψ, for which the principal is indifferent between running CrowdSort or performing all comparisons herself. Discussion of Figure 7. We provide additional parameter settings corresponding to Figure 1b. The parameters π and s along with the number of trials vary as in Figure 5. We find that as the number of agents to be incentivized grows, our theoretical analysis for the principal’s point of indifference becomes more accurate. Also, noise on the disutilites results in many agents not being incentivized to exert effort and therefore get paid. This increases the principal’s utility, especially when parameter π is large, therefore resulting in a less steep slope. As in Figure 1b, we note that the confidence intervals become larger with ψ, as the principal’s utility shows higher variance, when less agents are “good”. Discussion of Figure 10. In Figure 10 we offer a range of parameters corresponding to Figure 3 in Section 4. Specifically, we vary the probability than an effortful agent is “good”, which take values in {0.3, 0.6, 0.9} and the principal’s per-comparison disutility, ψ, and utility λ, which takes values in {4, 6, 8}. The multicolored plots indicate the principal’s utility for different sample sizes (namely, 10, 20, 100, and 500) drawn from D, which in this case is N (0.03, 0.01). Notice that we do not normalize the y-axis in this case. Results are averaged across 10 trials. We see that as agents become more reliable (higher value of π) and as the principal’s value for the correct sorting grows (λ), it becomes increasingly beneficial to the principal to run CrowdSort, than perform all pairwise comparisons on her own. Interestingly, even if π is 0.3, for sufficiently large values of ψ¯, the agent’s work is more profitable to the principal than sorting herself. This suggests that for a sufficiently large value of ψ/ψ ¯ , incentivizing agents under our proposed contract is substantially more beneficial to the principal than sorting the items herself. This paper is available on Arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. Authors: (1) Kiriaki Frangias; (2) Andrew Lin; (3) Ellen Vitercik; (4) Manolis Zampetakis. This paper is available on arxiv under CC 4.0 license. Authors: Authors: (1) Kiriaki Frangias; (2) Andrew Lin; (3) Ellen Vitercik; (4) Manolis Zampetakis. Table of Links Abstract and Introduction Abstract and Introduction Warm-up: Agents with known equal disutility Warm-up: Agents with known equal disutility Agents with unknown disutilites Agents with unknown disutilites Experiments Experiments Conclusions and future directions, References Conclusions and future directions, References A Summary of notation A Summary of notation B Omitted proofs from Section 2 B Omitted proofs from Section 2 C Omitted proofs from Section 3 C Omitted proofs from Section 3 D Additional Information about Experiments D Additional Information about Experiments D Additional Information about Experiments D.1 Post-processing algorithm Algorithm 4 takes as input “comparisons”, which is a list of s sublists, each containing the returned results of the binary comparisons performed by an agent. Each result is represented with a string of the form “ i < j”, representing that according to the agent, the true score of item T i is smaller than that of item T j . The input “identified” is a set containing the indeces of all agents who returned at least one verified comparison incorrectly. s i i j Algorithm 4 iterates through the comparisons returned by agents not in the set “identified” and adds them to the set “condensed”, which will be returned at the end. If the comparison is already in the set “condensed”, then it is not added again. If there has been another agent whose returned comparison disagrees with it, then the comparison is added to the set “conflicts” (if not already added), and is removed from the set “condensed”. D.2 Principal’s utility vs. parameter π The plot shows a monotonic increase in the principal’s utility as π increases. The shaded region corresponds to a 90% confidence interval across 15 iterations. We notice that for any value of π that is greater than 0.35, it is in the principal’s best interest to implement our contract instead of performing all pairwise comparisons herself. D.3 Additional Hyper-parameter settings from Section 4 For the remainder of this section, we fix the hyper-parameter settings found in Section 4, unless otherwise stated. Namely, we assume that π = 0.8, δ = 0.01, n = 100, s = 100, ψ¯ = 2, and ψ = 0.01. Discussion of Figure 5. In Figure 5 we provide additional hyper-parameter settings for Figure 1a in Section 4. First, we select the probability of an effortful agent being “good” to be a value in {0.3, 0.6, 0.9} and the number of agents and items to be in the set {50, 200}. In plots 5a-5c results are averaged across 15 trials, while in plots 5d- 5f, results are averaged across 10 trials. Discussion of Figure 5. The blue shading corresponds to a 90% confidence interval. We notice that the confidence intervals decrease with the number of agents, even for a small number of trials. The confidence intervals also decrease, as the agents’ work becomes more predictable (π increases). We find that as π increases, the principal’s utility also increases along with the threshold for ψ, below which it is in the principal’s best interest to run CrowdSort than perform all comparisons herself. This threshold also decreases with the number of agents, s. This implies that there is a sweet spot, in which the expected number of “good” agents is just enough that the ground-truth ordering of the items is retrieved and it is sufficiently small so that the principal can pay all of them. Discussion of Figure 6. In Figure 6 we provide additional hyper-parameter settings for Figure 1a in Section 4. Specifically, we select the per-comparison disutility of the principal, ψ¯, along with the per-comparison utility of the principal, λ, to take values in {4, 6, 8}. The number of agents and items is in the set {200, 500} and our results are averaged across 2 trials with the blue shading corresponding to a 90% confidence interval. Discussion of Figure 6. We notice that the utility of the principal dramatically increases with the value she assigns to the correct ordering. Interestingly, by comparing Figure 5f to Figures 6a-6a, we find that even when the reliability of the agents is smaller (π = 0.8 in 5f and π = 0.9 in 6a6c), the principal’s point of indifference is higher when her disutility is larger. As before, we notice that when a large number of agents is incentivized, their labor needs to be sufficiently inexpensive (smaller threshold on ψ) for their work to be more profitable to the principal than performing all pairwise comparisons herself. Lastly, our results show, that for a broad range of hyperparameters, our theoretical analysis accurately predicts the value of ψ, for which the principal is indifferent between running CrowdSort or performing all comparisons herself. Discussion of Figure 7. We provide additional parameter settings corresponding to Figure 1b. The parameters π and s along with the number of trials vary as in Figure 5. We find that as the number of agents to be incentivized grows, our theoretical analysis for the principal’s point of indifference becomes more accurate. Discussion of Figure 7. Also, noise on the disutilites results in many agents not being incentivized to exert effort and therefore get paid. This increases the principal’s utility, especially when parameter π is large, therefore resulting in a less steep slope. As in Figure 1b, we note that the confidence intervals become larger with ψ, as the principal’s utility shows higher variance, when less agents are “good”. Discussion of Figure 10. In Figure 10 we offer a range of parameters corresponding to Figure 3 in Section 4. Specifically, we vary the probability than an effortful agent is “good”, which take values in {0.3, 0.6, 0.9} and the principal’s per-comparison disutility, ψ, and utility λ, which takes values in {4, 6, 8}. The multicolored plots indicate the principal’s utility for different sample sizes (namely, 10, 20, 100, and 500) drawn from D, which in this case is N (0.03, 0.01). Discussion of Figure 10. Notice that we do not normalize the y-axis in this case. Results are averaged across 10 trials. We see that as agents become more reliable (higher value of π) and as the principal’s value for the correct sorting grows (λ), it becomes increasingly beneficial to the principal to run CrowdSort, than perform all pairwise comparisons on her own. Interestingly, even if π is 0.3, for sufficiently large values of ψ¯, the agent’s work is more profitable to the principal than sorting herself. This suggests that for a sufficiently large value of ψ/ψ ¯ , incentivizing agents under our proposed contract is substantially more beneficial to the principal than sorting the items herself. This paper is available on Arxiv under CC 4.0 license. This paper is available on Arxiv under CC 4.0 license. Arxiv

Additional Information About the Algorithmic Contract Design Experiments: What You Should Know

This paper is under CC 4.0 license. available on arxiv Authors: (1) Jinrui Yang, School of Computing & Information Systems, The University of Melbourne (Email: jinruiy@student.unimelb.edu.au); (2) Timothy Baldwin, School of Computing & Information Systems, The University of Melbourne and Mohamed bin Zayed University of Artificial Intelligence, UAE (Email: (tbaldwin,trevor.cohn)@unimelb.edu.au); (3) Trevor Cohn, School of Computing & Information Systems, The University of Melbourne. Table of Links Abstract and Intro Background and Related Work Multi-EuP Experiments and Findings Language Bias Discussion Conclusion, Limitations, Ethics Statement, Acknowledgements, References, and Appendix 6 Conclusion In this paper, we introduce Multi-EuP, a novel dataset for multilingual information retrieval across 24 languages, collected from European Parliament debates. The demographic information provided by the Multi-EuP dataset serves a dual purpose: not only does it contribute to multilingual retrieval tasks, but it also holds significant potential for advancing research in the realm of fairness and bias. This dataset can play a pivotal role in investigating issues of equitable representations and mitigation of biases within document ranking settings. Multi-EuP facilitates diverse information retrieval (IR) scenarios, encompassing one-vs-one, one-vs-many, and many-vs-many settings. We demonstrated the utility of Multi-EuP as a benchmark for evaluating both monolingual and multilingual IR. Our study reveals the presence of language bias in multilingual IR when employing BM25. We further validate the effectiveness of mitigating this bias through the strategic implementation of whitespace as a language tokenizer. We propose to conduct future work in three main areas. First, we intend to expand our investigation of language bias to encompass a broader range of ranking methods, including neural methods such as mDPR (Zhang et al., 2021), mColBERT (Lawrie et al., 2023) and PLAID-X(Santhanam et al., 2022). Second, we will expand the dataset by developing an automated API to retrieve data published by the European Parliament (EP), thereby ensuring realtime synchronization of our dataset. Lastly, our current experiments have explored language bias only, but we plan to further investigate gender bias, age bias, and nationality bias. Limitations The limitations of the Multi-EuP dataset are notable but navigable. Primarily, the temporal coverage of the dataset is confined to the past three years. This temporal constraint arises due to the fact that, preceding 2020, documents released by the EU were predominantly available in mono-lingual versions only. However, a potential remedy lies in the amalgamation of the Europarl (Koehn, 2005) collection, enabling a more comprehensive and holistic MultiEuP dataset. Furthermore, it is worth noting the domain skew of the dataset, in that Multi-EuP inevitably centers on political matters. While this presents challenges, particularly in terms of the intricate nuances of political language, it inherently serves as an excellent foundational stepping stone for delving into the intricacies of multilingual retrieval. We believe, however, that this dataset can serve as a launching pad for broader explorations encompassing crossdomain and open-domain transfer learning scenarios, thus contributing to the broader landscape of language understanding and retrieval. Ethics Statement The dataset contains publicly-available EP data that does not include personal or sensitive information, with the exception of information relating to public officeholders, e.g., the names of the active members of the European Parliament, European Council, or other official administration bodies. The collected data is licensed under the Creative Commons Attribution 4.0 International licence. [8] Acknowledgements This research was funded by Melbourne Research Scholarship and undertaken using the LIEF HPCGPGPU Facility hosted at the University of Melbourne. This facility was established with the assistance of LIEF Grant LE170100200. We would like to thank George Buchanan for providing valuable feedback. References Luiz Henrique Bonifacio, Israel Campiotti, Roberto de Alencar Lotufo, and Rodrigo Frassetto Nogueira. 2021. mMARCO: A multilingual version of MS MARCO passage ranking dataset. CoRR, abs/2108.13897. Ilias Chalkidis, Manos Fergadiotis, and Ion Androutsopoulos. 2021. MultiEURLEX - a multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6974–6996, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. 8 https://eur-lex.europa.eu/cont Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages. Transactions of the Association for Computational Linguistics, 8:454–470. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for opendomain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781, Online. Association for Computational Linguistics. Omar Khattab and Matei Zaharia. 2020. Colbert: Efficient and effective passage search via contextualized late interaction over BERT. CoRR, abs/2004.12832. Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Proceedings of Machine Translation Summit X: Papers, pages 79–86, Phuket, Thailand. Dawn Lawrie, Eugene Yang, Douglas W. Oard, and James Mayfield. 2023. Neural approaches to multilingual information retrieval. arXiv cs.IR 2209.01335. Jimmy Lin, Xueguang Ma, Sheng-Chieh Lin, JhengHong Yang, Ronak Pradeep, and Rodrigo Nogueira. 2021. Pyserini: A Python toolkit for reproducible information retrieval research with sparse and dense representations. https://github.com/ castorini/pyserini. Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. CoRR, abs/1611.09268. Ella Rabinovich, Raj Nath Patel, Shachar Mirkin, Lucia Specia, and Shuly Wintner. 2017. Personalized machine translation: Preserving original author traits. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 1074–1084, Valencia, Spain. Association for Computational Linguistics. Razieh Rahimi, Azadeh Shakery, and Irwin King. 2015. Multilingual information retrieval in the language modeling framework. Information Retrieval Journal, 18:246–281. Keshav Santhanam, Omar Khattab, Christopher Potts, and Matei Zaharia. 2022. PLAID: An efficient engine for late interaction retrieval. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, CIKM ’22, page 1747–1756, New York, NY, USA. Association for Computing Machinery. Jörg Tiedemann and Santhosh Thottingal. 2020. OPUSMT – building open translation services for the world. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, pages 479–480, Lisboa, Portugal. European Association for Machine Translation. Eva Vanmassenhove and Christian Hardmeier. 2018. Europarl datasets with demographic speaker information. In Proceedings of the 21st Annual Conference of the European Association for Machine Translation, page 391, Alicante, Spain. Denny Vrandeciˇ c and Markus Krötzsch. 2014. ´ Wikidata: A free collaborative knowledge base. Communications of the ACM, 57:78–85. Konrad Wojtasik, Vadim Shishkin, Kacper Wołowiec, Arkadiusz Janz, and Maciej Piasecki. 2023. BEIRPL: Zero shot information retrieval benchmark for the Polish language. arXiv cs.IR 2305.19840. Peilin Yang, Hui Fang, and Jimmy Lin. 2017. Anserini: Enabling the use of lucene for information retrieval research. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1253–1256. Xinyu Zhang, Xueguang Ma, Peng Shi, and Jimmy Lin. 2021. Mr. TyDi: A multi-lingual benchmark for dense retrieval. arXiv cs.CL 2108.08787. A. Appendix [8] https://eur-lex.europa.eu/content/ legal-notice/legal-notice.html

Multi-EuP: Analysis of Bias in Information Retrieval - Conclusion, Limitations, and Ethics Statement

This paper is under CC 4.0 license. available on arxiv Authors: (1) Jinrui Yang, School of Computing & Information Systems, The University of Melbourne (Email: jinruiy@student.unimelb.edu.au); (2) Timothy Baldwin, School of Computing & Information Systems, The University of Melbourne and Mohamed bin Zayed University of Artificial Intelligence, UAE (Email: (tbaldwin,trevor.cohn)@unimelb.edu.au); (3) Trevor Cohn, School of Computing & Information Systems, The University of Melbourne. Table of Links Abstract and Intro Background and Related Work Multi-EuP Experiments and Findings Language Bias Discussion Conclusion, Limitations, Ethics Statement, Acknowledgements, References, and Appendix 5 Language Bias Discussion In light of our findings in a one-vs-many setting, we were keen to delve further into the underlying causes of the disparity between languages. 5.1 Bias Detection Language bias is likely if the query language aligns better with one document language than another. As mentioned earlier, Pyserini supports different tokenizers, specifically language-specific tokenizers or simple whitespace tokenization. Therefore, in the one-vs-many setting, we analyze the composition of the top-100 rankings for the 100 topics. During indexing of the document collection, we used the simple whitespace tokenizer, given the multilingual nature of the collection. However, over the queries during retrieval, we employed two different tokenizers — a language-specific tokenizer, and the whitespace tokenizer. We conducted a correlation analysis between the language of the topics and the language of the top 100 relevant documents. From Table 2, we can see that relevance judgments in our test cases are consistent across languages, ensuring uniformity in the correlation matrix within the test set. However, Figure 2 reveals that both approaches generate strong language bias. In both cases, the query language aligns better with documents in its own language than others. The right plot appears to show that languages from the same family has strong correlation (e.g., PL, CS) and (IT, ES) since they may have some shared vocabulary. 5.2 Collection Distribution Factors Initially, we hypothesized that the disparity for each language may be a contributing factor to this bias. Figure 3 presents the regression line between the number of documents in a given language and MRR, which explains much of the variation across languages. However, note the outlier above the regression line (Polish: PL), which has a substantial number of documents but surprisingly low MRR performance. We refer to this phenomenon as a “BM25 unfriendly” language. According to Wojtasik et al. (2023), the main reason for the low performance of Polish lies in its highly-inflected morphology, giving rise to a a multitude of word forms per lexeme, including inflections of proper names, and complex morphological structure. In such cases, lexical matching is less effective than in other morphologically-simpler languages. Furthermore, LUCENE 8.5.1 API does not have a language-specific tokenizer for Polish. Conversely, languages below the regression line can be termed “BM25 friendly” languages, as they require fewer documents to achieve higher MRR in retrieval. 5.3 Language Tokenizer Factors Secondly, we speculated that the choice of language-specific Analyzer in LUCENE might be a contributing factor, as it influences word tokenization, token filter, synonym expansion and other processing. [7] To investigate this, we conducted a controlled experiment in the one-vs-many setting. When indexing the collection, given the multilingual nature of the collection, we employed whitespace as the tokenizer. However, over the queries, we experimented with either a language-specific tokenizer or whitespace tokenizer. We then compared the linear regression of MRR against the number of documents in Figure 3. On the right side of the plot, we can see a strong correlation when using whitespace tokenization for both the collection and the queries, reducing language bias. Furthermore, when transitioning from languagespecific tokenizers to whitespace tokenizers, the overall MRR across all languages declined modestly, from 15.02 to 14.18. That is, the original performance level was largely preserved, but language bias was diminished in using simple whitespace tokenization. [7] https://lucene.apache.org/core/8_0_0/core/ org/apache/lucene/analysis/package-summary.html# package.description

Stories, News & Mentions about INFARM

Infarm’s Stories on HackerNoon

Infarm's News & Mentions