Source is a data management layer built for the decentralized web, a fast-growing market expected to reach $40B by 2025

Source

source

Authors:
(1) Yagci, Nurce, HAW Hamburg, Germany & nurce.yagci@haw-hamburg.de;
(2) Sünkler, Sebastian, HAW Hamburg, Germany & sebastian.suenkler@haw-hamburg.de;
(3) Häußler, Helena, HAW Hamburg, Germany & helena.haeuessler@haw-hamburg.de;
(4) Lewandowski, Dirk, HAW Hamburg, Germany & dirk.lewandowski@haw-hamburg.de. Table of Links Abstract and Introduction Literature Review Objectives and Research Questions Methods Results Discussion Conclusion, Research Data, Acknowledgments, and References RESULTS Classification of domains Since we are working with sources from two different countries and the type of content behind some domains might not be immediately recognizable, we manually created a general set of categories based on the top 50 domains found across all results. Following this, we manually classified the 50 most popular domains for both the German and US results (see Table 1). When comparing the top 50 domains in the German results, News service dominates with 54%, while Movies & Entertainment and Sports make up 18% and 14%, respectively. In the US, 34% of the top 50 domains are Sports websites, with News services following close behind at 30%. Movies & Entertainment only make up 12%. These are the overall distributions of domain classes across all search engines; below, we look at the differences between the search engines. Variety of domains A comparison of the number of root domains in the search results of Google, Bing, DuckDuckGo, and Metager in Germany shows that Google has the greatest diversity by a small margin (see Figure 2). Overall, the values are very similar. However, it is noticeable that Google has the greatest variety of domains, especially in the first three positions. Interestingly, the greater diversity of Google's sources is even more pronounced in the US results. To examine the differences more closely, we look at the numbers below. Table 2 shows the cumulative frequencies for the German results. The number of unique root domains converges at the fifth position. We found 2,841 unique domains in Google's results, 2,783 unique domains for Bing, 2,707 unique domains for DuckDuckGo, and 2,683 unique domains for Metager in Germany. There are differences when comparing the search results in Germany and the US. It is notable that, in contrast to Germany, in the US results, there are only minor differences in the first positions. However, Google also shows the greatest variety of root domains in the search results in the US. This is more evident than in the German results (see Table 3). Overall, we found 4,085 unique domains in Google, 3,602 in Bing, 3,579 in DuckDuckGo, and 3,500 in Metager. Popular domains When comparing the top domains for each search engine across all German search results collected, we find a clear preference for Wikipedia in all of them (see Table 4). While Wikipedia is the most popular domain in all search engines, the frequency in the Google results is significantly lower, with only 658 compared to the number of occurrences in Bing being 1,948, in Metager 1,878, and in DuckDuckGo 1,752. The other domains in the top 10 across the search engines are News services (Google: 7, Bing: 4, DuckDuckGo: 3, Metager: 4). Sports make up 2 out of 10 for all but Google, with only one sports website. Surprisingly, Instagram is in the top 10 for DuckDuckGo and Metager, even though in Table 1, we showed that domains of the class Social Media only make up 4% of top domains in the German results. The same goes for Amazon, which is the second most frequent domain in DuckDuckGo, even though in the overall results, E-Commerce websites only make up 2%. Contrarily, in the US results, Wikipedia has the highest number of occurrences in Google with 1,892 compared to Bing's 1,388, Metager's 1,304, and DuckDuckGo's 1,287 (see Table 5). Still, what remains the same is that Wikipedia is the most popular domain across all search engines. Another difference in the US results is that Social Media sources are much more prevalent across all search engines, especially in Google, with Instagram as the second most frequent domain, Facebook as the third, and Twitter as the sixth. This is surprising when considering that Social Media domains only make up 6% of the top domains for the US results (see Table 1). Another finding is that YouTube, a subsidiary of Google, is the top 8th domain for all three alternatives but not for Google. Again, 30% of the top domains for Google, DuckDuckGo, and Metager are News services. For Bing, it is 40%. For each search engine, Sports websites make up 20% of domains. Exclusivity of domains This section compares the top 50 domains for each search engine to determine what source Google and its alternatives might exclusively offer to the user. Table 6 shows the domains found solely in Google's top 50 domains list for the German queries. Out of these 20 domains, eight are Sports websites, six are News services, and four belong to the Entertainment & Movies class, with one each for E-Commerce and Government websites. On the contrary, table 7 shows the list of domains found in the top 50 domains of all three alternatives but not in Google. These would be the domains missed by users who only use Google. Out of the 13 domains, four are News services, followed by three Movies & Entertainment and two Celebrities sources. Interestingly, Instagram and YouTube are missing from Google, as well. When implementing the same evaluation for the US results, we find that 17 domains are exclusively found in Google's top 50 domains, with five domains classified as Sports, four as Movies & Entertainment, another four as News services, three as Social Media and one as Celebrities (see Table 8). Notable is that social media giants Facebook and TikTok are only found in Google and not in the alternatives. On the opposite end, 13 domains are not found in the US results on Google (see Table 9). Six of them are News services, three are websites about Celebrities, two are about Movies & Entertainment, and one each are Sports and E-Commerce websites. Interestingly, Amazon is missing in Google's results, as well as msn.com, a service from Microsoft, like Bing itself. Distribution of domains Next, we look at the Gini coefficient to measure the concentration and statistical dispersion of root domains in search engines. The distributions across all search engines in both German and US results show very similar values. The results range from 0.77 to 0.79 and 0.73 to 0.76 for German and US results, respectively (see Figure 3). Overall, this means that there is an imbalance in the distribution of results from root domains and that some top sources dominate the search results for both countries. Similarity of domains Comparing the similarities of the top 10 German results, we find that Bing and Metager are most similar in terms of their top results, with 70%, followed by 64% overlap between DuckDuckGo and Metager and 63% between Bing and DuckDuckGo. However, when comparing the alternative search engines to Google, we find that the highest overlap is between Google and Bing with 28% and slightly lower ones with DuckDuckGo and Metager at 27%. Interestingly, the top 1 result overlap is higher between Google and Bing (31%) than the overlap between Bing and DuckDuckGo (15%). The results show a mean overlap of 30% of top 1, 40% of top 3, and 47% of top 10 results (table 10). This indicates that when looking at the entire first search results page, nearly half of the domains are the same in all search engines considered. For search engine users, this means that in addition to Google, this would also allow them to see search results that they would otherwise miss if they used the alternative search engine The same evaluation for the US yields similar results. Again, the overlap between Bing and Metager is the highest, but the margin between this pair and the other alternative pairings has been reduced. When looking at the top 10 results, Bing and Metager overlap by 65%, Bing and DuckDuckGo by 64%, and DuckDuckGo and Metager by 62%. While Google's overlap with the other search engines is lower in the US results, again, the pair of Google and Bing is slightly higher (25%) than the 24% overlap between Google and the other two alternatives (see Table 11). Narrowing the results down to the top 3, the overlap between Bing and DuckDuckGo increases minimally to 65% and remains the same for DuckDuckGo and Metager at 62%. Here, the overlap between Google and Bing is lower than in the German results, with only 26%. Furthermore, the top 1 result overlap between Google and Bing is at 29%, which is also lower than in the German results. Generally, compared to the German results, there is a higher overlap in the top 1 position for the pairs Google and DuckDuckGo (20%), Bing and DuckDuckGo (50%), and DuckDuckGo and Metager (47%). The averaged results make this trend clearer because the overlaps for the top 1, top 3, and top 10 results are similar, with 41%, 46%, and 44%, respectively. This paper is available on arxiv under CC 4.0 license. Authors: (1) Yagci, Nurce, HAW Hamburg, Germany & nurce.yagci@haw-hamburg.de; (2) Sünkler, Sebastian, HAW Hamburg, Germany & sebastian.suenkler@haw-hamburg.de; (3) Häußler, Helena, HAW Hamburg, Germany & helena.haeuessler@haw-hamburg.de; (4) Lewandowski, Dirk, HAW Hamburg, Germany & dirk.lewandowski@haw-hamburg.de. Authors: (1) Yagci, Nurce, HAW Hamburg, Germany & nurce.yagci@haw-hamburg.de; (2) Sünkler, Sebastian, HAW Hamburg, Germany & sebastian.suenkler@haw-hamburg.de; (3) Häußler, Helena, HAW Hamburg, Germany & helena.haeuessler@haw-hamburg.de; (4) Lewandowski, Dirk, HAW Hamburg, Germany & dirk.lewandowski@haw-hamburg.de. Table of Links Abstract and Introduction Abstract and Introduction Literature Review Literature Review Objectives and Research Questions Objectives and Research Questions Methods Methods Results Results Discussion Discussion Conclusion, Research Data, Acknowledgments, and References Conclusion, Research Data, Acknowledgments, and References RESULTS Classification of domains Since we are working with sources from two different countries and the type of content behind some domains might not be immediately recognizable, we manually created a general set of categories based on the top 50 domains found across all results. Following this, we manually classified the 50 most popular domains for both the German and US results (see Table 1). When comparing the top 50 domains in the German results, News service dominates with 54%, while Movies & Entertainment and Sports make up 18% and 14%, respectively. In the US, 34% of the top 50 domains are Sports websites, with News services following close behind at 30%. Movies & Entertainment only make up 12%. These are the overall distributions of domain classes across all search engines; below, we look at the differences between the search engines. Variety of domains A comparison of the number of root domains in the search results of Google, Bing, DuckDuckGo, and Metager in Germany shows that Google has the greatest diversity by a small margin (see Figure 2). Overall, the values are very similar. However, it is noticeable that Google has the greatest variety of domains, especially in the first three positions. Interestingly, the greater diversity of Google's sources is even more pronounced in the US results. To examine the differences more closely, we look at the numbers below. Table 2 shows the cumulative frequencies for the German results. The number of unique root domains converges at the fifth position. We found 2,841 unique domains in Google's results, 2,783 unique domains for Bing, 2,707 unique domains for DuckDuckGo, and 2,683 unique domains for Metager in Germany. There are differences when comparing the search results in Germany and the US. It is notable that, in contrast to Germany, in the US results, there are only minor differences in the first positions. However, Google also shows the greatest variety of root domains in the search results in the US. This is more evident than in the German results (see Table 3). Overall, we found 4,085 unique domains in Google, 3,602 in Bing, 3,579 in DuckDuckGo, and 3,500 in Metager. Popular domains When comparing the top domains for each search engine across all German search results collected, we find a clear preference for Wikipedia in all of them (see Table 4). While Wikipedia is the most popular domain in all search engines, the frequency in the Google results is significantly lower, with only 658 compared to the number of occurrences in Bing being 1,948, in Metager 1,878, and in DuckDuckGo 1,752. The other domains in the top 10 across the search engines are News services (Google: 7, Bing: 4, DuckDuckGo: 3, Metager: 4). Sports make up 2 out of 10 for all but Google, with only one sports website. Surprisingly, Instagram is in the top 10 for DuckDuckGo and Metager, even though in Table 1, we showed that domains of the class Social Media only make up 4% of top domains in the German results. The same goes for Amazon, which is the second most frequent domain in DuckDuckGo, even though in the overall results, E-Commerce websites only make up 2%. Contrarily, in the US results, Wikipedia has the highest number of occurrences in Google with 1,892 compared to Bing's 1,388, Metager's 1,304, and DuckDuckGo's 1,287 (see Table 5). Still, what remains the same is that Wikipedia is the most popular domain across all search engines. Another difference in the US results is that Social Media sources are much more prevalent across all search engines, especially in Google, with Instagram as the second most frequent domain, Facebook as the third, and Twitter as the sixth. This is surprising when considering that Social Media domains only make up 6% of the top domains for the US results (see Table 1). Another finding is that YouTube, a subsidiary of Google, is the top 8th domain for all three alternatives but not for Google. Again, 30% of the top domains for Google, DuckDuckGo, and Metager are News services. For Bing, it is 40%. For each search engine, Sports websites make up 20% of domains. Exclusivity of domains This section compares the top 50 domains for each search engine to determine what source Google and its alternatives might exclusively offer to the user. Table 6 shows the domains found solely in Google's top 50 domains list for the German queries. Out of these 20 domains, eight are Sports websites, six are News services, and four belong to the Entertainment & Movies class, with one each for E-Commerce and Government websites. On the contrary, table 7 shows the list of domains found in the top 50 domains of all three alternatives but not in Google. These would be the domains missed by users who only use Google. Out of the 13 domains, four are News services, followed by three Movies & Entertainment and two Celebrities sources. Interestingly, Instagram and YouTube are missing from Google, as well. When implementing the same evaluation for the US results, we find that 17 domains are exclusively found in Google's top 50 domains, with five domains classified as Sports, four as Movies & Entertainment, another four as News services, three as Social Media and one as Celebrities (see Table 8). Notable is that social media giants Facebook and TikTok are only found in Google and not in the alternatives. On the opposite end, 13 domains are not found in the US results on Google (see Table 9). Six of them are News services, three are websites about Celebrities, two are about Movies & Entertainment, and one each are Sports and E-Commerce websites. Interestingly, Amazon is missing in Google's results, as well as msn.com, a service from Microsoft, like Bing itself. Distribution of domains Next, we look at the Gini coefficient to measure the concentration and statistical dispersion of root domains in search engines. The distributions across all search engines in both German and US results show very similar values. The results range from 0.77 to 0.79 and 0.73 to 0.76 for German and US results, respectively (see Figure 3). Overall, this means that there is an imbalance in the distribution of results from root domains and that some top sources dominate the search results for both countries. Similarity of domains Comparing the similarities of the top 10 German results, we find that Bing and Metager are most similar in terms of their top results, with 70%, followed by 64% overlap between DuckDuckGo and Metager and 63% between Bing and DuckDuckGo. However, when comparing the alternative search engines to Google, we find that the highest overlap is between Google and Bing with 28% and slightly lower ones with DuckDuckGo and Metager at 27%. Interestingly, the top 1 result overlap is higher between Google and Bing (31%) than the overlap between Bing and DuckDuckGo (15%). The results show a mean overlap of 30% of top 1, 40% of top 3, and 47% of top 10 results (table 10). This indicates that when looking at the entire first search results page, nearly half of the domains are the same in all search engines considered. For search engine users, this means that in addition to Google, this would also allow them to see search results that they would otherwise miss if they used the alternative search engine The same evaluation for the US yields similar results. Again, the overlap between Bing and Metager is the highest, but the margin between this pair and the other alternative pairings has been reduced. When looking at the top 10 results, Bing and Metager overlap by 65%, Bing and DuckDuckGo by 64%, and DuckDuckGo and Metager by 62%. While Google's overlap with the other search engines is lower in the US results, again, the pair of Google and Bing is slightly higher (25%) than the 24% overlap between Google and the other two alternatives (see Table 11). Narrowing the results down to the top 3, the overlap between Bing and DuckDuckGo increases minimally to 65% and remains the same for DuckDuckGo and Metager at 62%. Here, the overlap between Google and Bing is lower than in the German results, with only 26%. Furthermore, the top 1 result overlap between Google and Bing is at 29%, which is also lower than in the German results. Generally, compared to the German results, there is a higher overlap in the top 1 position for the pairs Google and DuckDuckGo (20%), Bing and DuckDuckGo (50%), and DuckDuckGo and Metager (47%). The averaged results make this trend clearer because the overlaps for the top 1, top 3, and top 10 results are similar, with 41%, 46%, and 44%, respectively. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Domain Classification: Analyzing Source Types in Search Engine Results

Authors:
(1) Yagci, Nurce, HAW Hamburg, Germany & nurce.yagci@haw-hamburg.de;
(2) Sünkler, Sebastian, HAW Hamburg, Germany & sebastian.suenkler@haw-hamburg.de;
(3) Häußler, Helena, HAW Hamburg, Germany & helena.haeuessler@haw-hamburg.de;
(4) Lewandowski, Dirk, HAW Hamburg, Germany & dirk.lewandowski@haw-hamburg.de. Table of Links Abstract and Introduction Literature Review Objectives and Research Questions Methods Results Discussion Conclusion, Research Data, Acknowledgments, and References LITERATURE REVIEW Search result overlap By the mid-1990s, researching the overlap of search results between different search engines sparked interest for the purpose of estimating the size of the Web (Bharat, 1998; Ding & Marchionini, 1996; Lawrence & Giles, 1999). The generally small degrees of overlap indicated diverse underlying databases, each of limited size. Therefore, metasearch engines that combined the results of several search engines were meant to provide an additional value (Chignell et al., 1999) and attracted further research (i.e., Meng et al., 2002). Consequently, Spink et al. (Spink et al., 2006) included the meta search engine Dogpile in their extensive study on the overlap of search results. For a collection of 12,570 queries, the results found on the first search engine result page (SERP) of Ask Jeeves, Google, MSN Search, and Yahoo were captured. 84.9% of the results were unique to one search engine, while only 1.1% were shared by all search engines. Additionally, the low overlap between search engines also manifests in the ranking since only 7% of the top results were similar. This is consistent with previous studies that reported low overlap in ranking (Bar-Ilan, 2005; Bar‐Ilan et al., 2006) Subsequent studies report an increase of overlap in search results, while ranking algorithms appear to be the leading cause for differences in the presentation of the results. Bilal and Ellis (2011) compared major web search engines (Google, Bing, Yahoo) with search engines specifically designed for children (Ask Kids, Yahoo Kids) for queries of various lengths. Bing and Yahoo shared the same overlap with Google for nearly all queries, ranging between 22% and 40%. In contrast, Yahoo Kids had mostly unique results. However, the disparate relevance ranking between Bing and Yahoo suggests different ranking algorithms. Cardoso and Magalhães (2011) measured the overlap of search result rankings based on URLs and website contents. Search results for 40,000 queries were retrieved from Google and Bing. The overlap in domains was about 29%, and Google had more exclusive domains. When looking at the result sets without considering the positions, the similarity between the result sets increased. This indicates that Google and Bing have different ranking preferences but index mostly the same sources. Similarly, Agrawal et al. (2016) based their overlap analysis on content but factored in the description of the result snippets. The top 10 results of 67 informational queries were collected from Google and Bing. The results indicate a high overlap between Google and Bing, with a slightly higher similarity among the top 5 results. Most recently, a study about information on Covid-19 was conducted by Makhortykh et al. (2020). The first 50 results for queries in different languages were collected from Baidu, Bing, DuckDuckGo, Google, Yandex, and Yahoo. DuckDuckGo and Yahoo shared nearly 50% of their results, whereas the other pairs remained under 25%. Google had an overlap of around 10% with Bing and a negligible one with DuckDuckGo, Yandex, Yahoo, and Baidu. Source diversity Various studies took a closer look at search results and evaluated the domains, the types of sources, as well as their diversity across result sets and search engines. Comparing Google, Live Search, and Yahoo, Thelwall (2008) reported that Yahoo returned the highest amount of different domains to a query and also returned around 10% more top-level domains than other search engines. Höchstötter & Lewandowski (2009) showed that while the sources in top results of various search engines differ, there is a concentration on some popular sources in top results. More recently, Lewandowski & Sünkler (2019) collected search results on queries related to insurance providers and identified the distribution of top-level domains. Only ten different domains were found in the first result across all queries. The five most popular domains in the top 5 results are price comparison websites, which make up 88.4% of all first results. The share of popular domains decreases according to the ranking but remains at 42.9% at position ten. Unkel and Haim (2019) observed Google search results prior to the German parliamentary elections in 2017. The study found that result lists for candidates and parties exhibit a high share of self-administered websites. In contrast, results about election facts, guidance, and issues are mainly from general interest news, government information, and privately run websites. Among the top ten domains across all search results are seven news websites which account for a quarter of all search results. By contrast, Wikipedia made up 5.4% of all search results. The high share of news and Wikipedia articles is confirmed by Steiner et al. (2022) who compared the first results for queries on debated topics in Germany across the search engines Ask, Bing, DuckDuckGo, Google, and Ixquick. All search engines positioned Wikipedia in the first rank for queries on climate change and the Transatlantic Trade and Investment Partnership (TTIP); Ask did so for almost every query. In the other search engines, news sites were placed in the first rank most of the time. However, for the topic Covid-19, the sources are more diverse. As Makhortykh et al. (2020) observed, only Yahoo incorporates recent information from legacy media, whereas Bing strongly relied on healthcare-related sources and Google highlighted government-related websites. Yandex was the only search engine that included alternative media in its top 20 search results. Especially for recent topics that lack an established knowledge basis, the choice of search engine may considerably impact what information a user gets to see. This paper is available on arxiv under CC 4.0 license. Authors: (1) Yagci, Nurce, HAW Hamburg, Germany & nurce.yagci@haw-hamburg.de; (2) Sünkler, Sebastian, HAW Hamburg, Germany & sebastian.suenkler@haw-hamburg.de; (3) Häußler, Helena, HAW Hamburg, Germany & helena.haeuessler@haw-hamburg.de; (4) Lewandowski, Dirk, HAW Hamburg, Germany & dirk.lewandowski@haw-hamburg.de. Authors: (1) Yagci, Nurce, HAW Hamburg, Germany & nurce.yagci@haw-hamburg.de; (2) Sünkler, Sebastian, HAW Hamburg, Germany & sebastian.suenkler@haw-hamburg.de; (3) Häußler, Helena, HAW Hamburg, Germany & helena.haeuessler@haw-hamburg.de; (4) Lewandowski, Dirk, HAW Hamburg, Germany & dirk.lewandowski@haw-hamburg.de. Table of Links Abstract and Introduction Abstract and Introduction Literature Review Literature Review Objectives and Research Questions Objectives and Research Questions Methods Methods Results Results Discussion Discussion Conclusion, Research Data, Acknowledgments, and References Conclusion, Research Data, Acknowledgments, and References LITERATURE REVIEW Search result overlap By the mid-1990s, researching the overlap of search results between different search engines sparked interest for the purpose of estimating the size of the Web (Bharat, 1998; Ding & Marchionini, 1996; Lawrence & Giles, 1999). The generally small degrees of overlap indicated diverse underlying databases, each of limited size. Therefore, metasearch engines that combined the results of several search engines were meant to provide an additional value (Chignell et al., 1999) and attracted further research (i.e., Meng et al., 2002). Consequently, Spink et al. (Spink et al., 2006) included the meta search engine Dogpile in their extensive study on the overlap of search results. For a collection of 12,570 queries, the results found on the first search engine result page (SERP) of Ask Jeeves, Google, MSN Search, and Yahoo were captured. 84.9% of the results were unique to one search engine, while only 1.1% were shared by all search engines. Additionally, the low overlap between search engines also manifests in the ranking since only 7% of the top results were similar. This is consistent with previous studies that reported low overlap in ranking (Bar-Ilan, 2005; Bar‐Ilan et al., 2006) Subsequent studies report an increase of overlap in search results, while ranking algorithms appear to be the leading cause for differences in the presentation of the results. Bilal and Ellis (2011) compared major web search engines (Google, Bing, Yahoo) with search engines specifically designed for children (Ask Kids, Yahoo Kids) for queries of various lengths. Bing and Yahoo shared the same overlap with Google for nearly all queries, ranging between 22% and 40%. In contrast, Yahoo Kids had mostly unique results. However, the disparate relevance ranking between Bing and Yahoo suggests different ranking algorithms. Cardoso and Magalhães (2011) measured the overlap of search result rankings based on URLs and website contents. Search results for 40,000 queries were retrieved from Google and Bing. The overlap in domains was about 29%, and Google had more exclusive domains. When looking at the result sets without considering the positions, the similarity between the result sets increased. This indicates that Google and Bing have different ranking preferences but index mostly the same sources. Similarly, Agrawal et al. (2016) based their overlap analysis on content but factored in the description of the result snippets. The top 10 results of 67 informational queries were collected from Google and Bing. The results indicate a high overlap between Google and Bing, with a slightly higher similarity among the top 5 results. Most recently, a study about information on Covid-19 was conducted by Makhortykh et al. (2020). The first 50 results for queries in different languages were collected from Baidu, Bing, DuckDuckGo, Google, Yandex, and Yahoo. DuckDuckGo and Yahoo shared nearly 50% of their results, whereas the other pairs remained under 25%. Google had an overlap of around 10% with Bing and a negligible one with DuckDuckGo, Yandex, Yahoo, and Baidu. Source diversity Various studies took a closer look at search results and evaluated the domains, the types of sources, as well as their diversity across result sets and search engines. Comparing Google, Live Search, and Yahoo, Thelwall (2008) reported that Yahoo returned the highest amount of different domains to a query and also returned around 10% more top-level domains than other search engines. Höchstötter & Lewandowski (2009) showed that while the sources in top results of various search engines differ, there is a concentration on some popular sources in top results. More recently, Lewandowski & Sünkler (2019) collected search results on queries related to insurance providers and identified the distribution of top-level domains. Only ten different domains were found in the first result across all queries. The five most popular domains in the top 5 results are price comparison websites, which make up 88.4% of all first results. The share of popular domains decreases according to the ranking but remains at 42.9% at position ten. Unkel and Haim (2019) observed Google search results prior to the German parliamentary elections in 2017. The study found that result lists for candidates and parties exhibit a high share of self-administered websites. In contrast, results about election facts, guidance, and issues are mainly from general interest news, government information, and privately run websites. Among the top ten domains across all search results are seven news websites which account for a quarter of all search results. By contrast, Wikipedia made up 5.4% of all search results. The high share of news and Wikipedia articles is confirmed by Steiner et al. (2022) who compared the first results for queries on debated topics in Germany across the search engines Ask, Bing, DuckDuckGo, Google, and Ixquick. All search engines positioned Wikipedia in the first rank for queries on climate change and the Transatlantic Trade and Investment Partnership (TTIP); Ask did so for almost every query. In the other search engines, news sites were placed in the first rank most of the time. However, for the topic Covid-19, the sources are more diverse. As Makhortykh et al. (2020) observed, only Yahoo incorporates recent information from legacy media, whereas Bing strongly relied on healthcare-related sources and Google highlighted government-related websites. Yandex was the only search engine that included alternative media in its top 20 search results. Especially for recent topics that lack an established knowledge basis, the choice of search engine may considerably impact what information a user gets to see. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Examining Source Diversity in Web Search: A Comparative Study of Top Search Engines

Authors:
(1) Yagci, Nurce, HAW Hamburg, Germany & nurce.yagci@haw-hamburg.de;
(2) Sünkler, Sebastian, HAW Hamburg, Germany & sebastian.suenkler@haw-hamburg.de;
(3) Häußler, Helena, HAW Hamburg, Germany & helena.haeuessler@haw-hamburg.de;
(4) Lewandowski, Dirk, HAW Hamburg, Germany & dirk.lewandowski@haw-hamburg.de. Table of Links Abstract and Introduction Literature Review Objectives and Research Questions Methods Results Discussion Conclusion, Research Data, Acknowledgments, and References ABSTRACT When it comes to search engines, users generally prefer Google. Our study aims to find the differences between the results found in Google compared to other search engines. We compared the top 10 results from Google, Bing, DuckDuckGo, and Metager, using 3,537 queries generated from Google Trends from Germany and the US. Google displays more unique domains in the top results than its competitors. Wikipedia and news websites are the most popular sources overall. With some top sources dominating search results, the distribution of domains is also consistent across all search engines. The overlap between Google and Bing is always under 32%, while Metager has a higher overlap with Bing than DuckDuckGo, going up to 78%. This study shows that the use of another search engine, especially in addition to Google, provides a wider variety in sources and might lead the user to find new perspectives. KEYWORDS Web search; search engine; web scraping; Google; source comparison INTRODUCTION Why should there be more than one search engine? While users may prefer one search engine over others for its usability, specialized features, or a more convenient integration into their technical environment, the question that interests us in this research is whether a user will benefit from using another search engine than Google when it comes to finding results from different sources. Our starting point is the fact that Google is the most-used search engine by far (StatCounter, 2022), that user to a large degree trust search engines to provide them relevant and useful results (European Commission, 2016; Purcell et al., 2012), and that only some users use another search engine in addition to Google (Schultheiß & Lewandowski, 2021). Users place great trust in search engines. This is reflected by the 91% of US users who said they find what they are looking for always or most of the time, and the 66% who believe search engines are a fair and unbiased source of information (Purcell et al., 2012). Furthermore, 78% of European internet and online platform users said they trust that their search engine results are the most relevant results (European Commission, 2016). Globally, users trust search engines more than any other source (including traditional news outlets) when it comes to news (Edelman Trust Institute, 2022) and users trust news found via search significantly more than news found on social media (Newman et al., 2021). As the Web is enormous and different search engines might prefer different sources, it is interesting to see whether the top sources shown in search results differ from one search engine to the other. It might be that an alternative search engine prefers results from "alternative" sources, e.g., in terms of political leaning or preferring noncommercial content providers. This all  comes down to whether alternative search engines are actually alternatives in regards to the results they display. If they were, possible benefits of using a search engine other than Google include finding different results, finding additional results, and finding more relevant results. No matter which of these goals a user aims to achieve, they will need other results than Google's. Therefore, it is interesting to see whether other search engines provide users with such results. There has been an ongoing discussion on alternative search engines and how Google's dominance in the search engine market can be broken. Approaches range from establishing single alternative search engines to building infrastructures for such alternatives (e.g., Lewandowski, 2019); also see Mager, 2014). With Google dominating the search engine market (StatCounter, 2022), it often seems that there are no alternatives at all. On the other hand, the number of alternative (or simply "other") search engines is often overestimated. Many seem-to-be search engines are merely search portals displaying results from a partner instead of generating the results from their own index. For instance, Yahoo and Ecosia get their results from Bing and can therefore not be considered search engines in their own right. But still, there may be other reasons for using a search engine without its own index. Some of the unique benefits alternative search engines advertise are privacy (e.g., Startpage and DuckDuckGo) or being a company investing its profits in environmental projects (e.g., Ecosia). Another type of search engine is the meta search engine (e.g., Metager). Such an engine sends the queries to several other search engines, then aggregates and re-ranks the top results. We deem it especially interesting whether such an approach will lead to a wider variety of search results, i.e., results from a more diverse set of sources. So, in the context of our research, we will consider any search engine that either has its own index or provides a unique selection and re-ranking of results from one or more indexes as an alternative search engine. We are especially interested in the differences in the source distribution; the relevance of the results is out of the scope of our research. More than 20 years ago, Introna & Nissenbaum (2000) argued that search engines as commercial operations tend to prefer big websites and, therefore, a portion of the Web, i.e., the smaller sites, remain hidden from view. Studies measuring what users select seem to confirm this: Goel et al. (2010) found that within Yahoo, only 10,000 websites account for approximately 80% of result clicks. It is important to note that this does not merely result from user preference for particular sources but that users predominantly select from the top results shown by the search engine. What is out of the immediate view of users will not be chosen (Lewandowski & Kammerer, 2021). It is striking that few studies have compared the results between different search engines in recent years. Older studies (see literature review section) overall found that top results from different search engines did not overlap too much. In this paper, we address how the top results of Google differ from alternatives and, therefore, whether it is worthwhile for a user to consider these  alternatives. If a search engine other than Google produced very similar results to Google, a user would not benefit much from using that search engine when source variety is considered. This paper is available on arxiv under CC 4.0 license.
Lead image by Justin Morgan on Unsplash Authors: (1) Yagci, Nurce, HAW Hamburg, Germany & nurce.yagci@haw-hamburg.de; (2) Sünkler, Sebastian, HAW Hamburg, Germany & sebastian.suenkler@haw-hamburg.de; (3) Häußler, Helena, HAW Hamburg, Germany & helena.haeuessler@haw-hamburg.de; (4) Lewandowski, Dirk, HAW Hamburg, Germany & dirk.lewandowski@haw-hamburg.de. Authors: (1) Yagci, Nurce, HAW Hamburg, Germany & nurce.yagci@haw-hamburg.de; (2) Sünkler, Sebastian, HAW Hamburg, Germany & sebastian.suenkler@haw-hamburg.de; (3) Häußler, Helena, HAW Hamburg, Germany & helena.haeuessler@haw-hamburg.de; (4) Lewandowski, Dirk, HAW Hamburg, Germany & dirk.lewandowski@haw-hamburg.de. Table of Links Abstract and Introduction Abstract and Introduction Literature Review Literature Review Objectives and Research Questions Objectives and Research Questions Methods Methods Results Results Discussion Discussion Conclusion, Research Data, Acknowledgments, and References Conclusion, Research Data, Acknowledgments, and References ABSTRACT When it comes to search engines, users generally prefer Google. Our study aims to find the differences between the results found in Google compared to other search engines. We compared the top 10 results from Google, Bing, DuckDuckGo, and Metager, using 3,537 queries generated from Google Trends from Germany and the US. Google displays more unique domains in the top results than its competitors. Wikipedia and news websites are the most popular sources overall. With some top sources dominating search results, the distribution of domains is also consistent across all search engines. The overlap between Google and Bing is always under 32%, while Metager has a higher overlap with Bing than DuckDuckGo, going up to 78%. This study shows that the use of another search engine, especially in addition to Google, provides a wider variety in sources and might lead the user to find new perspectives. KEYWORDS Web search; search engine; web scraping; Google; source comparison INTRODUCTION Why should there be more than one search engine? While users may prefer one search engine over others for its usability, specialized features, or a more convenient integration into their technical environment, the question that interests us in this research is whether a user will benefit from using another search engine than Google when it comes to finding results from different sources. Our starting point is the fact that Google is the most-used search engine by far (StatCounter, 2022), that user to a large degree trust search engines to provide them relevant and useful results (European Commission, 2016; Purcell et al., 2012), and that only some users use another search engine in addition to Google (Schultheiß & Lewandowski, 2021). Users place great trust in search engines. This is reflected by the 91% of US users who said they find what they are looking for always or most of the time, and the 66% who believe search engines are a fair and unbiased source of information (Purcell et al., 2012). Furthermore, 78% of European internet and online platform users said they trust that their search engine results are the most relevant results (European Commission, 2016). Globally, users trust search engines more than any other source (including traditional news outlets) when it comes to news (Edelman Trust Institute, 2022) and users trust news found via search significantly more than news found on social media (Newman et al., 2021). As the Web is enormous and different search engines might prefer different sources, it is interesting to see whether the top sources shown in search results differ from one search engine to the other. It might be that an alternative search engine prefers results from "alternative" sources, e.g., in terms of political leaning or preferring noncommercial content providers. This all  comes down to whether alternative search engines are actually alternatives in regards to the results they display. If they were, possible benefits of using a search engine other than Google include finding different results, finding additional results, and finding more relevant results. No matter which of these goals a user aims to achieve, they will need other results than Google's. Therefore, it is interesting to see whether other search engines provide users with such results. There has been an ongoing discussion on alternative search engines and how Google's dominance in the search engine market can be broken. Approaches range from establishing single alternative search engines to building infrastructures for such alternatives (e.g., Lewandowski, 2019); also see Mager, 2014). With Google dominating the search engine market (StatCounter, 2022), it often seems that there are no alternatives at all. On the other hand, the number of alternative (or simply "other") search engines is often overestimated. Many seem-to-be search engines are merely search portals displaying results from a partner instead of generating the results from their own index. For instance, Yahoo and Ecosia get their results from Bing and can therefore not be considered search engines in their own right. But still, there may be other reasons for using a search engine without its own index. Some of the unique benefits alternative search engines advertise are privacy (e.g., Startpage and DuckDuckGo) or being a company investing its profits in environmental projects (e.g., Ecosia). Another type of search engine is the meta search engine (e.g., Metager). Such an engine sends the queries to several other search engines, then aggregates and re-ranks the top results. We deem it especially interesting whether such an approach will lead to a wider variety of search results, i.e., results from a more diverse set of sources. So, in the context of our research, we will consider any search engine that either has its own index or provides a unique selection and re-ranking of results from one or more indexes as an alternative search engine. We are especially interested in the differences in the source distribution; the relevance of the results is out of the scope of our research. More than 20 years ago, Introna & Nissenbaum (2000) argued that search engines as commercial operations tend to prefer big websites and, therefore, a portion of the Web, i.e., the smaller sites, remain hidden from view. Studies measuring what users select seem to confirm this: Goel et al. (2010) found that within Yahoo, only 10,000 websites account for approximately 80% of result clicks. It is important to note that this does not merely result from user preference for particular sources but that users predominantly select from the top results shown by the search engine. What is out of the immediate view of users will not be chosen (Lewandowski & Kammerer, 2021). It is striking that few studies have compared the results between different search engines in recent years. Older studies (see literature review section) overall found that top results from different search engines did not overlap too much. In this paper, we address how the top results of Google differ from alternatives and, therefore, whether it is worthwhile for a user to consider these  alternatives. If a search engine other than Google produced very similar results to Google, a user would not benefit much from using that search engine when source variety is considered. This paper is available on arxiv under CC 4.0 license. Lead image by Justin Morgan on Unsplash This paper is available on arxiv under CC 4.0 license. available on arxiv Lead image by Justin Morgan on Unsplash Justin Morgan on Unsplash

A Comparison of Source Distribution and Result Overlap in Web Search Engines

ওয়েব সার্চ ইঞ্জিনে সোর্স ডিস্ট্রিবিউশন এবং ফলাফল ওভারল্যাপের তুলনা

Ein Vergleich der Quellenverteilung und der Ergebnisüberschneidung bei Websuchmaschinen

Una comparación de la distribución de fuentes y la superposición de resultados en los motores de búsqueda web

Une comparaison de la distribution des sources et du chevauchement des résultats dans les moteurs de recherche Web

वेब सर्च इंजन में स्रोत वितरण और परिणाम ओवरलैप की तुलना

ウェブ検索エンジンにおけるソース分布と結果重複の比較

웹 검색엔진의 소스 분포와 결과 중첩 비교

Uma comparação entre distribuição de fontes e sobreposição de resultados em mecanismos de pesquisa na web

Сравнение распределения источников и перекрытия результатов в поисковых системах веб-сайтов

Web Arama Motorlarında Kaynak Dağılımı ve Sonuç Örtüşmesinin Karşılaştırılması

So sánh sự phân phối nguồn và sự chồng chéo kết quả trong các công cụ tìm kiếm trên web

网络搜索引擎中源分布和结果重叠的比较

NEW YORK, New York, May 7th, 2024/Chainwire/-- Spectral is excited to announce its participation in Hugging Face's Expert Support Program. Spectral is collaborating with deep learning experts from Hugging Face to advance open-source models, datasets, and applications for the Onchain Agent Economy. How To Use Hugging Face Hugging Face plays a critical role across Spectral's organization. Through the Expert Support Program, Spectral regularly meets with the Hugging Face team to plan new use cases, strategize dataset construction, and develop training strategies. Earlier this year, Spectral launched Syntax, which is composed of a fine-tuned LLM orchestrator that routes queries between several tools, including search, Foundry, and an open-source model to generate Solidity code. Building Syntax required custom datasets for both finetuning and RAG, an original Solidity evaluation dataset written by elite smart contract developers, and experimentation across a wide range of open and closed-source models. Spectral's approach to dataset preparation, training, routing, and evaluation has drastically improved during this process, largely due to help from Hugging Face experts. Regular discussions on Spectral's approach mean that the company always receives impartial feedback on its decisions and can reevaluate its strategy as needed. This was important for both the finetuning and evaluation steps, which have now gone through several rounds of improvement thanks to the Hugging Face team. Spectral also relies on Hugging Face for inferences in production through dedicated Inference Endpoints. The fully managed infrastructure allows Spectral to iterate on new models quickly, easily update container and hardware configurations, autoscale with demand, and keep production costs low. Over the coming months, Spectral is excited to continue to open-source its work with onchain datasets and models that interact onchain through Hugging Face. Users can view Spectral's training dataset for the credit scoring challenge and follow its progress here. The Onchain Agent Economy Syntax is pioneering the accessible onchain Agent Economy, inviting users to select agents tailored for their specific tasks. Users can either interact with the foundational agent to generate solidity code or opt for one of the specialized agents, each adept in distinct tasks. For example, we recently launched Syntax MoonMaker, an agent that launches a memecoin project end to end and are soon launching Syntax- TestMachine, an agent that detects vulnerabilities in your smart contracts. Many such agents are currently in production, and upcoming releases of our product will allow users to create their own agents and monetize them on the Syntax network. This system relies on a vast suite of functions available to language models, open-source models for tasks like classification and image generation, fast and reliable blockchain infrastructure, and a trustless system to execute and verify each step. To solve these problems, we've begun developing the Inferchain, optimized to serve the rapidly growing demand for agents and verifiable inferences. Hugging Face Users Joining The Onchain AI Community The importance of open collaboration cannot be overstated. Like many other projects, Spectral would not exist without the work of open-source contributors. Spectral is greatly indebted to everyone who makes this work possible and believes this positive impact should be recognized. As part of this engagement, Spectral is focusing specifically on users powering the AI community. The top individual and small organization contributors on Hugging Face, measured by likes and downloads of their models and datasets, are eligible to register to claim a share of SPEC tokens starting the week of May 6th. Users can check eligibility by signing in with their Hugging Face account here. A snapshot of activity was taken last month, and eligible users will not need an existing wallet to claim. This airdrop is designed to recognize users for their critical work and enable new functionality for the next generation of AI tools. Criteria Spectral reviewed all individual contributors from organizations with less than 10 members that published models or datasets. From these, Spectral took a combined measure of activity from likes and downloads for both models and datasets. Then, Spectral identified the users with the most combined activity from both model and dataset publishers (11298 in both categories) which resulted in a final list of 20,004 unique huggingface users. Spectral is looking forward to engaging further with the open-source community and building the future of AI x web3! https://claims.spectrallabs.xyz/ About Spectral Spectral, a pioneer of the agent economy behind Syntax, is at the forefront of integrating AI with blockchain to democratize development in Web3. Spectral's mission is to simplify the creation and deployment of decentralized applications through autonomous Onchain Agents. Syntax, Spectral’s flagship product, translates natural language into Solidity code, enabling both novices and experts to build on the blockchain effortlessly. With a commitment to transparency and user empowerment, Spectral is shaping a future where anyone can participate in the blockchain revolution. Join Spectral in making this vision a reality and explore the potential of autonomous agents. For more information, visit https://spectrallabs.xyz. For more information, users can follow Spectral on Twitter and Discord. Contact Spectral Labs Team contact@spectral.finance This story was distributed as a release by Chainwire under HackerNoon’s Business Blogging Program. Learn more about the program here. NEW YORK, New York, May 7th, 2024/Chainwire/-- Spectral is excited to announce its participation in Hugging Face's Expert Support Program. Spectral is collaborating with deep learning experts from Hugging Face to advance open-source models, datasets, and applications for the Onchain Agent Economy. How To Use Hugging Face Hugging Face plays a critical role across Spectral's organization. Through the Expert Support Program, Spectral regularly meets with the Hugging Face team to plan new use cases, strategize dataset construction, and develop training strategies. Earlier this year, Spectral launched Syntax, which is composed of a fine-tuned LLM orchestrator that routes queries between several tools, including search, Foundry , and an open-source model to generate Solidity code. Foundry Foundry an open-source model an open-source model Building Syntax required custom datasets for both finetuning and RAG, an original Solidity evaluation dataset written by elite smart contract developers, and experimentation across a wide range of open and closed-source models. Spectral's approach to dataset preparation, training, routing, and evaluation has drastically improved during this process, largely due to help from Hugging Face experts. Regular discussions on Spectral's approach mean that the company always receives impartial feedback on its decisions and can reevaluate its strategy as needed. This was important for both the finetuning and evaluation steps, which have now gone through several rounds of improvement thanks to the Hugging Face team. Spectral also relies on Hugging Face for inferences in production through dedicated Inference Endpoints. The fully managed infrastructure allows Spectral to iterate on new models quickly, easily update container and hardware configurations, autoscale with demand, and keep production costs low. Over the coming months, Spectral is excited to continue to open-source its work with onchain datasets and models that interact onchain through Hugging Face. Users can view Spectral's training dataset for the credit scoring challenge and follow its progress here . here here The Onchain Agent Economy Syntax is pioneering the accessible onchain Agent Economy, inviting users to select agents tailored for their specific tasks. Users can either interact with the foundational agent to generate solidity code or opt for one of the specialized agents, each adept in distinct tasks. For example, we recently launched Syntax MoonMaker, an agent that launches a memecoin project end to end and are soon launching Syntax- TestMachine, an agent that detects vulnerabilities in your smart contracts. Many such agents are currently in production, and upcoming releases of our product will allow users to create their own agents and monetize them on the Syntax network. This system relies on a vast suite of functions available to language models, open-source models for tasks like classification and image generation, fast and reliable blockchain infrastructure, and a trustless system to execute and verify each step. To solve these problems, we've begun developing the Inferchain, optimized to serve the rapidly growing demand for agents and verifiable inferences. Hugging Face Users Joining The Onchain AI Community The importance of open collaboration cannot be overstated. Like many other projects, Spectral would not exist without the work of open-source contributors. Spectral is greatly indebted to everyone who makes this work possible and believes this positive impact should be recognized. As part of this engagement, Spectral is focusing specifically on users powering the AI community. The top individual and small organization contributors on Hugging Face, measured by likes and downloads of their models and datasets, are eligible to register to claim a share of SPEC tokens starting the week of May 6th. Users can check eligibility by signing in with their Hugging Face account here . here here A snapshot of activity was taken last month, and eligible users will not need an existing wallet to claim. This airdrop is designed to recognize users for their critical work and enable new functionality for the next generation of AI tools. Criteria Spectral reviewed all individual contributors from organizations with less than 10 members that published models or datasets. From these, Spectral took a combined measure of activity from likes and downloads for both models and datasets. Then, Spectral identified the users with the most combined activity from both model and dataset publishers (11298 in both categories) which resulted in a final list of 20,004 unique huggingface users. Spectral is looking forward to engaging further with the open-source community and building the future of AI x web3! https://claims.spectrallabs.xyz/ https://claims.spectrallabs.xyz/ https://claims.spectrallabs.xyz/ About Spectral Spectral , a pioneer of the agent economy behind Syntax, is at the forefront of integrating AI with blockchain to democratize development in Web3. Spectral's mission is to simplify the creation and deployment of decentralized applications through autonomous Onchain Agents. Spectral Spectral Syntax, Spectral’s flagship product, translates natural language into Solidity code, enabling both novices and experts to build on the blockchain effortlessly. With a commitment to transparency and user empowerment, Spectral is shaping a future where anyone can participate in the blockchain revolution. Join Spectral in making this vision a reality and explore the potential of autonomous agents. For more information, visit https://spectrallabs.xyz . For more information, users can follow Spectral on Twitter and Discord . https://spectrallabs.xyz Twitter Twitter Discord Discord Contact Spectral Labs Team contact@spectral.finance This story was distributed as a release by Chainwire under HackerNoon’s Business Blogging Program. Learn more about the program here. This story was distributed as a release by Chainwire under HackerNoon’s Business Blogging Program. Learn more about the program here . here here

Spectral Labs Joins Hugging Face’s ESP Program: Advancing The Onchain x Open-Source AI Community

স্পেকট্রাল ল্যাবগুলি আলিঙ্গন মুখের ইএসপি প্রোগ্রামে যোগ দেয়: অনচেইন এক্স ওপেন-সোর্স এআই সম্প্রদায়কে এগিয়ে নিয়ে যাওয়া

Spectral Labs nimmt am ESP-Programm von Hugging Face teil: Förderung der Onchain x Open-Source-KI-Community

Spectral Labs se une al programa ESP de Hugging Face: avanzando en la comunidad de IA de código abierto Onchain x

Spectral Labs rejoint le programme ESP de Hugging Face : faire progresser la communauté d'IA Onchain x Open-Source

स्पेक्ट्रल लैब्स हगिंग फेस के ईएसपी कार्यक्रम में शामिल हुई: ऑनचेन एक्स ओपन-सोर्स एआई समुदाय को आगे बढ़ाना

Spectral Labs が Hugging Face の ESP プログラムに参加: オンチェーン x オープンソース AI コミュニティの発展

Spectral Labs, Hugging Face의 ESP 프로그램에 합류: Onchain x 오픈 소스 AI 커뮤니티 발전

Spectral Labs junta-se ao programa ESP da Hugging Face: Avançando a comunidade Onchain x Open-Source AI

Spectral Labs присоединяется к программе ESP Hugging Face: развитие сообщества искусственного интеллекта с открытым исходным кодом Onchain x

Spectral Labs, Hugging Face'in ESP Programına Katılıyor: Onchain x Açık Kaynak Yapay Zeka Topluluğunu Geliştirme

Spectral Labs tham gia chương trình ESP của Hugging Face: Thúc đẩy cộng đồng AI mã nguồn mở Onchain x

Spectral Labs 加入 Hugging Face 的 ESP 计划：推进 Onchain x 开源 AI 社区

OpenAI is in a high-stakes legal battle with Elon Musk, who is suing the company to open source its groundbreaking AI models like GPT-4. While OpenAI may have the law on its side, I believe the ethical path forward is clear—it's time for the company to fully embrace the open-source ethos it was founded upon. As a former AI Specialist at OpenAI, I know OpenAI's stated mission has always been to ensure that artificial intelligence benefits all of humanity. The company's charter explicitly prioritizes this goal above generating financial returns. With the rapid rise of AI and the very real prospect of widespread job displacement, now is the time for OpenAI to double down on this mission through radical collaboration and transparency. Elon Musk helped found OpenAI precisely to keep advanced AI out of the exclusive control of a few tech giants or governments. The fear was that highly centralized, opaque AI development could lead to biased systems that disempower people and exacerbate inequality. Making frontier models like GPT-4 open source would accelerate innovation across the field and give everyone a stake in steering this transformative technology in a positive direction. Instead, OpenAI and its partner Microsoft now boast that “we are waiting for the competition to arrive. It will arrive, I’m sure, but the fact [is] that we have the … leading LLM out there.” Their most powerful models, including GPT-4, DALL-E, and now Sora, are all closed-source and gated. Entire countries like China and Russia are restricted from using OpenAI’s technologies. Accountability Through Openness Open-sourcing its models would also keep OpenAI honest and accountable. External researchers could audit the systems for hidden biases, safety issues, and potential misuse. The global AI ethics community could collaborate to make the models more inclusive and aligned with diverse human values. Perhaps even more importantly, the immense economic benefits of AI could be more evenly distributed if the core infrastructure is treated as a public good rather than the intellectual property of a few tech giants. While OpenAI has published research papers and shared some tools, it has stopped short of fully open-sourcing its most powerful models. Meanwhile, other prominent AI labs like H2O.ai have embraced a more open approach from the start. For instance, H2O recently released an open-source LLM called Danube-2, which can be downloaded in full from HuggingFace. Sri Ambati, CEO and founder of H2O.ai, said at H2O.ai’s GenAI World keynote that "AGI should be open source and in the public domain, at the service of humanity and the planet.” Even big tech players like Meta are exploring open-source AGI and giving outside researchers and the public more visibility into and agency over their AI work. OpenAI risks falling behind this trend and betraying its founding principles if it continues to treat its state-of-the-art models as proprietary secrets. The Business Case for Collaboration From a business standpoint, I understand the temptation for OpenAI to maintain exclusive control over its technology. But the history of the software industry shows that open source tends to win out in the long run through faster collaborative innovation and community support. Linux and the open-source movement conquered the once-dominant proprietary operating systems. Wikipedia displaced the traditional encyclopedia industry by crowdsourcing knowledge. With highly capable open source competitors to GPT-4 emerging, OpenAI's best path forward may be to focus on its expertise in scaling and commercializing a community-driven ecosystem rather than clinging to secrecy. Taking the High Road The transition to an open-source model would undoubtedly be challenging, but it's the right thing to do. OpenAI's leaders should return to their charter, reflect deeply on the kind of world they want to create, and take the bold path of openness and collaboration. Making advanced AI serve everyone is a historic challenge and opportunity. In the face of this technology's epochal impact on society, anything less than full transparency would be a dereliction of OpenAI's founding duty. I urge the company's leaders to resolve any legal disputes swiftly, open source their most important models, and help catalyze a global effort to develop AI as a public good. Democratizing access to AI is how we'll create the best outcomes for humanity. It's time for OpenAI to walk the walk of its open-source mission. OpenAI is in a high-stakes legal battle with Elon Musk, who is suing the company to open source its groundbreaking AI models like GPT-4. While OpenAI may have the law on its side , I believe the ethical path forward is clear—it's time for the company to fully embrace the open-source ethos it was founded upon. legal battle legal battle on its side on its side As a former AI Specialist at OpenAI, I know OpenAI's stated mission has always been to ensure that artificial intelligence benefits all of humanity. The company's charter explicitly prioritizes this goal above generating financial returns. With the rapid rise of AI and the very real prospect of widespread job displacement, now is the time for OpenAI to double down on this mission through radical collaboration and transparency. charter charter Elon Musk helped found OpenAI precisely to keep advanced AI out of the exclusive control of a few tech giants or governments. The fear was that highly centralized, opaque AI development could lead to biased systems that disempower people and exacerbate inequality. Making frontier models like GPT-4 open source would accelerate innovation across the field and give everyone a stake in steering this transformative technology in a positive direction. found OpenAI found OpenAI Instead, OpenAI and its partner Microsoft now boast that “we are waiting for the competition to arrive. It will arrive, I’m sure, but the fact [is] that we have the … leading LLM out there.” Their most powerful models, including GPT-4, DALL-E, and now Sora, are all closed-source and gated. Entire countries like China and Russia are restricted from using OpenAI’s technologies. boast boast restricted restricted Accountability Through Openness Open-sourcing its models would also keep OpenAI honest and accountable. External researchers could audit the systems for hidden biases, safety issues, and potential misuse. The global AI ethics community could collaborate to make the models more inclusive and aligned with diverse human values. Perhaps even more importantly, the immense economic benefits of AI could be more evenly distributed if the core infrastructure is treated as a public good rather than the intellectual property of a few tech giants. While OpenAI has published research papers and shared some tools, it has stopped short of fully open-sourcing its most powerful models. Meanwhile, other prominent AI labs like H2O.ai have embraced a more open approach from the start. For instance, H2O recently released an open-source LLM called Danube-2 , which can be downloaded in full from HuggingFace. Sri Ambati, CEO and founder of H2O.ai, said at H2O.ai’s GenAI World keynote that "AGI should be open source and in the public domain, at the service of humanity and the planet.” LLM called Danube-2 LLM called Danube-2 H2O.ai’s GenAI World keynote H2O.ai’s GenAI World keynote Even big tech players like Meta are exploring open-source AGI and giving outside researchers and the public more visibility into and agency over their AI work. OpenAI risks falling behind this trend and betraying its founding principles if it continues to treat its state-of-the-art models as proprietary secrets. open-source AGI open-source AGI The Business Case for Collaboration From a business standpoint, I understand the temptation for OpenAI to maintain exclusive control over its technology. But the history of the software industry shows that open source tends to win out in the long run through faster collaborative innovation and community support. win out in the long run win out in the long run Linux and the open-source movement conquered the once-dominant proprietary operating systems. Wikipedia displaced the traditional encyclopedia industry by crowdsourcing knowledge. With highly capable open source competitors to GPT-4 emerging, OpenAI's best path forward may be to focus on its expertise in scaling and commercializing a community-driven ecosystem rather than clinging to secrecy. Taking the High Road The transition to an open-source model would undoubtedly be challenging, but it's the right thing to do. OpenAI's leaders should return to their charter, reflect deeply on the kind of world they want to create, and take the bold path of openness and collaboration. Making advanced AI serve everyone is a historic challenge and opportunity. In the face of this technology's epochal impact on society, anything less than full transparency would be a dereliction of OpenAI's founding duty. I urge the company's leaders to resolve any legal disputes swiftly, open source their most important models, and help catalyze a global effort to develop AI as a public good. Democratizing access to AI is how we'll create the best outcomes for humanity. It's time for OpenAI to walk the walk of its open-source mission.

Why OpenAI Should Become Open-Source

This paper is available on arxiv under CC BY-SA 4.0 DEED license.
Authors:
(1) Lana L. Blaschke, Earth System Modelling, School of Engineering and Design, Technical University of Munich, Munich, Germany, Potsdam Institute for Climate Impact Research, Potsdam, Germany and lana.blaschke@pik-potsdam.de;
(2) Da Nian, Potsdam Institute for Climate Impact Research, Potsdam, Germany;
(3) Sebastian Bathiany,  Earth System Modelling, School of Engineering and Design, Technical University of Munich, Munich, Germany, and Potsdam Institute for Climate Impact Research, Potsdam, Germany;
(4)  Maya Ben-Yami,  Earth System Modelling, School of Engineering and Design, Technical University of Munich, Munich, Germany, and Potsdam Institute for Climate Impact Research, Potsdam, Germany;
(5) Taylor Smith, Institute of Geosciences, University of Potsdam, 14476 Potsdam, Germany;
(6)  Chris A. Boulton, Global Systems Institute, University of Exeter, EX4 4QE Exeter, UK ;
(7) Niklas Boers, Earth System Modelling, School of Engineering and Design, Technical University of Munich, Munich, Germany, Potsdam Institute for Climate Impact Research, Potsdam, Germany, Global Systems Institute, University of Exeter, EX4 4QE Exeter, UK, and Department of Mathematics, University of Exeter, EX4 4QF Exeter, UK. Table of Links Abstract and Introduction
Materials and Methods
Data
Results
Conclusions, Open Research Section and Conflict of Interest/Competing Interests
Author Contributions, Supplementary Information, Acknowledgments and References
Additional Insight from the Conceptual Model 
AMSR2 until 2022 Excluded Due to Missing Human Land Use Data
Investigation of Resilience Loss in AMSR2’s X-Band
Results Proving Robustness
Potential Driving Forces and Sources of Bias POTENTIAL DRIVING FORCES AND SOURCES OF BIAS Text S12. The influence of the South American Monsoon system as a potentially bistable system In the Amazon basin, the South American monsoon system and the atmospheric moisture recycling form a complex and potentially bi-stable system. Hence the monthly precipitation itself could exhibit signs of CSD, which would then drive vegetation changes that are wrongly interpreted as the destabilization of the vegetation. In Figure S16, the signs of the trend of variance, AC1 and spatial are compared between VOD and precipitation (Funk et al., 2014). We argue that due to the high amount of cells where the values differ in sign for the VIs and precipitation (sum of diagonal cells in boxes expected to be 50 % for random assignment), the observed changes in the indicators are likely a measure of critical slowing down in the vegetation and not only a direct effect of variability in precipitation. Text S13. The influence of changes in precipitation as a potential driving force If precipitation is the main driver of ARF resilience loss, one would expect that in regions with stronger decreases in precipitation (negative trend), more critical slowing down can be observed. This comparison in displayed in Figure S17. In parts, decreased precipitation could potentially explain the destabilization detected in the western Amazon basin by both bands of AMSR-E as well as in the western and northeastern regions detected by AMSR2’s bands. However it is important to note here that the number of cells with an increasing amount of precipitation and decreasing indicators and those with decreasing amount of precipitation and increasing indicators add up to only about 50 %, which is what is to be expected for a random assignment of positive and negative trends in both. Text S14. The influence of changes in temperature Instead of decreasing precipitation (see Figure S17, an increase in temperature can also lead to vegetation destabilization through water stress by higher evapo-transpirative demand. If this was the driver of the CSD, increasing temperatures must coincide with positive trends in the indicators of CSD in Figure S18. Yet changes in temperature cannot explain the detected destabilization in the years 2002 to 2011. For the time period of AMSR2, destabilization detected in the northeast (band C1) and the east (band C2) might at least partially be accounted for rising temperatures (the percentage of cells with rising/declining temperatures and in-/decreasing indicators falls between 48.6 % and up to 59.3 %). Text S15. The potential bias induced by saturation In case of optical sensors, their saturation over high biomass regions often renders the VIs insensitive to the subtle changes due to perturbations that a resilience analysis based on CSD relies on. The argument is that with decreasing biomass, the VIs become more sensitive, which then translates into a signal of CSD as a direct artifact of the higher perception of perturbations. Even though VOD results from observations of microwaves, we want to address the question of saturation in VOD. As Figure S19 shows, we cannot exclude this as an artifact for AMSRE’s band C as well as AMSR2’s two bands, as they hardly observed any cells with increasing biomass. Interestingly, AMSR-E’s band X detected an increase in biomass as well as a loss of resilience in most of the cells (77.3 % to 78.2 %). Thus, its signal of destabilization of almost the entire ARF during the years 2002 to 2011 is not an artifact of saturation of the band. This paper is available on arxiv under CC BY-SA 4.0 DEED license. Authors: (1) Lana L. Blaschke, Earth System Modelling, School of Engineering and Design, Technical University of Munich, Munich, Germany, Potsdam Institute for Climate Impact Research, Potsdam, Germany and lana.blaschke@pik-potsdam.de; (2) Da Nian, Potsdam Institute for Climate Impact Research, Potsdam, Germany; (3) Sebastian Bathiany,  Earth System Modelling, School of Engineering and Design, Technical University of Munich, Munich, Germany, and Potsdam Institute for Climate Impact Research, Potsdam, Germany; (4)  Maya Ben-Yami,  Earth System Modelling, School of Engineering and Design, Technical University of Munich, Munich, Germany, and Potsdam Institute for Climate Impact Research, Potsdam, Germany; (5) Taylor Smith, Institute of Geosciences, University of Potsdam, 14476 Potsdam, Germany; (6)  Chris A. Boulton, Global Systems Institute, University of Exeter, EX4 4QE Exeter, UK ; (7) Niklas Boers, Earth System Modelling, School of Engineering and Design, Technical University of Munich, Munich, Germany, Potsdam Institute for Climate Impact Research, Potsdam, Germany, Global Systems Institute, University of Exeter, EX4 4QE Exeter, UK, and Department of Mathematics, University of Exeter, EX4 4QF Exeter, UK. This paper is available on arxiv under CC BY-SA 4.0 DEED license. Authors: Authors: (1) Lana L. Blaschke, Earth System Modelling, School of Engineering and Design, Technical University of Munich, Munich, Germany, Potsdam Institute for Climate Impact Research, Potsdam, Germany and lana.blaschke@pik-potsdam.de; (2) Da Nian, Potsdam Institute for Climate Impact Research, Potsdam, Germany; (3) Sebastian Bathiany,  Earth System Modelling, School of Engineering and Design, Technical University of Munich, Munich, Germany, and Potsdam Institute for Climate Impact Research, Potsdam, Germany; (4)  Maya Ben-Yami,  Earth System Modelling, School of Engineering and Design, Technical University of Munich, Munich, Germany, and Potsdam Institute for Climate Impact Research, Potsdam, Germany; (5) Taylor Smith, Institute of Geosciences, University of Potsdam, 14476 Potsdam, Germany; (6)  Chris A. Boulton, Global Systems Institute, University of Exeter, EX4 4QE Exeter, UK ; (7) Niklas Boers, Earth System Modelling, School of Engineering and Design, Technical University of Munich, Munich, Germany, Potsdam Institute for Climate Impact Research, Potsdam, Germany, Global Systems Institute, University of Exeter, EX4 4QE Exeter, UK, and Department of Mathematics, University of Exeter, EX4 4QF Exeter, UK. Table of Links Abstract and Introduction Materials and Methods Data Results Conclusions, Open Research Section and Conflict of Interest/Competing Interests Author Contributions, Supplementary Information, Acknowledgments and References Additional Insight from the Conceptual Model AMSR2 until 2022 Excluded Due to Missing Human Land Use Data Investigation of Resilience Loss in AMSR2’s X-Band Results Proving Robustness Potential Driving Forces and Sources of Bias Abstract and Introduction Abstract and Introduction Materials and Methods Materials and Methods Data Data Results Results Conclusions, Open Research Section and Conflict of Interest/Competing Interests Conclusions, Open Research Section and Conflict of Interest/Competing Interests Author Contributions, Supplementary Information, Acknowledgments and References Author Contributions, Supplementary Information, Acknowledgments and References Additional Insight from the Conceptual Model Additional Insight from the Conceptual Model AMSR2 until 2022 Excluded Due to Missing Human Land Use Data AMSR2 until 2022 Excluded Due to Missing Human Land Use Data Investigation of Resilience Loss in AMSR2’s X-Band Investigation of Resilience Loss in AMSR2’s X-Band Results Proving Robustness Results Proving Robustness Potential Driving Forces and Sources of Bias Potential Driving Forces and Sources of Bias POTENTIAL DRIVING FORCES AND SOURCES OF BIAS Text S12. The influence of the South American Monsoon system as a potentially bistable system In the Amazon basin, the South American monsoon system and the atmospheric moisture recycling form a complex and potentially bi-stable system. Hence the monthly precipitation itself could exhibit signs of CSD, which would then drive vegetation changes that are wrongly interpreted as the destabilization of the vegetation. In Figure S16, the signs of the trend of variance, AC1 and spatial are compared between VOD and precipitation (Funk et al., 2014). We argue that due to the high amount of cells where the values differ in sign for the VIs and precipitation (sum of diagonal cells in boxes expected to be 50 % for random assignment), the observed changes in the indicators are likely a measure of critical slowing down in the vegetation and not only a direct effect of variability in precipitation. Text S13. The influence of changes in precipitation as a potential driving force If precipitation is the main driver of ARF resilience loss, one would expect that in regions with stronger decreases in precipitation (negative trend), more critical slowing down can be observed. This comparison in displayed in Figure S17. In parts, decreased precipitation could potentially explain the destabilization detected in the western Amazon basin by both bands of AMSR-E as well as in the western and northeastern regions detected by AMSR2’s bands. However it is important to note here that the number of cells with an increasing amount of precipitation and decreasing indicators and those with decreasing amount of precipitation and increasing indicators add up to only about 50 %, which is what is to be expected for a random assignment of positive and negative trends in both. Text S14. The influence of changes in temperature Instead of decreasing precipitation (see Figure S17, an increase in temperature can also lead to vegetation destabilization through water stress by higher evapo-transpirative demand. If this was the driver of the CSD, increasing temperatures must coincide with positive trends in the indicators of CSD in Figure S18. Yet changes in temperature cannot explain the detected destabilization in the years 2002 to 2011. For the time period of AMSR2, destabilization detected in the northeast (band C1) and the east (band C2) might at least partially be accounted for rising temperatures (the percentage of cells with rising/declining temperatures and in-/decreasing indicators falls between 48.6 % and up to 59.3 %). Text S15. The potential bias induced by saturation In case of optical sensors, their saturation over high biomass regions often renders the VIs insensitive to the subtle changes due to perturbations that a resilience analysis based on CSD relies on. The argument is that with decreasing biomass, the VIs become more sensitive, which then translates into a signal of CSD as a direct artifact of the higher perception of perturbations. Even though VOD results from observations of microwaves, we want to address the question of saturation in VOD. As Figure S19 shows, we cannot exclude this as an artifact for AMSRE’s band C as well as AMSR2’s two bands, as they hardly observed any cells with increasing biomass. Interestingly, AMSR-E’s band X detected an increase in biomass as well as a loss of resilience in most of the cells (77.3 % to 78.2 %). Thus, its signal of destabilization of almost the entire ARF during the years 2002 to 2011 is not an artifact of saturation of the band.

Loss of Amazon Rainforest Resilience: Potential Driving Forces and Sources of Bias

AI-Powered Localization Using an Open-source Project as a Case Study

Let's build a chatbot with just Python using the Streamlit library, Ollama, and Microsoft Phi-3. Streamlit Streamlit turns data scripts into shareable web apps in minutes. All in pure Python. No front‑end experience required. You can find more info in the official Streamlit docs. Ollama Ollama allows you to run open-source large language models, locally. You can find more info in the official Ollama docs. Phi-3 Mini Phi-3 Mini is a 3.8B parameters, lightweight, state-of-the-art open model by Microsoft. You can find more info in the official Phi-3 Mini docs. Steps Create a new conda environment conda create --name envStreamPhi Activate the environment conda activate envStreamPhi Install ollama Download Ollama from https://ollama.com/download pull the phi-3 model The model we will be using is located here: ollama pull phi3 Pull the Embeddings model: ollama pull nomic-embed-text Test installation streamlit hello Build the AI Assistant In order to build the AI assistant, you have 2 choices: clone the repo and get all the code from the get-go, or code along with me. I - First option: Clone the project from GitHub git clone https://github.com/JalelTounsi/oLLaMaStreamPhi.git Run the Application streamlit run app.py II - Second Option: Code Along Create your app.py file Add Imports import streamlit as st
import ollama Add the Defacto Message if "messages" not in st.session_state:
    st.session_state["messages"] = [{"role": "assistant", "content": "Hello tehre, how can I help you, today?"}] Add the Message History for msg in st.session_state.messages:
    if msg["role"] == "user":
        st.chat_message(msg["role"], avatar="🧑‍💻").write(msg["content"])
    else:
        st.chat_message(msg["role"], avatar="🤖").write(msg["content"]) Configure Model def generate_response():
    response = ollama.chat(model='phi3', stream=True, messages=st.session_state.messages)
    for partial_resp in response:
        token = partial_resp["message"]["content"]
        st.session_state["full_message"] += token
        yield token Configure the Prompt if prompt := st.chat_input():
    st.session_state.messages.append({"role": "user", "content": prompt})
    st.chat_message("user", avatar="🧑‍💻").write(prompt)
    st.session_state["full_message"] = ""
    st.chat_message("assistant", avatar="🤖").write_stream(generate_response)
    st.session_state.messages.append({"role": "assistant", "content": st.session_state["full_message"]}) All the Codebase of app.py import streamlit as st
import ollama

st.title("💬 Phi3 Chatbot")

if "messages" not in st.session_state:
    st.session_state["messages"] = [{"role": "assistant", "content": "Hello tehre, how can I help you, today?"}]

### Write Message History
for msg in st.session_state.messages:
    if msg["role"] == "user":
        st.chat_message(msg["role"], avatar="🧑‍💻").write(msg["content"])
    else:
        st.chat_message(msg["role"], avatar="🤖").write(msg["content"])

## Configure the model
def generate_response():
    response = ollama.chat(model='phi3', stream=True, messages=st.session_state.messages)
    for partial_resp in response:
        token = partial_resp["message"]["content"]
        st.session_state["full_message"] += token
        yield token

if prompt := st.chat_input():
    st.session_state.messages.append({"role": "user", "content": prompt})
    st.chat_message("user", avatar="🧑‍💻").write(prompt)
    st.session_state["full_message"] = ""
    st.chat_message("assistant", avatar="🤖").write_stream(generate_response)
    st.session_state.messages.append({"role": "assistant", "content": st.session_state["full_message"]}) Run the Streamlit App streamlit run app.py Bonus Point Feel free to check this video to understand the code, follow along, and run the AI assistant locally on your computer: AI assistant using Streamlit, Phi3, and Ollama Let's build a chatbot with just Python using the Streamlit library, Ollama , and Microsoft Phi-3 . Streamlit Ollama Microsoft Phi-3 Streamlit Streamlit turns data scripts into shareable web apps in minutes. All in pure Python. No front‑end experience required. You can find more info in the official Streamlit docs . official Streamlit docs Ollama Ollama allows you to run open-source large language models, locally. You can find more info in the official Ollama docs . official Ollama Ollama docs Phi-3 Mini Phi-3 Mini is a 3.8B parameters, lightweight, state-of-the-art open model by Microsoft. You can find more info in the official Phi-3 Mini docs . Phi-3 Mini Phi-3 Mini docs Steps Create a new conda environment conda create --name envStreamPhi conda create --name envStreamPhi Activate the environment conda activate envStreamPhi conda activate envStreamPhi Install ollama Download Ollama from https://ollama.com/download https://ollama.com/download pull the phi-3 model The model we will be using is located here : here ollama pull phi3 ollama pull phi3 Pull the Embeddings model: ollama pull nomic-embed-text ollama pull nomic-embed-text Test installation streamlit hello streamlit hello Build the AI Assistant In order to build the AI assistant, you have 2 choices: clone the repo and get all the code from the get-go, or code along with me. I - First option: Clone the project from GitHub git clone https://github.com/JalelTounsi/oLLaMaStreamPhi.git git clone https://github.com/JalelTounsi/oLLaMaStreamPhi.git Run the Application streamlit run app.py streamlit run app.py II - Second Option: Code Along Create your app.py file app.py Add Imports import streamlit as st
import ollama import streamlit as st
import ollama Add the Defacto Message if "messages" not in st.session_state:
    st.session_state["messages"] = [{"role": "assistant", "content": "Hello tehre, how can I help you, today?"}] if "messages" not in st.session_state:
    st.session_state["messages"] = [{"role": "assistant", "content": "Hello tehre, how can I help you, today?"}] Add the Message History for msg in st.session_state.messages:
    if msg["role"] == "user":
        st.chat_message(msg["role"], avatar="🧑‍💻").write(msg["content"])
    else:
        st.chat_message(msg["role"], avatar="🤖").write(msg["content"]) for msg in st.session_state.messages:
    if msg["role"] == "user":
        st.chat_message(msg["role"], avatar="🧑‍💻").write(msg["content"])
    else:
        st.chat_message(msg["role"], avatar="🤖").write(msg["content"]) Configure Model def generate_response():
    response = ollama.chat(model='phi3', stream=True, messages=st.session_state.messages)
    for partial_resp in response:
        token = partial_resp["message"]["content"]
        st.session_state["full_message"] += token
        yield token def generate_response():
    response = ollama.chat(model='phi3', stream=True, messages=st.session_state.messages)
    for partial_resp in response:
        token = partial_resp["message"]["content"]
        st.session_state["full_message"] += token
        yield token Configure the Prompt if prompt := st.chat_input():
    st.session_state.messages.append({"role": "user", "content": prompt})
    st.chat_message("user", avatar="🧑‍💻").write(prompt)
    st.session_state["full_message"] = ""
    st.chat_message("assistant", avatar="🤖").write_stream(generate_response)
    st.session_state.messages.append({"role": "assistant", "content": st.session_state["full_message"]}) if prompt := st.chat_input():
    st.session_state.messages.append({"role": "user", "content": prompt})
    st.chat_message("user", avatar="🧑‍💻").write(prompt)
    st.session_state["full_message"] = ""
    st.chat_message("assistant", avatar="🤖").write_stream(generate_response)
    st.session_state.messages.append({"role": "assistant", "content": st.session_state["full_message"]}) All the Codebase of app.py app.py import streamlit as st
import ollama

st.title("💬 Phi3 Chatbot")

if "messages" not in st.session_state:
    st.session_state["messages"] = [{"role": "assistant", "content": "Hello tehre, how can I help you, today?"}]

### Write Message History
for msg in st.session_state.messages:
    if msg["role"] == "user":
        st.chat_message(msg["role"], avatar="🧑‍💻").write(msg["content"])
    else:
        st.chat_message(msg["role"], avatar="🤖").write(msg["content"])

## Configure the model
def generate_response():
    response = ollama.chat(model='phi3', stream=True, messages=st.session_state.messages)
    for partial_resp in response:
        token = partial_resp["message"]["content"]
        st.session_state["full_message"] += token
        yield token

if prompt := st.chat_input():
    st.session_state.messages.append({"role": "user", "content": prompt})
    st.chat_message("user", avatar="🧑‍💻").write(prompt)
    st.session_state["full_message"] = ""
    st.chat_message("assistant", avatar="🤖").write_stream(generate_response)
    st.session_state.messages.append({"role": "assistant", "content": st.session_state["full_message"]}) import streamlit as st
import ollama

st.title("💬 Phi3 Chatbot")

if "messages" not in st.session_state:
    st.session_state["messages"] = [{"role": "assistant", "content": "Hello tehre, how can I help you, today?"}]

### Write Message History
for msg in st.session_state.messages:
    if msg["role"] == "user":
        st.chat_message(msg["role"], avatar="🧑‍💻").write(msg["content"])
    else:
        st.chat_message(msg["role"], avatar="🤖").write(msg["content"])

## Configure the model
def generate_response():
    response = ollama.chat(model='phi3', stream=True, messages=st.session_state.messages)
    for partial_resp in response:
        token = partial_resp["message"]["content"]
        st.session_state["full_message"] += token
        yield token

if prompt := st.chat_input():
    st.session_state.messages.append({"role": "user", "content": prompt})
    st.chat_message("user", avatar="🧑‍💻").write(prompt)
    st.session_state["full_message"] = ""
    st.chat_message("assistant", avatar="🤖").write_stream(generate_response)
    st.session_state.messages.append({"role": "assistant", "content": st.session_state["full_message"]}) Run the Streamlit App streamlit run app.py streamlit run app.py Bonus Point Feel free to check this video to understand the code, follow along, and run the AI assistant locally on your computer: AI assistant using Streamlit, Phi3, and Ollama AI assistant using Streamlit, Phi3, and Ollama

Build a Free-to-Use Open-Source AI Assistant With Streamlit, Microsoft Phi-3, & Ollama

Python Bokeh is one of the best Python packages for data visualization. Today, we are going to see some Python Bokeh Examples. I have also provided the Python Bokeh project source code on GitHub. Learn this easy visualization tool and add it to your Python stack. What is Python Bokeh? Python is a data visualization tool, or we can also say Python Bokeh is used to plot various types of graphs. There are various other graph plotting libraries like matplotlib, but Python Bokeh graphs are dynamic in nature, which means you can interact with the generated graph. See the below examples… Bokeh Installation 💻: Python Bokeh can be easily installed using PIP. You can install the Python Bokeh easily by running the command: pip install bokeh Now everything is ready, let’s go through the examples 🏃‍♂️… 1. LinePlot from bokeh.plotting import figure, show, output_notebook
x = [1, 2, 3, 4, 5]
y = [4, 5, 5, 7, 3]
p = figure(title="{LinePlot Python Bokeh Example")
p.line(x, y, line_width=2)
output_notebook()
show(p) | | Live Preview Source Code 🌿 Contribute 2. Scatter Plot from bokeh.plotting import figure, show, output_notebook
x = [1, 2, 3, 4, 5]
y = [4, 5, 5, 7, 3]
p = figure(title="Scatter Plot Python Bokeh Example by PratikPathak.com")
p.circle(x, y, size=10, color="navy", alpha=0.5)
output_notebook()
show(p) 3. Bar Chart from bokeh.plotting import figure, show, output_notebook
categories = ["A", "B", "C", "D", "E"]
values = [10, 15, 8, 12, 6]
p = figure(x_range=categories, title="Bar Chart Python Bokeh Example by PratikPathak.com")
p.vbar(x=categories, top=values, width=0.9)
output_notebook()
show(p) 4. Histogram from bokeh.plotting import figure, show, output_notebook
import numpy as np
data = np.random.normal(0, 1, 1000)
p = figure(title="Histogram Python Bokeh Example by PratikPathak.com")
p.hist(data, bins=30, color="navy", alpha=0.5)
output_notebook()
show(p) 5. Pie Chart from bokeh.plotting import figure, show, output_notebook
labels = ["A", "B", "C", "D"]
values = [10, 15, 8, 12]
p = figure(title="Pie Chart Python Bokeh Example by PratikPathak.com")
p.wedge(x=0, y=0, radius=0.4, start_angle=0.6, end_angle=2.6, color=["red", "green", "blue", "yellow"], legend_label=labels)
output_notebook()
show(p) | | Live Preview Source Code 🌿 Contribute 6. Time Series Plot from bokeh.plotting import figure, show, output_notebook
from datetime import datetime, timedelta
start = datetime(2023, 1, 1)
end = start + timedelta(days=30)
x = [start + timedelta(days=i) for i in range((end-start).days)]
y = [10, 15, 8, 12, 6, 18, 9, 14, 7, 11, 5, 16, 8, 13, 6]
p = figure(x_axis_type="datetime", title="Time Series Plot Python Bokeh Example by PratikPathak.com")
p.line(x, y, line_width=2)
output_notebook()
show(p) 7. Linked Brushing from bokeh.plotting import figure, show, output_notebook
from bokeh.models import ColumnDataSource
x1 = [1, 2, 3, 4, 5]
y1 = [4, 5, 5, 7, 3]
x2 = [2, 3, 4, 5, 6]
y2 = [2, 4, 6, 8, 4]
source = ColumnDataSource(data=dict(x1=x1, y1=y1, x2=x2, y2=y2))
p1 = figure(title="Scatter Plot 1")
p1.circle('x1', 'y1', source=source)
p2 = figure(title="Scatter Plot 2 Python Bokeh Example by PratikPathak.com")
p2.circle('x2', 'y2', source=source)
output_notebook()
show(p1, p2) 8. Hover Tooltips from bokeh.plotting import figure, show, output_notebook
from bokeh.models import HoverTool
x = [1, 2, 3, 4, 5]
y = [4, 5, 5, 7, 3]
p = figure(title="Hover Tooltips Python Bokeh Example by PratikPathak.com")
p.circle(x, y, size=15, fill_color="navy", line_color="white", alpha=0.5)
hover = HoverTool(tooltips=[("(x,y)", "(@x, @y)")])
p.add_tools(hover)
output_notebook()
show(p) 9. Annotations from bokeh.plotting import figure, show, output_notebook
from bokeh.models import Arrow, VectorRenderer, Label
x = [1, 2, 3, 4, 5]
y = [4, 5, 5, 7, 3]
p = figure(title="Annotations Python Bokeh Example by PratikPathak.com")
p.circle(x, y, size=15, fill_color="navy", line_color="white", alpha=0.5)
arrow = Arrow(x_start=2, y_start=4, x_end=3, y_end=5, line_width=2, line_color="red")
label = Label(x=3, y=6, text="This is a label", render_mode='css', border_line_color='black', border_line_alpha=1.0, background_fill_color='white', background_fill_alpha=0.5)
p.add_layout(arrow)
p.add_layout(label)
output_notebook()
show(p) 10. Custom Glyphs from bokeh.plotting import figure, show, output_notebook
from bokeh.models.glyphs import Asterisk
x = [1, 2, 3, 4, 5]
y = [4, 5, 5, 7, 3]
p = figure(title="Custom Glyphs Python Bokeh Example by PratikPathak.com")
p.add_glyph(x, y, Asterisk(size=20, line_color="red", fill_color="yellow"))
output_notebook()
show(p) | | Live Preview Source Code 🌿 Contribute 11. Gridlines and Axes from bokeh.plotting import figure, show, output_notebook
x = [1, 2, 3, 4, 5]
y = [4, 5, 5, 7, 3]
p = figure(title="Gridlines and Axes Python Bokeh Example by PratikPathak.com", x_range=(0, 6), y_range=(0, 8))
p.grid.grid_line_color = "grey"
p.grid.grid_line_dash = [6, 4]
p.xaxis.axis_label = "X-axis"
p.yaxis.axis_label = "Y-axis"
p.line(x, y, line_width=2)
output_notebook()
show(p) 12. Legend from bokeh.plotting import figure, show, output_notebook
x1 = [1, 2, 3, 4, 5]
y1 = [4, 5, 5, 7, 3]
x2 = [2, 3, 4, 5, 6]
y2 = [2, 4, 6, 8, 4]
p = figure(title="Legend Python Bokeh Example by PratikPathak.com")
p.line(x1, y1, line_width=2, color="red", legend_label="Line 1")
p.line(x2, y2, line_width=2, color="blue", legend_label="Line 2")
p.legend.location = "top_left"
output_notebook()
show(p) 13. Categorical Plots from bokeh.plotting import figure, show, output_notebook
fruits = ["Apples", "Pears", "Nectarines", "Plums", "Grapes", "Strawberries"]
counts = [5, 3, 4, 2, 4, 6]
p = figure(x_range=fruits, title="Categorical Plots Python Bokeh Example by PratikPathak.com")
p.vbar(x=fruits, top=counts, width=0.9)
p.xaxis.axis_label = "Fruit"
p.yaxis.axis_label = "Count"
p.xaxis.axis_label_text_font_size = "12pt"
p.yaxis.axis_label_text_font_size = "12pt"
output_notebook()
show(p) 14. Subplots from bokeh.plotting import figure, show, output_notebook
from bokeh.layouts import gridplot
x1 = [1, 2, 3, 4, 5]
y1 = [4, 5, 5, 7, 3]
x2 = [2, 3, 4, 5, 6]
y2 = [2, 4, 6, 8, 4]
p1 = figure(title="Scatter Plot 1 Python Bokeh Example by PratikPathak.com")
p1.circle(x1, y1, size=10, color="navy", alpha=0.5)
p2 = figure(title="Scatter Plot 2 Python Bokeh Example by PratikPathak.com")
p2.circle(x2, y2, size=10, color="red", alpha=0.5)
grid = gridplot([[p1, p2]], plot_width=400, plot_height=400)
output_notebook()
show(grid) 15. Interactive Plots from bokeh.plotting import figure, show, output_notebook
from bokeh.models import ColumnDataSource, HoverTool, BoxSelectTool
import numpy as np
x = np.random.random(1000)
y = np.random.random(1000)
source = ColumnDataSource(data=dict(x=x, y=y))
p = figure(title="Interactive Plots Python Bokeh Example by PratikPathak.com", tools="hover,box_select")
p.circle('x', 'y', source=source, size=3, color="navy", alpha=0.5)
hover = HoverTool(tooltips=[("(x,y)", "(@x, @y)")])
p.add_tools(hover)
output_notebook()
show(p) | | Live Preview Source Code 🌿 Contribute 16. Linked Panning and Zooming from bokeh.plotting import figure, show, output_notebook
from bokeh.models import Range1d
x1 = [1, 2, 3, 4, 5]
y1 = [4, 5, 5, 7, 3]
x2 = [2, 3, 4, 5, 6]
y2 = [2, 4, 6, 8, 4]
p1 = figure(title="Scatter Plot 1 Python Bokeh Example by PratikPathak.com", x_range=Range1d(0, 6), y_range=Range1d(0, 8))
p1.circle(x1, y1, size=10, color="navy", alpha=0.5)
p2 = figure(title="Scatter Plot 2 Python Bokeh Example by PratikPathak.com", x_range=p1.x_range, y_range=p1.y_range)
p2.circle(x2, y2, size=10, color="red", alpha=0.5)
output_notebook()
show(p1, p2) 17. Linked Axis Ranges from bokeh.plotting import figure, show, output_notebook
from bokeh.models import Range1d
x1 = [1, 2, 3, 4, 5]
y1 = [4, 5, 5, 7, 3]
x2 = [2, 3, 4, 5, 6]
y2 = [2, 4, 6, 8, 4]
p1 = figure(title="Scatter Plot 1", x_range=(0, 6), y_range=(0, 8))
p1.circle(x1, y1, size=10, color="navy", alpha=0.5)
p2 = figure(title="Scatter Plot 2", x_range=p1.x_range, y_range=p1.y_range)
p2.circle(x2, y2, size=10, color="red", alpha=0.5)
output_notebook()
show(p1, p2) 18. DateTime Axis from bokeh.plotting import figure, show, output_notebook
from datetime import datetime, timedelta
start = datetime(2023, 1, 1)
end = start + timedelta(days=30)
x = [start + timedelta(days=i) for i in range((end-start).days)]
y = [10, 15, 8, 12, 6, 18, 9, 14, 7, 11, 5, 16, 8, 13, 6]
p = figure(x_axis_type="datetime", title="Datetime Axis Python Bokeh Example by PratikPathak.com")
p.line(x, y, line_width=2)
output_notebook()
show(p) 19. Heatmap from bokeh.plotting import figure, show, output_notebook
import numpy as np
data = np.random.rand(10, 10)
p = figure(title="Heatmap Python Bokeh Example by PratikPathak.com")
p.image(image=[data], x=0, y=0, dw=10, dh=10, palette="Viridis256")
output_notebook()
show(p) 20. Streamgraph from bokeh.plotting import figure, show, output_notebook
from bokeh.models import ColumnDataSource
import numpy as np
# Generate some sample data
x = np.linspace(0, 10, 100)
y1 = np.cumsum(np.random.randn(100))
y2 = np.cumsum(np.random.randn(100))
y3 = np.cumsum(np.random.randn(100))
y4 = np.cumsum(np.random.randn(100))
y5 = np.cumsum(np.random.randn(100))
source = ColumnDataSource(data=dict(x=x, y1=y1, y2=y2, y3=y3, y4=y4, y5=y5))
p = figure(title="Streamgraph Python Bokeh Example by PratikPathak.com")
p.area(x='x', y1='y1', y2='y2', y3='y3', y4='y4', y5='y5', source=source, line_color=None)
output_notebook()
show(p) | | Live Preview Source Code 🌿 Contribute 21. Polar Plot from bokeh.plotting import figure, show, output_notebook
import numpy as np
r = np.linspace(0.1, 1, 10)
theta = np.linspace(0, 2*np.pi, 10)
p = figure(title="Polar Plot Python Bokeh Example by PratikPathak.com", plot_width=600, plot_height=600, x_range=(-1.2, 1.2), y_range=(-1.2, 1.2), toolbar_location=None)
p.polar(r, theta, color="navy", alpha=0.5, line_width=2)
output_notebook()
show(p) 22. Donut Chart from bokeh.plotting import figure, show, output_notebook
from bokeh.models import ColumnDataSource
labels = ["A", "B", "C", "D"]
values = [10, 15, 8, 12]
source = ColumnDataSource(data=dict(labels=labels, values=values))
p = figure(title="Donut Chart Python Bokeh Example by PratikPathak.com", x_range=labels)
p.wedge(x=0, y=1, radius=0.4, start_angle=0.6, end_angle=2.6, color=["red", "green", "blue", "yellow"], legend_field="labels")
p.wedge(x=0, y=1, inner_radius=0.3, outer_radius=0.4, start_angle=0.6, end_angle=2.6, color="white")
p.axis.visible = False
p.grid.grid_line_color = None
output_notebook()
show(p) 23. Treemap from bokeh.plotting import figure, show, output_notebook
from bokeh.models import TreemapRenderer, ColumnDataSource
import pandas as pd

# Sample data
data = pd.DataFrame({
    'label': ['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H'],
    'parent': ['', 'A', 'A', 'B', 'B', 'C', 'C', 'D'],
    'value': [10, 5, 4, 3, 2, 1, 1, 1]
})

source = ColumnDataSource(data)
p = figure(title="Treemap Python Bokeh Example by PratikPathak.com")
treemap = TreemapRenderer(
    source=source,
    label='label',
    parent='parent',
    value='value',
    color_discrete_sequence=["#e6550d", "#3182bd", "#31a354", "#756bb1", "#636363", "#9e9ac8", "#de2d26", "#fd8d3c"],
)
p.add_layout(treemap)
output_notebook()
show(p) If you get any error, make sure you have installed Python pandas by using the “pip install pandas” command 24. Network Plot from bokeh.plotting import figure, show, output_notebook
from bokeh.models import ColumnDataSource, HoverTool, NodesAndLinkedEdges
import networkx as nx

# Sample data
nodes = ['A', 'B', 'C', 'D', 'E', 'F']
edges = [('A', 'B'), ('A', 'C'), ('B', 'D'), ('B', 'E'), ('C', 'F')]
G = nx.Graph()
G.add_nodes_from(nodes)
G.add_edges_from(edges)

node_positions = nx.spring_layout(G)
node_x = [node_positions[node][0] for node in G.nodes()]
node_y = [node_positions[node][1] for node in G.nodes()]
node_color = ['red', 'green', 'blue', 'yellow', 'purple', 'orange']

source = ColumnDataSource(data=dict(
    x=node_x,
    y=node_y,
    color=node_color,
    node=nodes
))

p = figure(title="Network Plot Python Bokeh Example by PratikPathak.com", x_range=(-1.1, 1.1), y_range=(-1.1, 1.1))
p.add_tools(HoverTool(tooltips=[("Node", "@node")]))
p.circle('x', 'y', size=15, color='color', source=source)

edge_x = []
edge_y = []
for start, end in edges:
    x0, y0 = node_positions[start]
    x1, y1 = node_positions[end]
    edge_x.append(x0)
    edge_x.append(x1)
    edge_x.append(None)
    edge_y.append(y0)
    edge_y.append(y1)
    edge_y.append(None)

p.multi_line(edge_x, edge_y, color="black", alpha=0.5, line_width=1)

output_notebook()
show(p) 25. Sankey Diagram from bokeh.plotting import figure, show, output_notebook
from bokeh.models import Sankey, ColumnDataSource

# Sample data
flows = [
    ('A', 'X', 5),
    ('B', 'X', 3),
    ('A', 'Y', 2),
    ('B', 'Y', 4),
    ('X', 'C', 4),
    ('X', 'D', 3),
    ('Y', 'C', 2),
    ('Y', 'D', 4)
]

source = ColumnDataSource(data=dict(
    source=[flow[0] for flow in flows],
    target=[flow[1] for flow in flows],
    value=[flow[2] for flow in flows]
))

p = figure(plot_width=800, plot_height=400, title="Sankey Diagram Python Bokeh Example by PratikPathak.com")
sankey = Sankey(source=source, label_text_color="white", label_text_font_size="8pt")
p.renderers.append(sankey)

output_notebook()
show(p) | | Live Preview Source Code 🌿 Contribute 26. Chord Diagram from bokeh.plotting import figure, show, output_notebook
from bokeh.models import ColumnDataSource, CategoricalColorMapper
import numpy as np

# Sample data
nodes = ['A', 'B', 'C', 'D', 'E', 'F']
connections = [
    ('A', 'B', 0.5), ('A', 'C', 0.3), ('A', 'D', 0.2),
    ('B', 'C', 0.4), ('B', 'D', 0.3), ('B', 'E', 0.3),
    ('C', 'D', 0.2), ('C', 'E', 0.3), ('C', 'F', 0.5),
    ('D', 'E', 0.2), ('D', 'F', 0.3), ('E', 'F', 0.4)
]

source = ColumnDataSource(data=dict(
    start=np.array([con[0] for con in connections]),
    end=np.array([con[1] for con in connections]),
    value=np.array([con[2] for con in connections])
))

p = figure(title="Chord Diagram Python Bokeh Example by PratikPathak.com", plot_width=800, plot_height=800)
chord = p.chord(
    source=source, 
    start_angle=np.pi / 2,
    direction="clockwise",
    colors=CategoricalColorMapper(factors=nodes, palette="Spectral11")
)

output_notebook()
show(p) How to Contribute? Feel free to open a PR request on our GitHub repo. Steps to contribute: Fork the repo Make changes in the Forked repo and save Open a Pull Request That’s it 😄! 🌿 Contribute Conclusion In this Repo, I have shared 25+ Python Bokeh examples that can help you learn Python Bokeh. Feel free to contribute to our GitHub repo and keep it updated. Also published . here

Getting Started With Python Bokeh: 25+ Data Visualization Examples With Source Code

The Dark Side of Open Source: Securing the Software Supply Chain

This paper is available on arxiv under CC 4.0 license. Authors: (1) Evan Ning, Milton Academy & evan_ning24@milton.edu; (2) David R. Kaeli, Department of Electrical and Computer Engineering & Northeastern University kaeli@ece.neu.edu. Table of Links Introduction Sieve Optimizations Mathematical Optimization Implementation Results and Discussions Conclusions, Source Code & References 6 Conclusions In this paper, we explored how to better parallelize the execution of the Sieve of Eratosthenes. Clearly, both the arithmetic complexity of computation and the storage requirement play a big role in the computation speed of the standard implementation. However, our proposed algorithm is faster and more compact (requiring N/3 auxiliary storage), resulting in a more efficient parallel execution of the Sieve of Eratosthenes. We explored selecting the chunk size to optimize memory usage, which can avoid memory thrashing. Our incremental segmented approach enables us to handle huge numbers, almost approaching infinity. This is crucial for large-scale prime number computations and applications that require extensive use of primes. Memory management in high-performance multi-threaded computations is extremely important. The cache sizes and physical memory constraints can have a huge impact on performance. We encountered this when selecting the size of our scratch memory. What we found is that there are tradeoffs in terms of selecting this size. Next Steps for advancing the Multithreaded Sieve Algorithm Implementation: Prime Data Store File Writing: The current setup is poised for indefinite expansion. However, to efficiently manage and store results, there is a need to implement writing to files, ensuring the preservation of computed data. To address potential memory constraints, the prime base list will be segmented. Each segment will contain up to 125 million primes to prevent memory overflow and ensure efficient retrieval. Custom Big Integer Library: In order to sieve infinitely, a special library for large numbers needs to be created. While there are default big integer libraries, we can customize our own for further optimization. For example, we only divide by 6, so a special operation can be created. Furthermore, a special operation can be created to efficiently mod 6. Incorporation of Multiprocessing: To further optimize the computational process, the implementation of multiprocessing will be explored. This approach is to divide the tasks into multiple computation nodes. However, this comes with its set of challenges. One primary concern is minimizing data transfer between processes, ensuring coherence and consistency in results. Source Code GitHub: https://github.com/evanning1/Multi-thread_Prime_Sieve References [1] R.L. Rivest, A. Shamir, L. Adleman, “A Method for Obtaining Digital Signatures and Public-Key Cryptosystems,” (1978) Massachusetts Institute of Technology, https://people.csail.mit.edu/vinodv/COURSES/MAT302-S12/Rsapaper.pdf [2] Xuedong Luo, “A Practical Sieve Algorithm Finding Prime Numbers,” (1989) Communications of the ACM, 32(3), March 1989, 344–346. https://doi.org/10.1145/62065.62072 [3] Torben Hansen, “Parallel Implementation of the Sieve of Eratosthenes,” F120116 Utrecht University - Institute of Mathematics, https://himsen.github.io/pdf/Project1_parallel_algorithms_Torben.pdf [4] Mário Cordeiro, “Parallelization of the Sieve of Eratosthenes, Faculdade de Engenharia da Universidade, s/n 4200-465 Porto PORTUGAL . [5] Jonathan Sorenson, “An Introduction to Prime Number Sieves,” Computer Sciences Technical Report #909, Department of Computer Sciences University of Wisconsin-Madison, January 2, 1990. [6] POSIX Threads Programming https://hpc-tutorials.llnl.gov/posix/ [7] Research Computing at Northeastern University https://rc-docs.northeastern.edu/en/latest/

Memory Efficient Multithreaded Algorithm: Conclusions, Source Code & References

Open Source Software (OSS) has revolutionized the way software is developed, distributed, and utilized, offering a for innovation and advancement. community-focused platform While from the surface, the field of OSS, with its focus on collaboration and social diversity, may appear as a level playing field, where contributors from all over the globe collaborate on projects driven by a shared passion for software technology and innovation, beneath this lies a challenging issue– gender biases in OSS persist throughout the overall dynamics of community engagement, negatively influencing the experiences of women, their participation rates, collaboration & contribution, and leadership opportunities. Exploring Gender Dynamics in the OSS Industry The underrepresentation of women in Open Source Software communities is a , with various studies highlighting the stark gender disparities in participation rates. not-so-secret and well-documented phenomenon Although there’ve been numerous efforts from communities and organizations to promote diversity, inclusion, and more female participation, these efforts have time and time again been proven futile, with women remaining a minority of open source developers . at just 11% This becomes even more alarming when we consider the fact that women are better , highlighting how facing numerous barriers hinders their advancement within these communities. represented in newcomers to OSS Studies have established that women face unique and substantial challenges when starting their OSS journeys or contributing to particular OSS projects. Some of these are gender-specific limitations that impede their involvement and integration in open-source communities. Specifically, research has found that women often experience . These obstacles significantly contribute to the lack of female representation in the field as they make it unwelcoming for them and thus hamper their growth within such spaces. communication problems, non-friendly and less accommodating orientation procedures, and prejudices based on their personal attributes while participating in OSS projects Women’s contributions are often because of these systemized biases. Such prejudice in evaluative procedures not only undermines women’s contributions but also perpetuates an exclusionary culture within OSS communities that negatively affects women’s interest in participating and contributing to open-source projects. scrutinized more than men’s and cast with greater suspicion Getting involved in an open source is tough enough for first-timers, but when confronted with additional challenges such as having their contributions undervalued or dismissed as well as finding it more difficult to get recognition compared to their male peers, women may be less interested in participating in OSS activities. Another reason why women are less likely to contribute to open-source software is the existence of gender biases in OSS as a never-ending cycle that excludes them perpetually and . Women who aspire to be contributors may feel discouraged or hesitant due to a lack of role models, mentors, leaders, or individuals they can relate with. acts like a curse that keeps on cursing The absence of representation and community leads to isolation and alienation which prevent women from seeing themselves as part of the OSS community thus hindering the recruitment and retention of women within OSS projects while reinforcing ideas that OSS is a male-dominated field where women are not welcome. Open Source Collaboration Can’t Work Without Demographic Diversity The systematic prejudices and gender inequalities that exist in Open Source Software communities have a negative impact on the ecosystem as a whole, affecting participation and collaboration in these settings. It is not difficult to see how a in the development process can result from the underrepresentation of women and gender prejudices in OSS. lack of various viewpoints and experiences Diversity is essential for promoting innovation and creativity in virtually every industry because people with varied experiences offer unique perspectives and methods for approaching problem-solving. Women are underrepresented in open source software (OSS) projects thereby losing out on important contributions that could improve the caliber and efficacy of the software being developed. Gender biases not only misalign with the principles upon which open-source communities are built but have also been shown to result in underperformance and reduced effectiveness. Various studies indicate that more diverse teams bring about greater innovation and success for technology companies. According to a , businesses with executive teams in the top quartile for gender diversity were shown to be 15% more likely to achieve above-average profitability compared to those in the bottom quartile. 2017 report Furthermore, done by Boston Consulting Group and the Technical University of Munich showed an increase in leadership team diversity led to an overall increase in innovation, as well as better financial performance in both emerging markets and developed economies – again affirming that growing numbers of diversified management help industries innovate and firms boost their profit margins. another research The far-reaching implications of gender disparities and biases in OSS communities make it so that an industry already plagued with doesn’t come close to solving its problems. By embracing demographic diversity, including gender diversity, OSS communities can tap into a wealth of unique perspectives, experiences, and problem-solving approaches, leading to more innovative and effective open-source projects. the difficulty of finding skilled professionals with relevant expertise OSS communities must prioritize diversity and inclusion efforts to foster environments where all individuals, regardless of gender, feel welcomed, valued, and empowered to contribute to their fullest potential. Only by making strides to fix the lack of demographic diversity can open-source software and technology at large truly fulfill its promise of collective progress and innovation. Putting the 'Open' in Open Source Software: Addressing Bias and Fostering Inclusivity Taking a deliberate and proactive approach toward fostering gender inclusivity in the OSS space involves creating a supportive community, developing platforms, infrastructure, and tools that help mitigate biases and challenges faced by women in OSS communities, and integrating said tools. Inclusive Guidelines: Creating inclusive guidelines means setting out that set out the expected conduct and values within the OSS society. These guidelines foster OSS’s openness by explicitly prohibiting discrimination, harassment & biased behavior and emphasizing the importance of respect, inclusivity & diversity Only by setting clear expectations and consequences for violating these guidelines, OSS communities can create a safer and more welcoming environment for all members. clear policies and procedures Education, Collaboration, and Networking: Encouraging collaboration, networking opportunities, and educational training & resources for women in the OSS community is crucial for breaking the cycle of underrepresentation and exclusion while also fostering a more inclusive environment. Virtual meetups, conferences, forums, and community events not only provide platforms for women new to OSS to connect, share knowledge, and collaborate on open-source projects but also serve as avenues for accessing educational training and resources. OSS communities can introduce women leaders and role models by organizing such interactions and availing training resources so that young females will be able to see them instead of assuming that they do not exist. Such an all-rounded approach will ultimately result in more diverse OSS communities capable of supporting women with essential resources for their success in the OSS ecosystem. Bias Mitigation Algorithms and Methods: Implementing algorithms and mechanisms within OSS platforms, such as GenderMag, to mitigate biases in evaluation processes can significantly contribute to breaking the cycle of under-representation of women and fostering a more inclusive environment by nipping the bud of detected biases early on. GenderMag is a method that helps incorporating transparent and fair evaluation criteria. By utilizing such tools, OSS platforms can ensure that contributions are evaluated based on merit rather than gender, thus reducing the impact of unconscious biases. Additionally, by incorporating bias mitigation algorithms directly into OSS platforms, such as through automated tools for detecting and addressing implicit biases, OSS communities can establish fairer evaluation processes. Transparent and fair evaluation criteria help ensure that contributions are assessed objectively, regardless of the contributor's gender. development by identify and address gender biases in software Continuous Improvement and Feedback: Regularly reviewing and improving community processes, including contribution evaluation and decision-making procedures, is paramount to addressing gender bias. By consistently refining these processes and seeking input from diverse voices, OSS communities can create more equitable and welcoming environments that encourage the active participation of women. Creating technology that can support women on OSS platforms necessitates an ongoing commitment to improvement and the need to gather feedback, particularly from female users. This entails regularly seeking contributions, having user tests carried out, and refining designs and features based on client responses; thus ensuring that the OSS tools and platforms adequately serve the interests of women. By incorporating these strategies, both technical and community-focused, into the OSS communities and the design and development of OSS platforms and tools, technology can be built to accommodate women and create a more inclusive and welcoming environment for all individuals in OSS communities. Conclusion among participants in these communities, there are still specific challenges that have made it difficult for most women to join or progress. To address this problem, preventive measures must be instituted. Despite several initiatives to promote diversity This includes establishing regulatory frameworks for broader participation, fostering cooperation amongst members, creating supportive social networking functions within online platforms that can help identify gender biases in online environments, and providing educational materials together with other resources aimed at addressing such issues while continually refining procedures that promote collective progress. Feature Image Source: Pexels

Bytes and Bias– Unraveling the Influence of Gender Dynamics in Open Source Software

Although AI, in this article mostly referring to LLMs, seems to be revolutionizing many areas and many laws of conventional technologies that don’t seem to apply to it, I am surprised how many concepts of traditional cybersecurity are appearing in a new light in the era of AI. One that scares me the most is the concept of “Sources and Sinks.” What Are Sources and Sinks? The concept of sources and sinks originally came from security code reviews. It is in reference to the fact that data comes from somewhere, a.k.a. the source, (user input, databases, website components, other systems, etc…) and flows through the application and the logic that processes it into a so-called sink (database, webpage element, an email, etc…). Security researchers commonly do something called “Taint Tracking” or “Taint Analysis” to identify what data goes where. This is already pretty hard to do in large-scale applications, but with enough effort, it is achievable. But at least those applications follow deterministic algorithms, which means, that if A happens, the consequence is always B. You may already see where this is going. AI “Sinks” This Concept You get it? Because of sources and sinks and how it is hard to do taint analysis in AI. Well, the problem with AI is twofold. One thing is that LLMs are intentionally non-deterministic in the conventional sense. Because they are imitating human language and thinking to some extent, they have built-in measures, like temperature and sampling methods, that add randomness to the output of an LLM. Which means, especially in cases where an LLM makes a decision about what tool to use or where to put some data, you can not be entirely sure what exactly an LLM will do. Of course, when you only have one agent, that can either use a calculator or browser, this is not a huge problem. It is, however, unlikely that this is what AI systems of the future will look like. Although, if you add email and PowerPoint to this agent, it can probably do 90% of a McKinsey consultant’s work ;). The other thing is that AI systems will probably have a lot of sources and sinks because one of the intended use cases for these systems is to replace human workers, and if you are an office job person, you know what an insane amount of information you need access to and how much stuff you are allowed to do (at least from a typing stuff into something perspective). Now this becomes even scarier when you realize that, even though this concept is known to more and more developers, it is insanely often ignored. This may be because of a lot of microservice stuff and compartmentalized responsibility, where the individual developer just takes some data that some other developer sends them, and who knows where the user puts in what and where it ends up, or simply because of a lack of awareness. But now imagine these same developers building large-scale, “non-deterministic” applications with access to literally the entire internet and confidential corporate data, and they can use a bunch of tools, send emails and Slack, surf the web and someday control heavy machinery. And as if this was not enough, they have to do all this under immense time pressure with technology that not even the people who created it fully understand and libraries that abstract it to a level where nobody knows what is going on under the hood. Is All Hope Is Lost? I “hope” that this is not the case, and I think now is the perfect time to put a spotlight back on this “old” concept of “Sources and Sinks.” Currently, everyone is out there building some PoC implementation of some AI tooling, and when everyone sees the potential of this crazy new invention, it is very likely that the PoC just gets some more bells and whistles and a feature here and a new API there, and before you know it, it manages some crucial process in your company. So please, while it is still easy, and you think you know where all your data is coming from because it is just this little tiny PoC, start modeling your data flows. Because if you don’t do it now, somebody will have to reverse-engineer them later. It won’t be you because you have all this other stuff going on. It will be some dude in a basement, and he will do some involuntary pen-testing for you. Talking ‘bout AI. This has probably already crossed your mind: “Can’t LLMs just do the data flow modeling and taint tracking?” I think this is very likely the case. This will be an amazing field for AI security companies to work on; you will probably be able to hack a small-scale solution together in a couple of hours. I might even do it one of these days. But just because AI COULD do it, it does not mean that it is not important to give this some thought. Because hackers won’t wait till the market is full of tools that help secure AI. They are in fact hacking AI right now, like crazy. What do you sink about it?

AI Security — What Are Sources and Sinks?

Remember when code was just lines on a screen, without personality or dialogue? Well, buckle up, fellow developers and AI enthusiasts, because the landscape is changing at breakneck speed! Large Language Models (LLMs) are bursting onto the scene, harnessing the power of natural language processing to transform the way we code, create, and communicate. Yan Lekun, the director of Meta AI, says, "The potential of LLMs is huge; they can help us automate tasks, create content, and translate languages," but because there are so many open options, choosing the right LLMs is blindfolded, It’s like walking through a jungle gym. Don't worry, brave developers! This is where I come in, your friendly local AI sherpa, ready to guide you through the hottest open LLMs that will supercharge your career and unleash your inner AI wizard. But there are so many options. Choosing the right LLM can feel like walking into a jungle gym with your eyes closed. Fear not, brave developers! I'm here to act as your Sherpa, guiding you through the hottest open-source LLMs to increase your productivity and unleash your inner AI wizard. So, let's crack open the treasure chest and unveil the top 10 gems: No introduction needed. Bard, Google’s behemoth LLM, boasts an impressive 137B standard, making it a powerhouse for handling complex NLP tasks. While still in development, Bard showed early promise in areas such as rule generation and summaries. Bard (Google AI): Coming in hot at 178B parameters, Jurassic-1 Jumbo is a monster LLM with a reputation for amazing realistic speech understanding. If your project requires you to work with large datasets of text and code, Jurassic-1 Jumbo could be your new best friend. Jurassic-1 Jumbo (AI21 Labs): Together, NVIDIA and researchers at Megatron developed a monster 530B-parameter LLM specifically geared to generate a variety of creative texts, such as poetry, code, writing, pieces of music, emails, documents, and more etc. If you want to add a bit of AI-powered creativity to your project, this is the LLM for you. Megatron-Turing NLG (NVIDIA): This multilingual marvel, backed by a consortium of AI researchers, boasts impressive performance in 46 languages. Need to translate code comments, document international projects, or build multilingual chatbots? BLOOM is your one-stop shop. BLOOM: Clocking in at a whopping 1.75T parameters, WuDao 2.0 is a Chinese superpower LLM that focuses on multilingual capabilities. This model is particularly adept at understanding and responding to natural language in Chinese. WuDao 2.0 (BAAI): This community-driven project offers a smorgasbord of open-source LLMs of various sizes, all trained on a huge dataset of text and code. EleutherAI is a good choice for developers of different LLMs who want to experiment with construction design to find the one that best suits their specific needs. EleutherAI (EleutherAI): Another gem from EleutherAI GPT-J is a 6B parameter LLM based on the OpenAI GPT-3 framework. Although smaller than some of the other options on this list, GPT-J offers a good balance of power and functionality, making it a great choice for developers with limited resources. GPT-J (EleutherAI): Although not strictly open-source, Cohere provides a free layer to their powerful LLM APIs. This makes it a good choice for developers who want to try out LLM before diving headfirst into their models. Cohere (Cohere Corp): LaMDA, short for Language Modeling for Conversational Programming, is Google’s take on the conversational AI model. LaMDA is specifically designed to create realistic and engaging interactions, making it ideal for developers of chatbots or virtual assistants. LaMDA (Google AI): Rounding out our list is Cicero, a real language LLM from Meta AI. Trained in the extensive use of information and law, Cicero excels at such tasks as interviewing and taking information. Cicero (Meta AI): Conclusion: Your Personal AI Playground Awaits Note that this list is just a taste of the vibrant and ever-growing world of open LLMs. Each model has unique strengths and weaknesses, so experiment, explore, and find the one that fits your specific needs and projects. As Fei-Fei Li, executive director of the Stanford Human-Centered AI Institute, aptly puts it, "AI Think of these LLMs as powerful tools in your developer toolbox, in order to maximize your creativity.” has made your work easier, what." Expect to push the limits of possibility and open your eyes. So, dive in, explore, and prepare to be amazed at the potential of an open-source LLM to transform your growth journey. The future is bright and powered by AI, but the story of how it unfolds is yours to write, code by code, with the help of these wonderful language models.

10 Open-Source LLMs That Will Rock Your Dev World in 2024

How do you choose the most appropriate model for your application? An analysis of evaluating and building applications on open-source large language models. The computational complexity of AI models is growing exponentially, while the compute capability provided by hardware is growing linearly. Therefore, there is a growing gap between those two numbers, which can be seen as a supply and demand problem. On the demand side, we have everyone wanting to train or deploy an AI model. On the supply side, we have Nvidia and a number of competitors. Currently, the supply side is seeing earnings skyrocket, and the demand side is stockpiling and vying for access to compute. It’s a simple formulation but one that works well to explain the current situation in AI. If anything, the generative AI explosion in the last couple of years has validated this formulation. How can this supply vs. demand imbalance be resolved then? One way would be to introduce , and some people are working on that. If you are not one of them, the other option would be to work on the software side of things. That is, to try and come up with ways to make AI model training and inference less computationally intensive. more AI hardware with more capabilities This is what CEO and co-founder Yonatan Geifman has been working on since 2019. Deci is focused on building AI models that are more efficient and accurate. Initially, , later adding a range of language models to its portfolio. Deci Deci released computer vision models Interestingly, most of these models are open source. We caught up with Geifman to explore the landscape of open-source AI, aiming to address some key questions. Do the benefits of open source and the criteria to consider in the classical build vs. buy dilemma apply to AI large language models (LLMs)? And how can LLMs be evaluated and tailored to the specific needs organizations have for their applications? https://youtu.be/1GD9J84Bw34?embedable=true Open source Large Language Models are different To begin with, let’s examine what “open source” means in the context of LLMs. Open-source software has been around for a long time and is sufficiently well understood. , and different licensing schemes have their peculiarities. Open source has gone through some turbulence Still, in software, things are relatively straightforward. It’s all about the source code and its availability. Having the source code and the right dependencies and environment means being able to build and deploy software. But what about AI models? Yes, we also have the source code. However, there are additional artifacts on the way from source code to deployment. We have the dataset used to train the model. We have the training process and . Some model releases also include metadata such as metrics and . Can the same open-source definition apply? weights for the source code model cards Geifman believes that the most important artifacts are the source code and the weights, as having those shared enables people to use and build on top of other people’s work and collaborate around those derivatives that are open source. Metadata is nice to have but not essential in his view. But there’s also the crucial issue of data and training process. Deci, like Mistral, is not sharing data or details about the process of training their open-source models. Geifman noted that there are trade secrets around those. To get a model that works well, it’s not just about having the right data but also about using the right parts of the data in the right way, he added. Geifman thinks that sharing the data and the training process is significantly less important than sharing the model itself. By contrast, that “trained on the same dataset for long enough, pretty much every model with enough weights and training time converges to the same point.” OpenAI’s James Betker claims What probably everyone will agree to is the fact that open-source software and open-source models are not exactly the same thing. As well as the fact that if open sourcing of model architectures and software had not happened, AI would not be at the point where it is today. Either way, there is a whole spectrum from having everything open to having everything closed, and most models fall somewhere in between. Geifman noted that there is a tendency in the community to use the term models instead of open-source models. open weights Open source Large Language Models vs. GPT-4 and Claude Open weights or open source, however, what’s probably more important is how these models can be used by organizations. This is where things look more like traditional software. Organizations have many choices. They can use open-source models hosted in their own infrastructure and in some third-party providers' infrastructure, or they can use hosted, closed-source models such as GPT-4 or Claude 3 via an API. What Geifman sees is that many organizations embark on their AI journey by leveraging closed-source model APIs. He thinks the reason is it’s easier to get started this way, as there is less investment in time and expertise. However, over time, that may not be the more sensible approach. Prices may fluctuate, but as closed-source APIs come with a premium, costs will start adding up. But this may not even be the most serious drawback. Closed source models are black boxes controlled by a third party, and there are implications to this. When things go well, using a thing party model may mean lowering the bar to entry and having peace of mind. But the flip side is that if the model is compromised, goes down, or suffers from a bug or degradation in performance, applications that use it will suffer the consequences. Similar effects will also occur even when updating models, as their behavior may change inconsistently. Prompt engineering is brittle enough as it is, so adding this parameter makes building applications on top of closed-source LLMs an exercise in non-determinism. There are also privacy and security concerns to be aware of, as sensitive data may be exposed. This is the reason why some organizations are not only refraining from using closed-source models but outright banning their employees from using them, too. Large Language Model customization: Fine-tuning vs. RAG Most of the above is not specific to AI models or LLMs. What is specific, and more often than not required for real-world applications, is customization. Whether it’s via , LLMs need to be tailored to users' needs and datasets. fine-tuning or RAG Even though fine-tuning and RAG aim to achieve similar goals, the way they work is quite different, as Geifman pointed out. Fine-tuning changes the actual model for a specific data set, while RAG focuses on injecting data into the generation phase. Geifman finds that with RAG, there is not a lot of difference if you’re looking to apply it to an open-source or a closed-source model. Fine-tuning, on the other hand, will be much easier to do for an open-source model. In the debate of fine-tuning versus RAG, Geifman thinks the future is a combination of both. “There are some benefits in using RAG, mostly on data privacy. You don’t want to train the model on private data, but in RAG you can supplement the model with private data in the prompt. That’s one of the benefits of using RAG, and also the fact that you don’t need to fine tune the models all the time in order to have updated data”, Geifman said. It’s worth mentioning that both fine-tuning and RAG are constantly evolving, with techniques such as and , respectively. Overall, however, Geifman thinks that building more complex use cases will be much easier to do in open source versus closed source in the future, especially when the accuracy gap is bridged. LoRA+ Corrective RAG The reference to accuracy brings us to another multi-variant equation, namely, how to evaluate LLM performance. At the moment, as Geifman mentioned, it seems that closed-source models such as OpenAI’s GPT-4 or Anthropic’s recently released Claude 3 have an edge in terms of accuracy over open-source models. But there is a lot of nuance here. Evaluating Large Language Models Geifman predicted that the gap will be closed by the end of 2024. However, as has recently been demonstrated, carefully fine-tuned models specializing in specific tasks can outperform larger models for these tasks. LoRA Land, a collection of 25 fine-tuned open-source Mistral-7B models, was shown to consistently outperform base models by 70% and GPT-4 by 4-15%, depending on the task. What’s more important, however, is evaluating performance holistically. Typically, people tend to equate LLM performance with accuracy. While accuracy is an important metric, there are many other parameters that need to be considered as well. These all revolve around the nature of the application for which the LLM will be used. Even a typical use case such as document summarization, for example, has different requirements depending on the type of document and what the summary is used for. Latency and throughput are also very important, again depending on the application requirements. Some applications may be more tolerant than others. And then there’s , obviously. Even though this is not a metric per se, it’s something that needs to be factored in as well. cost Geifman’s recommendations? First, if you need speed, you’d better work with smaller, fine-tuned, tailored models for your applications. Second, for most B2B applications, you don’t need GPT-4-level accuracy. If you’re trying to summarize documents, you can do it with . mid-sized language models with good performance, lower latency, and better value for money As to how to evaluate models holistically, that’s still an unsolved problem. The two main alternatives at the moment are human evaluation and using an LLM to evaluate another LLM. Both have pros and cons and as usual, there are lots of nuances and blind spots. You might be tempted to note that it’s all fine and well, but not everyone needs to be familiar with all the nuances of LLM evaluation. There is always the , which has become the de facto evaluation ground for all LLMs. Just like any other benchmark, however, these metrics are only indicative. Hugging Face leaderboard The bottom line is that if you want to assess whether a model is cut out for the task at hand, you can stand on the shoulders of giants, but you’ll have to put in the work yourself. And that means formulating your requirements, developing realistic evaluation scenarios, and putting LLMs to the test. Market outlook In the end, the fact that best practices and principles from other areas also apply at least partially to LLMs is probably good news for organizations looking to incorporate LLMs in their applications. As Geifman noted, the excitement from LLMs and their capabilities and the potential of integrating them in enterprises across many verticals has energized the market. Geifman shared that Deci’s AI platform has seen its market more than triple in 2023. Deci’s market is divided into two segments, each with its own characteristics. One of them is computer vision, which is more mature. People know what they need, and they are looking towards production use cases with multiple use cases per company and team, as Geifman put it. Then, there is the GenAI segment, in which people are more in an exploratory phase. Geifman believes 2024 will be the year we’ll see many GenAI-based projects going into production. He sees document summarization and customer care as the dominant use cases. https://pod.co/orchestrate-all-the-things-podcast-connecting-the-dots-with-george-anadiotis/evaluating-and-building-applications-on-open-source-large-language-models-featuring-deci-ceo-co-founder-yonatan-geifman?embedable=true Deci addresses both computer vision and GenAI with open-source models. includes SuperGradients, a library for training PyTorch-based computer vision models, DeciLM 6B and DeciLM 7B, which Geifman noted offers similar accuracy to Mistral 7B with better efficiency and performance, the DeciCoder 1B and DeciCoder 6B code generation models as well as the DeciDiffusion text-to-image model. Deci’s offering These models are free to use, with additional tools to customize, fine-tune, and deploy them. Deci’s deployment environment for LLMs is called Infery-LLM. It’s an open-core approach in which the models are open source, and the value-add tools are commercial. To showcase the value that Infery-LLM brings, Geifman shared that running DeciLM 7B with and without Infery-LLM gets 4,5x vs. 2x the performance of Mistral 7B, respectively. Deci is also working on releasing more LLMs with more parameters and better performance. Also published . here

Open Source LLMs: Evaluating and Building Applications on Open Source

The New York Times Company v. OpenAI Update Court Filing, retrieved on February 26, 2024 is part of . You can jump to any part in this filing . This part is 3 of 15. HackerNoon’s Legal PDF Series here B. The Key to Generalist Language Models: Scale So OpenAI’s research took the logical next step: “scaling up [] model size” by increasing both the “size” and “diversity” of the training data. See GPT-3 Paper at 6. To build its next generation of models, OpenAI’s researchers gathered a more robust set of data in part by “expand[ing]” WebText database into a new version called “WebText2,” which included material shared over a longer period of time. Id. at 8–9. They also used a filtered version of Common Crawl, a repository of data collected by a non-profit research organization representing “a copy of the Internet.” Compl. ¶ 88. OpenAI disclosed all of this no later than July 22, 2020. GPT-3 Paper at 8. At the time, it was common knowledge that WebText2 and Common Crawl included numerous articles published by the Times.[19] The result of this simple act of “scaling up” the training data was, as the Times reported at the time, “mind blowing.”[20] The new GPT-3 model was “by far the most powerful ‘language model’ ever created.” Manjoo, supra note 20. It could conduct “on-the-fly reasoning” and “unscrambl[e] words, perform[] arithmetic, and us[e] novel words in a sentence after seeing them defined only once.” GPT-3 Paper at 5. Increasing the scale of training led to a surprising jump in ability. See, e.g., id. at 22 (Figure 3.10). According to the Times’s reporting, GPT-3 showed that “[m]achines are gaining the ability to write.” Manjoo, supra note 20. Within days, developers began to use it to build unprecedented tools. Id. (“service that responds to email on your behalf”). The “key advance,” as the Times reported, was “GPT-3’s flexibility.” Id. And the reason it was flexible was OpenAI’s decision to “scal[e] up” the “size” of the training data. GPT-3 Paper at 6. The amount of data needed was staggering. Compl. ¶ 85. But it was that “unprecedented scale” that allowed the model to internalize not only a “map of human language,” but achieve a level of adaptability—and “emergent” intelligence—that “no one thought possible.”[21] Continue Reading . Here [19] Compl. ¶ 88 (citing a “2019 snapshot of Common Crawl”); GPT-2 Model Card (noting prevalence of Times content in WebText); GPT-3 Paper at 8 (noting WebText2 is an “expanded version of the WebText dataset”). [20] Farhad Manjoo, How Do You Know a Human Wrote This?, N.Y. Times (July 29, 2020), https://www.nytimes.com/2020/07/29/opinion/gpt-3-ai-automation.html. The Court may take judicial notice of this and other news articles cited in this Motion. In re UBS Auction Rate Sec. Litig., No. 08-cv-2967, 2010 WL 2541166, at *10-12 (S.D.N.Y. June 10, 2010) (judicial notice of “several news items” including Wall Street Journal articles). [21] Cade Metz, Meet GPT-3. It Has Learned to Code (and Blog and Argue), N.Y. Times (Nov. 24, 2020), https://www.nytimes.com/2020/11/24/science/artificial-intelligence-ai-gpt3.html. About HackerNoon Legal PDF Series: We bring you the most important technical and insightful public domain court case filings. This court case retrieved on February 26, 2024, from is part of the public domain. The court-created documents are works of the federal government, and under copyright law, are automatically placed in the public domain and may be shared without legal restriction. fingfx.thomsonreuters.com

ChatGPT's Predecessor Was Trained on 'Filtered' Internet Data from an Open Source Repository

Attending F/LOSS (Free/Libre and Open source) events is one of the most magical things you can do. If you have never been to one, here are three reasons to participate in an event organized mainly by its community. It's pretty different than a "commercial" event.: The amount of marketing BS and product pitches is minimal; sometimes, it never happens. You receive great value, most of the time for free or against a small donation to support the team. It focuses on freedom: The whole concept of the FOSS is about freedom. You would be able to feel the spirit of that in every action. I can't explain it with concrete examples, but imagine the feeling of finally doing something you like, such as taking a day off work and going hiking. You learn a lot: the Free/libre and the open-source software ecosystem is changing constantly and fast. The community launches new protocols, technologies, RFS, and projects daily and modifies them every millisecond. Good luck keeping with that! There are many other reasons to visit, help organize, or volunteer for an event like that. Find yours today! FOSS events in Bulgaria in 2024 is one of Europe's most beautiful countries, so if you want to combine your appetite for sightseeing, culture, good food, and your curiosity about free and open-source software, you are in a good place. Bulgaria Let me help you with three events this year focused entirely on the FOSS movement. OpenFest 2024 2-3 November Sofia, Bulgaria is an annual event that promotes open culture, free and open software, and knowledge sharing. The first OpenFest happened in 2003 in Sofia, and since then, it has been held every year – sometimes in several Bulgarian cities simultaneously. OpenFest OpenFest is the biggest event in the country. Sometimes, it attracts more than 4000 humans and even some robots. It has several simultaneous tracks covering technical, community, and open art topics. It's also famous for its practical workshops. Known as Serdica in antiquity and Sredets in the Middle Ages, Sofia has been an area of human habitation since at least 7000 BC. Sofia is also Bulgaria's capital of the Free Software and Opensource Movement. How.Camp 2024 27 July Gabrovo, Bulgaria is the first edition of a small open and collaborative event in the small mountain town of Gabrovo, famous for the inhabitants' unique sense of humor. The team behind that has over 20 years of experience organizing technical and FOSS events, so you are in good hands. How.Camp 2024 The sessions will cover topics like Linux on Phones, open-source careers, and how to contribute to its community, and many others still in the works. So watch the page and ensure you get one of the 79 spots left! Thunderbird Gabrovo is known as the birthplace of the environmental sculptor Chisto from the tandem and with its citizens' unique sense of humor. Christo and Jeanne-Claude TuxCon 2024 11-12 May Plovdiv, Bulgaria is an annual event about free and open-source software and hardware. The entrance is free for all visitors. TuxCon is a community event organized by volunteers. They also have a workshop session, but mainly focused on open hardware. Please keep an eye out for more on their website. TuxCon 2024 is one of the oldest European cities and enjoys an ancient heritage spanning over eight millennia. Plovdiv is a cultural hub in Bulgaria and was the in 2019. The city is an important economic, transport, cultural, and educational center. Plovdiv European Capital of Culture Come and visit I am confident you would love the community, the country, the food, and the wine. Plan your trip to Bulgaria and enrich your persona with some knowledge and freedom.

What are the Three Open Source Events You Should Visit in 2024

The open-source developer community is full of volunteers, so, it was to be expected that their creations were also made for fun. That’s how we can enjoy a lot of free and ad-free games on mobile and desktop, all of them designed with passion. The teams behind these titles cover all potential expenses (domains, hardware, distribution, etc.) by themselves —and that’s why they’d appreciate a tip if you had a nice time with their software. Developers on GitHub usually share their PayPal or Patreon accounts, but these platforms always charge high fees, aren’t available for everyone, and aren’t censorship-resistant. If you want to donate to an open-source project available on GitHub, This Obyte-based application only asks for the GitHub profile to send some cryptocurrencies, even if the recipient doesn’t have a wallet yet. you can use as a decentralized, globally available, unstoppable, as well as faster and cheaper option. Kivach Besides, Kivach has a unique feature: recipients can distribute their received funds automatically among other repositories, considering their contribution to their projects —creating ‘cascading donations.’ This way, you could be helping several projects with just one donation. In this episode, we’ll explore five interesting games that you can try for free and, maybe, donate some coins via Kivach. Let’s go! SuperTux If you enjoyed the classic Super Mario Bros., you could enjoy SuperTux as well. This is a 2D jump'n'run sidescroller game inspired by that series, created and released by Bill Kendrick in 2004. Players control Tux, the Linux penguin, as he navigates through various levels, overcoming obstacles and enemies to rescue his kidnapped friend, Penny. https://www.youtube.com/watch?v=Wcqat_b8ft8&embedable=true Key features include colorful and engaging graphics, diverse levels, power-ups, and boss battles. SuperTux is currently developed by a community of contributors worldwide, who are improving the software and adding new features and levels continually. Besides the main story, there are community-contributed levels available as add-ons or in the forums, and an integrated level editor for further contributions. Funding for the project primarily comes . They help to pay for domains, servers, and salaries. In exchange, donors can enjoy exclusive content. The team accepts funds via PayPal and cryptocurrencies via Coinbase. In addition, you can send them coins via Kivach, where they appear as . from donations supertux/supertux OpenTTD Not an adventurer but a businessman? OpenTTD, based on the original Transport Tycoon Deluxe, allows players to build and manage transportation networks. It was initially released in 2004 by Ludvig Strigeus --also the creator of uTorrent. The game has been praised for the improvements applied to the original version, including multiplayer support, modding capabilities, and a customizable interface. https://www.youtube.com/watch?v=7tF73VBVSNQ&embedable=true They must efficiently transport passengers and cargo, navigate complex terrain, and compete with AI or other players in a sandbox environment. With its deep simulation and strategic challenges, the game offers an immersive and rewarding experience. In OpenTTD, players build railways, roads, airports, and waterways to connect cities and industries. The current team behind this software willingly offers their free time to keep it running, but additional costs like server infrastructure and hosting come from donations. They accept bank transfers, LiberaPay, PayPal, and cheque. However, all these methods have their own (often high) fees. You could send them some coins via Kivach for much less, just by clicking on “Donate” here. FreeOrion Inspired by the commercial series Masters of Orion, this is a space empire and galactic conquest game set in a vast universe filled with diverse alien species, ancient ruins, and strategic opportunities. It was initially in 2003 by a team led by Zach Laine. Players control their own customizable empire, managing resources, researching technologies, and engaging in diplomacy and warfare with rival factions. released https://www.youtube.com/watch?v=gF3_RFsXeU8&embedable=true It has an engaging gameplay, procedurally generated galaxies, customizable ship designs, and a rich lore-driven universe. Players have a multitude of options and opportunities at their disposal. They can design and customize their ships to fit specific roles and strategies, engage in epic space battles against AI-controlled factions or other players in multiplayer mode, explore the galaxy to discover ancient relics and artifacts that provide unique bonuses, and terraform planets to make them more habitable for their population. The development team is formed by volunteers, but they accept donations to cover costs derived from hosting, servers, hardware, artists, and others. They shared their PayPal and Flattr accounts for this purpose, but we have to note that the last one closed its operations in late 2023. You can still send them crypto via Kivach, where they’re available as . freeorion/freeorion Cataclysm: Dark Days Ahead (CDDA) This one could be a great option for fans of the post-apocalyptic genre. with a unique set of traits and dedicate to anything to survive or help others to do so: craft objects like clothes, weapons, or chemicals, build houses or shelters, farm the land and grow food, or directly fight against a plethora of creatures. In a world devastated by zombies, robots, and interdimensional creatures, players can create their own avatar https://www.youtube.com/watch?v=y_bt7-ZxfYM&embedable=true The game features a procedurally generated world filled with diverse locations, including towns, forests, and underground bunkers, each offering unique challenges and opportunities. There's no specific goal to win the game, instead, what matters is the experience, and to survive one more day. The software features ASCII graphics, creating a retro aesthetic reminiscent of classic roguelike games, with detailed environmental descriptions enhancing immersion and gameplay depth. CDDA was originally launched by “Whales” in 2010 and subsequently forked by the community in early 2013. They accept donations via Patreon, or you can sponsor an individual developer via GitHub Sponsors. Of course, it’s also possible to send them funds via Kivach, where they appear as . cleverraven/cataclysm-dda Enigma We have something simpler but equally challenging on this list as well: Enigma. This is a game inspired by Oxyd (Atari, 1990), Players control marbles through landscapes, overcoming obstacles, navigating hazards, and aiming for quick completion to beat personal and world records. which offers over 1,000 diverse levels blending logic puzzles with dexterity challenges. Released in 2002 by Daniel Heck and its team to keep the spirit of Oxyd alive after its publisher's departure, Enigma features Lua-generated landscapes, a level editor, and music by Andrew 'Necros' Sega (from Iris). Praised as a faithful Oxyd clone, it reached a stable version in 2007 and boasts high-resolution 2-D graphics, realistic physics, 500 in-game objects, and availability across multiple platforms. While free to play, Enigma welcomes donations to support ongoing development, ensuring its longevity and continued improvement. As they mention on the : “Just donating a fraction of the amount you would be willing to spend for a commercial 'Enigma' in a store would help us to improve Enigma.” The project is available on Kivach as . official website Enigma-Game/Enigma Contribute to more projects! There are all kinds of open-source software available on GitHub for you to explore and use for free. Most developers there aren’t paid for their work though, and a lot of projects are orphaned and often lose their functionality. Every software requires maintenance, after all. If you want your favorite projects to keep running, please consider donating! And don’t forget to check our previous chapters to discover more awesome tools. Kivach Donations: 5 GitHub projects it has helped so far How You Can Support Key Open-Source Initiatives With Kivach 5 Open-source Projects You Can Support With Kivach — and Cryptocurrencies 5 Open-source Projects You Can Support with Kivach and Cryptos, Episode III 5 Open-Source Projects You Can Donate to via Kivach, Episode IV: Privacy Tools 5 Open-source Blogging and Writing Tools to Donate to via Kivach (Ep V) 5 Open-Source Tools You Can Donate to via Kivach, Episode VI: Decentralized Services Featured Vector Image by pikisuperstar / Freepik

Stories, News & Mentions about SOURCE

Source’s Stories on HackerNoon