CrawlQ

crawlq

A mysterious botnet, identified by the user-agent string Mozilla/5.0 (X11; Linux x86_64), is aggressively scraping WordPress websites at an alarming rate, flagrantly disregarding the crawl delay directives set in the  robots.txt file. The botnet has been found to be causing significant server strain, bypassing web administrators' efforts to control access and protect their site resources. Despite the bot's user-agent string mimicking a legitimate Linux browser, its bad behavior is anything but lawful. Not only does it ignore robots.txt instructions, but it also carries out non-stop scraping activities that severely impact the performance and security of affected websites. The unregulated scraping can lead to slowed website loading times and increased bandwidth consumption. IP Addresses and ASN Sources Involved The IP addresses behind this botnet appear to originate from multiple autonomous system numbers (ASNs), primarily hosted by less-reputable service providers. The IPs exhibit coordinated behavior, systematically scraping WordPress websites with no regard for the implemented crawl-delay directives. The ASNs and their affiliated IP addresses are: ASN 13886 - Cloud-South


ASN 21769 - AS-COLOAM


ASN 263735 - Sociedad Buena Hosting, S.A.


ASN 52393 - Corporacion Dana S.A. 185.199.117.126
216.10.7.2
104.233.51.70
104.249.4.61
186.179.1.64
181.177.71.218
185.195.215.202
185.193.73.239
104.239.116.255
185.199.118.137
185.193.72.244
216.10.3.36
186.179.11.6
181.177.70.128
104.239.115.2
186.179.10.14
104.249.5.146
185.199.116.245
104.233.54.62
185.207.97.112
185.199.116.218
181.177.71.59
104.249.0.172
104.233.49.164
185.188.78.129
216.10.2.44
104.233.48.205
216.10.6.143
104.249.1.226
185.196.191.240
67.227.122.211
185.195.221.166
181.177.79.165
186.179.25.11
185.199.117.248
185.195.220.198
104.239.114.195
181.177.66.111
67.227.127.113
185.205.196.57
199.168.122.42
186.179.27.9
185.193.75.105
216.10.3.10
216.10.2.161
185.188.77.24
104.233.48.70
185.188.79.15
186.179.2.186
181.177.72.76
216.10.6.205
186.179.13.13
181.177.78.227
181.177.72.12
181.177.79.141
186.179.25.210
104.233.49.36
104.249.3.157
104.239.117.199
104.233.48.52
104.233.51.106
216.10.3.78
216.10.0.49
185.207.99.161
67.227.120.127
67.227.121.26
104.233.55.225
104.249.3.194
185.195.220.41
181.177.71.84
104.233.48.111
104.249.2.46
181.177.67.170
104.249.2.195
186.179.13.80
67.227.124.74
104.239.116.151
104.239.119.180
185.195.221.180
104.249.2.99
104.239.114.190
104.239.117.112
181.177.71.49
67.227.121.109
199.168.121.93
185.195.223.14
181.177.67.30
181.177.76.63
181.177.77.79
181.177.66.29
181.177.77.197
186.179.24.252
185.196.188.94
181.177.76.62
216.10.7.75
181.177.68.33
186.179.11.165
181.177.71.188
185.195.213.87
185.193.74.97
67.227.122.57
185.196.189.63
216.10.1.142
199.168.122.193
186.179.2.117
181.177.72.134
181.177.66.212
185.188.77.122
185.207.96.121
199.168.121.246
104.249.0.31
185.195.222.160 Potential Mitigation Strategies Web administrators are urged to monitor their logs for suspicious activity involving the Mozilla/5.0 (X11; Linux x86_64) user-agent string and IPs associated with the above ASNs. Immediate actions to consider include: Blocking or rate-limiting the offending IPs through firewall rules.


Implementing CAPTCHA systems for suspicious traffic.


Using bot protection plugins or services such as Cloudflare to prevent excessive scraping. The persistence of this botnet highlights the importance of continually refining web security measures to protect digital assets from unauthorized data scraping and potential attacks. This unknown botnet's activity is a pressing issue for WordPress site owners, and if left unchecked, its presence could lead to server overloads, data leaks, or even downtime for many websites. Interestingly, despite the aggressive nature of this botnet, none of the associated IPs have been flagged or reported in popular databases like AbuseIPDB. Even more surprising, these IPs are not natively blocked by Cloudflare’s managed rules, which typically catch such malicious behavior early on. This suggests that the botnet is currently flying under the radar, operating in a gray zone of anonymity. The lack of detection and reporting raises concerns about how sophisticated and stealthy this botnet may be. For now, it seems to be an entirely unknown entity in the cybersecurity landscape. However, it does seem look like it is scraping WordPress content. A mysterious botnet, identified by the user-agent string Mozilla/5.0 (X11; Linux x86_64) , is aggressively scraping WordPress websites at an alarming rate, flagrantly disregarding the crawl delay directives set in the  robots.txt file. The botnet has been found to be causing significant server strain, bypassing web administrators' efforts to control access and protect their site resources. Mozilla/5.0 (X11; Linux x86_64) Despite the bot's user-agent string mimicking a legitimate Linux browser, its bad behavior is anything but lawful. Not only does it ignore robots.txt instructions, but it also carries out non-stop scraping activities that severely impact the performance and security of affected websites. The unregulated scraping can lead to slowed website loading times and increased bandwidth consumption. robots.txt IP Addresses and ASN Sources Involved The IP addresses behind this botnet appear to originate from multiple autonomous system numbers (ASNs), primarily hosted by less-reputable service providers. The IPs exhibit coordinated behavior, systematically scraping WordPress websites with no regard for the implemented crawl-delay directives. The ASNs and their affiliated IP addresses are: ASN 13886 - Cloud-South ASN 21769 - AS-COLOAM ASN 263735 - Sociedad Buena Hosting, S.A. ASN 52393 - Corporacion Dana S.A. ASN 13886 - Cloud-South ASN 13886 - Cloud-South ASN 13886 - Cloud-South ASN 21769 - AS-COLOAM ASN 21769 - AS-COLOAM ASN 21769 - AS-COLOAM ASN 263735 - Sociedad Buena Hosting, S.A. ASN 263735 - Sociedad Buena Hosting, S.A. ASN 263735 - Sociedad Buena Hosting, S.A. ASN 52393 - Corporacion Dana S.A. ASN 52393 - Corporacion Dana S.A. ASN 52393 - Corporacion Dana S.A. 185.199.117.126
216.10.7.2
104.233.51.70
104.249.4.61
186.179.1.64
181.177.71.218
185.195.215.202
185.193.73.239
104.239.116.255
185.199.118.137
185.193.72.244
216.10.3.36
186.179.11.6
181.177.70.128
104.239.115.2
186.179.10.14
104.249.5.146
185.199.116.245
104.233.54.62
185.207.97.112
185.199.116.218
181.177.71.59
104.249.0.172
104.233.49.164
185.188.78.129
216.10.2.44
104.233.48.205
216.10.6.143
104.249.1.226
185.196.191.240
67.227.122.211
185.195.221.166
181.177.79.165
186.179.25.11
185.199.117.248
185.195.220.198
104.239.114.195
181.177.66.111
67.227.127.113
185.205.196.57
199.168.122.42
186.179.27.9
185.193.75.105
216.10.3.10
216.10.2.161
185.188.77.24
104.233.48.70
185.188.79.15
186.179.2.186
181.177.72.76
216.10.6.205
186.179.13.13
181.177.78.227
181.177.72.12
181.177.79.141
186.179.25.210
104.233.49.36
104.249.3.157
104.239.117.199
104.233.48.52
104.233.51.106
216.10.3.78
216.10.0.49
185.207.99.161
67.227.120.127
67.227.121.26
104.233.55.225
104.249.3.194
185.195.220.41
181.177.71.84
104.233.48.111
104.249.2.46
181.177.67.170
104.249.2.195
186.179.13.80
67.227.124.74
104.239.116.151
104.239.119.180
185.195.221.180
104.249.2.99
104.239.114.190
104.239.117.112
181.177.71.49
67.227.121.109
199.168.121.93
185.195.223.14
181.177.67.30
181.177.76.63
181.177.77.79
181.177.66.29
181.177.77.197
186.179.24.252
185.196.188.94
181.177.76.62
216.10.7.75
181.177.68.33
186.179.11.165
181.177.71.188
185.195.213.87
185.193.74.97
67.227.122.57
185.196.189.63
216.10.1.142
199.168.122.193
186.179.2.117
181.177.72.134
181.177.66.212
185.188.77.122
185.207.96.121
199.168.121.246
104.249.0.31
185.195.222.160 Potential Mitigation Strategies Web administrators are urged to monitor their logs for suspicious activity involving the Mozilla/5.0 (X11; Linux x86_64) user-agent string and IPs associated with the above ASNs. Immediate actions to consider include: Mozilla/5.0 (X11; Linux x86_64) Blocking or rate-limiting the offending IPs through firewall rules. Implementing CAPTCHA systems for suspicious traffic. Using bot protection plugins or services such as Cloudflare to prevent excessive scraping. Blocking or rate-limiting the offending IPs through firewall rules. Blocking or rate-limiting the offending IPs through firewall rules. Implementing CAPTCHA systems for suspicious traffic. Implementing CAPTCHA systems for suspicious traffic. Using bot protection plugins or services such as Cloudflare to prevent excessive scraping. Using bot protection plugins or services such as Cloudflare to prevent excessive scraping. The persistence of this botnet highlights the importance of continually refining web security measures to protect digital assets from unauthorized data scraping and potential attacks. This unknown botnet's activity is a pressing issue for WordPress site owners, and if left unchecked, its presence could lead to server overloads, data leaks, or even downtime for many websites. Interestingly, despite the aggressive nature of this botnet, none of the associated IPs have been flagged or reported in popular databases like AbuseIPDB. Even more surprising, these IPs are not natively blocked by Cloudflare’s managed rules, which typically catch such malicious behavior early on. This suggests that the botnet is currently flying under the radar, operating in a gray zone of anonymity. The lack of detection and reporting raises concerns about how sophisticated and stealthy this botnet may be. For now, it seems to be an entirely unknown entity in the cybersecurity landscape. However, it does seem look like it is scraping WordPress content. unknown entity

Unknown Botnet Using Mozilla/5.0 (X11; Linux x86_ User Agent Ignoring Crawl Delay on WordPress Sites

In recent weeks, webmasters have raised urgent concerns regarding Facebook's aggressive crawling practices, particularly highlighting the behavior of its user agent, “facebookexternalhit.” Many site owners report that these bots are creating significant strain on their web hosting servers, leading to alarming spikes in traffic that threaten site reliability. Several site owners have come forward with their experiences, describing the overwhelming impact of Facebook’s crawling activities on their websites. One webmaster recounted their situation, said, “Our website gets hammered every 45 to 60 minutes with spikes of approximately 400 requests per second from 20 to 30 different IP addresses within Facebook’s netblocks. Between these spikes, the traffic is manageable, but the sudden load is risky.” This sentiment resonates with many, as webmasters are advocating for a more balanced distribution of requests from Facebook’s bots, akin to the behavior exhibited by Googlebot and other reputable search engine crawlers. The consequences of these excessive requests extend beyond mere inconvenience; they disrupt the user experience and lead to costly resource consumption for site owners. Smaller websites, in particular, have found themselves severely impacted. In response to the relentless onslaught, some webmasters have taken proactive measures by implementing stricter rules in their robots.txt files to shield their servers from the overwhelming traffic. However, because Facebook’s bot functions as a scraper rather than a traditional crawler, it disregards these instructions, further complicating the situation. This growing issue has sparked widespread discussions within the web development community, with experts urging Facebook to reconsider its crawling strategies. The collective voice of these webmasters underscores a critical need for a more sustainable approach to web scraping and crawling practices. In a bid to manage the excessive requests, many webmasters are turning to tools like Cloudflare, which provides robust features for managing traffic and implementing rate limiting. By configuring a rate-limiting WAF rule, webmasters can effectively throttle the number of requests originating from Facebook’s bots, alleviating server strain during peak traffic periods. One webmaster expressed their perspective on the necessity of a balanced approach, stating, “I don’t want to block the bot entirely, but the current pattern is unsustainable. Using Cloudflare’s rate limiting has allowed us to protect our site while still enabling Facebook to access our content for link previews.” Concerns regarding Facebook’s crawling practices have been echoed on various platforms. In a post on Cloudflare, one user articulated their frustrations: “I am writing to express my concern about the excessive crawling activity of Facebook’s crawler. “This excessive crawling is causing significant performance issues and potential downtime for our website.” They went on to detail, “Our web server logs indicate that Facebook’s crawler (facebookexternalhit/1.1 – 2a03:2880:22ff:7::face is making multiple requests to our WordPress website every second, even during off-peak hours,” “During peak hours, the crawler’s activity spikes to tens of thousands of requests per minute. This excessive crawling is overwhelming our servers and causing them to slow down or even crash.” While webmasters recognize the necessity of Facebook’s crawler for indexing purposes and making their content accessible to users, they firmly believe that the current level of crawling is excessive and unreasonable. As a result, many remain vigilant, closely monitoring their server performance and adjusting settings to mitigate the challenges posed by Facebook’s bots. The unfolding situation highlights a critical juncture for the web development community, with potential implications for how major tech companies manage web scraping and crawling practices in the future. As webmasters advocate for a more equitable solution, the outcome of this discussion could set important precedents in the industry, influencing the relationship between webmasters and tech giants moving forward. In recent weeks, webmasters have raised urgent concerns regarding Facebook's aggressive crawling practices , particularly highlighting the behavior of its user agent, “facebookexternalhit.” Many site owners report that these bots are creating significant strain on their web hosting servers, leading to alarming spikes in traffic that threaten site reliability. Facebook's aggressive crawling practices Several site owners have come forward with their experiences, describing the overwhelming impact of Facebook’s crawling activities on their websites. Several site owners have come forward with their experiences, describing the overwhelming impact of Facebook’s crawling activities on their websites. One webmaster recounted their situation, said, “Our website gets hammered every 45 to 60 minutes with spikes of approximately 400 requests per second from 20 to 30 different IP addresses within Facebook’s netblocks. Between these spikes, the traffic is manageable, but the sudden load is risky.” This sentiment resonates with many, as webmasters are advocating for a more balanced distribution of requests from Facebook’s bots , akin to the behavior exhibited by Googlebot and other reputable search engine crawlers. Facebook’s bots The consequences of these excessive requests extend beyond mere inconvenience; they disrupt the user experience and lead to costly resource consumption for site owners. Smaller websites, in particular, have found themselves severely impacted. In response to the relentless onslaught, some webmasters have taken proactive measures by implementing stricter rules in their robots.txt files to shield their servers from the overwhelming traffic. However, because Facebook’s bot functions as a scraper rather than a traditional crawler, it disregards these instructions, further complicating the situation. This growing issue has sparked widespread discussions within the web development community, with experts urging Facebook to reconsider its crawling strategies. The collective voice of these webmasters underscores a critical need for a more sustainable approach to web scraping and crawling practices. In a bid to manage the excessive requests, many webmasters are turning to tools like Cloudflare, which provides robust features for managing traffic and implementing rate limiting. By configuring a rate-limiting WAF rule, webmasters can effectively throttle the number of requests originating from Facebook’s bots, alleviating server strain during peak traffic periods. One webmaster expressed their perspective on the necessity of a balanced approach, stating, “I don’t want to block the bot entirely, but the current pattern is unsustainable. Using Cloudflare’s rate limiting has allowed us to protect our site while still enabling Facebook to access our content for link previews.” Concerns regarding Facebook’s crawling practices have been echoed on various platforms. In a post on Cloudflare, one user articulated their frustrations: “I am writing to express my concern about the excessive crawling activity of Facebook’s crawler. “This excessive crawling is causing significant performance issues and potential downtime for our website.” They went on to detail, “Our web server logs indicate that Facebook’s crawler (facebookexternalhit/1.1 – 2a03:2880:22ff:7::face is making multiple requests to our WordPress website every second, even during off-peak hours,” “During peak hours, the crawler’s activity spikes to tens of thousands of requests per minute. This excessive crawling is overwhelming our servers and causing them to slow down or even crash.” While webmasters recognize the necessity of Facebook’s crawler for indexing purposes and making their content accessible to users, they firmly believe that the current level of crawling is excessive and unreasonable. As a result, many remain vigilant, closely monitoring their server performance and adjusting settings to mitigate the challenges posed by Facebook’s bots. The unfolding situation highlights a critical juncture for the web development community, with potential implications for how major tech companies manage web scraping and crawling practices in the future. As webmasters advocate for a more equitable solution, the outcome of this discussion could set important precedents in the industry, influencing the relationship between webmasters and tech giants moving forward.

FaceBook Bots, Crawlers And User Agents Causing Resource Drains On Websites And Hosting Accounts

Authors:
(1) Hamid Reza Saeidnia, Department of Information Science and Knowledge Studies, Tarbiat Modares University, Tehran, Islamic Republic of Iran;
(2) Elaheh Hosseini, Department of Information Science and Knowledge Studies, Faculty of Psychology and Educational Sciences, Alzahra University, Tehran, Islamic Republic of Iran;
(3) Shadi Abdoli, Department of Information Science, Université de Montreal, Montreal, Canada
(4) Marcel Ausloos, School of Business, University of Leicester, Leicester, UK and Bucharest University of Economic Studies, Bucharest, Romania. Table of Links Abstract and Introduction Materials and Methods Results RQ 1: AI and scientometrics RQ 2: AI and webometrics RQ 3: AI and bibliometrics Discussion RQ 4: Future of Scientometrics, Webometrics, and Bibliometrics with AI RQ 5: Ethical Considerations of Scientometrics, Webometrics, and Bibliometrics with AI Conclusion, Limitations, and References RQ 2: AI and webometrics In webometrics, AI can provide several specific benefits including Web Crawling and Data Collection, Web Link Analysis, Web Content Analysis, Social Media Analysis, Web Impact Analysis, and Recommender Systems as sketched in Figure 3, and e.g. it has been demonstrated through papers like [9, 10, 21, 36-45]. These 6 considerations point to the potential benefits and suggest focused strategies for utilizing AI capabilities in webometrics. The resulting findings highlight how AI can improve the quality, accessibility, and data collection processes in webometrics analyses, as outlined in Table 2. Indeed, algorithms based on artificial intelligence can automatically crawl and collect data from websites, including institutional websites, scientific research portals, and online repositories [39, 42]. This enables researchers to gather large amounts of web-based information for analysis, including publication data, author profiles, and citation patterns. In order to understand the relationship between publications, websites, and authors, artificial intelligence approaches can analyze hyperlink structures and web link patterns [9, 43]. By analyzing the link structure, AI algorithms can identify influential websites and authors, as well as detect communities, collaborations, and research networks within the web-based scientific ecosystem [17]. AI techniques, such as natural language processing and machine learning, can be employed to analyze the content of webpages and scientific publications available online [40, 41]. This enables researchers to extract key information, such as keywords, topics, and sentiments, from web-based documents, facilitating comprehensive analysis and understanding of research outputs. AI can analyze social media platforms, such as Twitter, to understand the online discussions, trends, and interactions related to scientific research [36, 38, 44]. By analyzing hashtags, mentions, and user behavior, AI algorithms can identify influential research topics, key opinion leaders, and potential collaborations within the online scientific community, as demonstrated in such previous works. AI can assess the impact and visibility of scientific research on the web [37, 46]. Indeed, by analyzing web traffic, page views, and social media metrics, AI algorithms can provide insights into the online visibility, dissemination, and engagement of scientific publications, authors, and research institutions. “Finally”, AI-powered recommender systems can assist researchers in discovering relevant scientific websites, online resources, and research collaborations [35, 45]. These papers, based on user preferences, reading behavior, and web usage data, show that personalized recommendations can be generated using AI algorithms, making it easier for researchers to explore the web-based scientific landscape and discover new opportunities for further research. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Hamid Reza Saeidnia, Department of Information Science and Knowledge Studies, Tarbiat Modares University, Tehran, Islamic Republic of Iran; (2) Elaheh Hosseini, Department of Information Science and Knowledge Studies, Faculty of Psychology and Educational Sciences, Alzahra University, Tehran, Islamic Republic of Iran; (3) Shadi Abdoli, Department of Information Science, Université de Montreal, Montreal, Canada (4) Marcel Ausloos, School of Business, University of Leicester, Leicester, UK and Bucharest University of Economic Studies, Bucharest, Romania. Authors: Authors: (1) Hamid Reza Saeidnia, Department of Information Science and Knowledge Studies, Tarbiat Modares University, Tehran, Islamic Republic of Iran; (2) Elaheh Hosseini, Department of Information Science and Knowledge Studies, Faculty of Psychology and Educational Sciences, Alzahra University, Tehran, Islamic Republic of Iran; (3) Shadi Abdoli, Department of Information Science, Université de Montreal, Montreal, Canada (4) Marcel Ausloos, School of Business, University of Leicester, Leicester, UK and Bucharest University of Economic Studies, Bucharest, Romania. Table of Links Abstract and Introduction Abstract and Introduction Materials and Methods Materials and Methods Results Results RQ 1: AI and scientometrics RQ 1: AI and scientometrics RQ 2: AI and webometrics RQ 2: AI and webometrics RQ 3: AI and bibliometrics RQ 3: AI and bibliometrics Discussion Discussion RQ 4: Future of Scientometrics, Webometrics, and Bibliometrics with AI RQ 4: Future of Scientometrics, Webometrics, and Bibliometrics with AI RQ 5: Ethical Considerations of Scientometrics, Webometrics, and Bibliometrics with AI RQ 5: Ethical Considerations of Scientometrics, Webometrics, and Bibliometrics with AI Conclusion, Limitations, and References Conclusion, Limitations, and References RQ 2: AI and webometrics In webometrics, AI can provide several specific benefits including Web Crawling and Data Collection, Web Link Analysis, Web Content Analysis, Social Media Analysis, Web Impact Analysis, and Recommender Systems as sketched in Figure 3, and e.g. it has been demonstrated through papers like [9, 10, 21, 36-45]. These 6 considerations point to the potential benefits and suggest focused strategies for utilizing AI capabilities in webometrics. The resulting findings highlight how AI can improve the quality, accessibility, and data collection processes in webometrics analyses, as outlined in Table 2. Indeed, algorithms based on artificial intelligence can automatically crawl and collect data from websites, including institutional websites, scientific research portals, and online repositories [39, 42]. This enables researchers to gather large amounts of web-based information for analysis, including publication data, author profiles, and citation patterns. In order to understand the relationship between publications, websites, and authors, artificial intelligence approaches can analyze hyperlink structures and web link patterns [9, 43]. By analyzing the link structure, AI algorithms can identify influential websites and authors, as well as detect communities, collaborations, and research networks within the web-based scientific ecosystem [17]. AI techniques, such as natural language processing and machine learning, can be employed to analyze the content of webpages and scientific publications available online [40, 41]. This enables researchers to extract key information, such as keywords, topics, and sentiments, from web-based documents, facilitating comprehensive analysis and understanding of research outputs. AI can analyze social media platforms, such as Twitter, to understand the online discussions, trends, and interactions related to scientific research [36, 38, 44]. By analyzing hashtags, mentions, and user behavior, AI algorithms can identify influential research topics, key opinion leaders, and potential collaborations within the online scientific community, as demonstrated in such previous works. AI can assess the impact and visibility of scientific research on the web [37, 46]. Indeed, by analyzing web traffic, page views, and social media metrics, AI algorithms can provide insights into the online visibility, dissemination, and engagement of scientific publications, authors, and research institutions. “Finally”, AI-powered recommender systems can assist researchers in discovering relevant scientific websites, online resources, and research collaborations [35, 45]. These papers, based on user preferences, reading behavior, and web usage data, show that personalized recommendations can be generated using AI algorithms, making it easier for researchers to explore the web-based scientific landscape and discover new opportunities for further research. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

"In webometrics, AI can provide several specific benefits including Web Crawling, Data Collection...

"ওয়েবমেট্রিক্সে, AI ওয়েব ক্রলিং, ডেটা সংগ্রহ সহ বেশ কিছু নির্দিষ্ট সুবিধা প্রদান করতে পারে...

„In der Webometrie kann KI mehrere spezifische Vorteile bieten, darunter Web-Crawling und Datenerfassung …

"En webmetría, la IA puede proporcionar varios beneficios específicos, incluido el rastreo web, la recopilación de datos...

"En webométrie, l'IA peut apporter plusieurs avantages spécifiques, notamment l'exploration du Web, la collecte de données...

"वेबमेट्रिक्स में, एआई वेब क्रॉलिंग, डेटा संग्रह सहित कई विशिष्ट लाभ प्रदान कर सकता है...

「ウェブメトリクスでは、AI は Web クロール、データ収集など、いくつかの具体的なメリットを提供できます...」

"웹 측정학에서 AI는 웹 크롤링, 데이터 수집 등 여러 가지 구체적인 이점을 제공할 수 있습니다.

"Na webometria, a IA pode fornecer vários benefícios específicos, incluindo rastreamento da Web, coleta de dados...

«В вебометрике искусственный интеллект может предоставить несколько конкретных преимуществ, включая веб-сканирование, сбор данных…

"Webometride yapay zeka, Web Taraması, Veri Toplama gibi çeşitli belirli faydalar sağlayabilir...

"Trong webometrics, AI có thể mang lại một số lợi ích cụ thể bao gồm Thu thập thông tin trên web, Thu thập dữ liệu...

“在网络计量学中，人工智能可以提供一些特定的好处，包括网络爬虫、数据收集……

OpenAI launches a default opt-in crawler to scrape the Internet, while FTC pursues an obscure consumer deception investigation Last week, Open AI (maker of ChatGPT) officially announced their — this is a piece of software that scrapes content from all websites across the internet, which is then used for AI model training. web crawler The existence of the crawler is not surprising and several legitimate web crawlers exist today, including Google’s crawler that indexes the entire internet. However, this is the first time OpenAI explicitly announced its existence and also provided a mechanism for websites to opt out of being scraped. Note that the crawler is , i.e., you need to explicitly change a piece of code on your website to ask the crawler not to scrape your data. Opt-in /out defaults are sticky and often determine what the majority behavior is because most people don’t take the effort to change defaults. opt-in by default It is the same reason why have had a major impact on the digital advertising industry. Apple’s iOS14 privacy changes So, why even provide the opt-out? This is likely a preemptive move from OpenAI in response to against the company alleging that content owners’ copyright was infringed (deeper article on if you want to poke more). recent lawsuits data scraping ChatGPT competitor Google Bard faces a similar challenge but Google has not yet announced an equivalent solution — they did put out a request for comment on how to upgrade to address this issue (written with some ). robots.txt neat PR penmanship In this article, we’ll dive into: Implications of OpenAI’s crawler for content owners FTC’s current investigation into OpenAI Today’s legal landscape that we operate in Why the FTC’s approach of going after OpenAI is (yet another) misstep Implications of OpenAI’s Crawler for Content Owners While the announcement provides an option for advertisers to block OpenAI’s crawler from scraping their data, a couple of things are not great: It’s opt-in by default, which means OpenAI can keep scraping till sites explicitly tell them not to There (which would essentially be the case with anyone who is forced into a default opt-in) hasn’t been a clear legal ruling one way or another about content owners’ rights when their data is scraped for model training without consent Today, there are two legal constructs that determine whether it’s okay or not for language models to take all this data without consent — . Copyright and Fair Use Copyright provides protection to specific types of content but also has carve-outs /exceptions: Copyright protection subsists, in accordance with this title, in original works of authorship fixed in any tangible medium of expression, now known or later developed, from which they can be perceived, reproduced, or otherwise communicated, either directly or with the aid of a machine or device. Works of authorship include the following categories: (1) literary works; (2) musical works, including any accompanying words; (3) dramatic works, including any accompanying music; (4) pantomimes and choreographic works; (5) pictorial, graphic, and sculptural works; (6) motion pictures and other audiovisual works; (7) sound recordings; and (8) architectural works. (b) In no case does copyright protection for an original work of authorship , regardless of the form in which it is described, explained, illustrated, or embodied in such work extend to any idea, procedure, process, system, method of operation, concept, principle, or discovery For example, copyright protects most original work (e.g., if you wrote an original blog article or book on a topic) but (e.g., you cannot claim that you were the first person to write about how AI impacts data rights, and therefore the idea belongs to you). does not protect broad ideas Another carve-out/exception from Copyright protection is Fair Use: The fair use of a copyrighted work, including such use by reproduction in copies or phonorecords or by any other means specified by that section, for purposes such as criticism, comment, news reporting, teaching (including multiple copies for classroom use), scholarship, or research, is not an infringement of copyright. In determining whether the use made of a work in any particular case is a fair use the factors to be considered shall include (1) the purpose and character of the use, including or is for nonprofit educational purposes; (2) the nature of the copyrighted work; (3) the amount and substantiality of the portion used in relation to the copyrighted work as a whole; and (4) the effect of the use upon the potential market for or value of the copyrighted work. whether such use is of a commercial nature For example, if you picked up content from a research paper and wrote a critique about it, that’s okay, and you are not infringing on the content owner’s copyright. It’s the same situation when I link another article from this page and add quoted text from that article. Both of these concepts were created to protect content owners’ rights while also allowing the free flow of information, especially in the context of education, research, and critique. I am not a legal expert but based on my research/understanding of the language above, is: where this gets fuzzy with AI models scraping training content AI companies typically scrape full text from a content owner’s website (this is protected by Copyright), train the models to learn the “idea”/“concept”/“principle” (this is not protected by Copyright), and then the models eventually spit out different text. In this case, does the content owner receive Copyright protection or not? Since the trained language models are now eventually used for commercial purposes (e.g., ChatGPT Plus is a paid product), is that a violation of the content owner’s Copyright (because the Fair Use exception no longer applies)? There have been no court rulings around this yet, so it’s hard to predict where this lands. My not-a-lawyer take is that the second one is probably easier to land: OpenAI scraped data and used it to create a commercial product, and therefore, they do not get an exception under Fair Use. I would imagine the first one (did the model train on an “idea” or just original text) is anyone’s guess. Note that both those bullets need to be in content owners’ favor for them to win, i.e., content owners only win if both the above exceptions (“idea” exception or Fair Use exception) don’t apply to OpenAI. I bring up this nuance because in the spectrum of AI risks (non-exhaustive) — from content owners’ rights to amplifying fraud to jobs being automated to AGI / destruction of humanity — the , as evidenced by the flurry of lawsuits and the impact on content platforms (e.g., ). most pressing near-term issue is content owners’ rights the StackOverflow story While regulators like the FTC can ponder about the really long-term problems and come up with hypothetical/creative ways to address these risks, their real short-term potential lies in being able to tackle risks that will impact us in the 5–10 year horizon. Like copyright infringement. Which brings us to what the FTC is doing about it. FTC’s Current Investigation Into OpenAI In mid-July, FTC announced that it is investigating OpenAI. What makes it interesting (and frustrating) is . the reason FTC is investigating them for The maker of ChatGPT is being investigated to evaluate whether the company broke any by . consumer protection laws putting personal reputation and data at risk Doesn’t make sense? You’re not alone. Let’s lay out some more background on how this came to be. The FTC’s on AI regulation came out in April: “There is no AI exemption to the laws on the books, and the FTC will vigorously enforce the law to combat unfair or deceptive practices or unfair methods of competition.” most vocal stance Then came a couple of defamation-related issues: Radio host after ChatGPT accused him of defrauding a non-profit, and a law professor was . Mark Walters sued OpenAI falsely accused by ChatGPT of sexual harassment Both these scenarios suck for the people involved, and I empathize with that. However, it’s a known fact that language models (like GPT) and products built on top of them (like ChatGPT) “hallucinate” and are often incorrect. The first half of FTC’s premise for the investigation is that — ChatGPT hallucinates and therefore creates reputational harm. In a heated Congressional hearing, one representative (rightfully) why they are going after defamation and libel, which are typically handled by state laws. FTC Chairperson Lina Khan gives a : asks the FTC convoluted argument Khan responded that libel and defamation aren’t a focus of FTC enforcement, but that misuse of people’s private information in AI training could be a form of fraud or deception under the FTC Act. “We’re focused on, ‘Is there substantial injury to people?’ Injury can look like all sorts of things,” Khan said. To tie up the full argument — FTC is saying that . ChatGPT’s hallucination produces incorrect information (including defamation), which then could be a form of consumer deception Additionally, sensitive user private information could have been used/leaked (based on that OpenAI quickly fixed). one bug As part of the investigation, the FTC has asked for a long list of things from OpenAI —  from details about how their model is trained to what data sources they use to how they position their product to customers to situations where model releases have been paused because of identified risks. The question is — Is the best approach for the FTC to regulate what is arguably going to be one of the largest AI companies, especially given the current legal landscape? Today’s Legal Landscape That We Operate in To critique FTC’s strategy with OpenAI, it’s useful to understand the legal landscape we operate in today. We won’t go into too much detail, but let’s do this briefly with the as an example: history of anti-trust In the 1900s, massive conglomerates (“trusts”) came into existence, and the balance of public-private power shifted to these companies. In response, the Sherman Act of 1890 was passed to add checks on private power and preserve competition; this law was used to litigate and break down “trusts” that were engaged in anti-competitive practices (predatory pricing, cartel deals, distribution monopoly). Around the 1960s, judges faced a lot of backlash for judging based on the spirit of the law instead of the letter of the law; for example, interpreting the Sherman law to determine if a set of companies “unreasonably restrain trade” involved subjectivity, and judges were accused of engaging in judicial activism. To introduce objectivity, the Chicago School pioneered the consumer welfare standard — “courts should be guided exclusively by consumer welfare” (e.g., a monopoly increasing prices in a blatant manner is wrong but, for other activities, the burden of proof is on regulators to prove consumer harm.) This continues to be the standard today and is one of the reasons the FTC and DOJ have a difficult job taking down big tech — for example, the FTC cannot make the argument that Google is increasing prices since most of their products are free, even if Google is engaged in other anti-competitive practices. The takeaway from this is — we continue to operate today in a landscape where cases are litigated heavily on the “letter of the law” and not the “spirit of the law.” This, along with the composition of the US Supreme Court today, has resulted in fairly conservative interpretations of the law. What this means for the FTC is to embrace the reality of this landscape and . The operating model of the FTC and DOJ (rightfully so) is to go after a handful of big cases and lay down harsh enforcement so that the long tail of companies think twice before breaking laws. figure out a way to win cases To make that happen, and it needs a . FTC needs to win big on a few issues, winning strategy within the constraints of the current legal landscape Why the FTC’s Approach of Going After OpenAI Is (Yet Another) Misstep The FTC has had a streak of losses against Big Tech, and I would argue that the losses can all be attributed to a failed “we hate everything big tech”, hammer-not-scalpel strategy of taking on these companies. For example, FTC took a brute-force approach to stop the $69B Microsoft-Activision acquisition and (pretty badly, I’d say). FTC argued that Microsoft acquiring Activision would kill competition in the gaming market. lost The judge wrote a fairly throwing out all of FTC’s arguments; here’s one of the judge’s comments: blunt ruling There are no internal documents, emails, or chats contradicting Microsoft’s stated intent not to make Call of Duty exclusive to Xbox consoles. Despite the completion of extensive discovery in the FTC administrative proceeding, including production of nearly 1 million documents and 30 depositions, the FTC has not identified a single document which contradicts Microsoft’s publicly-stated commitment to make Call of Duty available on PlayStation (and Nintendo Switch). Another brute force case was the FTC’s attempt to block Meta’s acquisition of a VR company Within, and . Why did they pursue this? They wanted to test out the waters to see if there was an appetite to block acquisitions before a particular market becomes large, and given the current legal landscape, it was unsurprisingly thrown out. they lost The problem with FTC’s investigation of OpenAI is similar: They are going after (what in my opinion) is a pretty trivial issue and a known limitation of language models — hallucinations; they should instead be focusing on actual AI issues that matter in the 5–10 year horizon, like Copyright. Despite multiple “creative” legal approaches being thrown out in the current legal landscape, they are attempting another creative argument: hallucination → defamation → consumer deception. The generous interpretation of their actions is that they want to set a precedent for their “AI is not exempt from existing laws” stance and that this wild goose chase gets them a large amount of self-reported data from OpenAI (FTC issues ). 20 pages of asks However, given their track record of repeatedly pursuing brute force/anything big tech is uncompetitive approach + combining those with creative arguments which are getting repeatedly dismissed in courts, I believe that the FTC has not earned the benefit of the doubt in this case. Conclusion I absolutely think OpenAI should be regulated. Not because their LLMs hallucinate (of course, they do) but because they are blatantly using creators’ content without permission. Not because it will change the past but because it will help set up content owners for a healthy future where their copyrights cannot be blatantly infringed upon. But the FTC is repeating its missteps with the hammer-not-scalpel approach. There is a clear precedent for successes against big tech with a scalpel approach, the most notable one being UK’s Competition and Markets Authority. The two big cases they won against Google have focused on specific anti-competitive mechanisms: to its own product in the AdTech stack and allowing for in-app payments. stopping Google from providing preferential treatment other payment providers If FTC continues on its current path, its streak of losses is going to embolden tech companies to continue doing whatever they want because they know they can win in court. It’s time the FTC reflected on its failures, learned from other regulators’ successes, and course corrected. 🚀 If you liked this piece, consider subscribing to Every week, I publish one in the form of a 10-minute read. . my weekly newsletter deep-dive analysis on a current tech topic/product strategy Best, Viggy. Also published here

A Look Inside OpenAI's Web Crawler and the Continuous Missteps of the FTC

OpenAI-এর ওয়েব ক্রলার এবং FTC-এর ক্রমাগত ভুল-ভ্রান্তির ভিতরের দিকে নজর দিন

Ein Blick in den Webcrawler von OpenAI und die ständigen Fehltritte der FTC

Una mirada al interior del rastreador web de OpenAI y los continuos errores de la FTC

Un regard sur le Web Crawler d'OpenAI et les faux pas continus de la FTC

ओपनएआई के वेब क्रॉलर और एफटीसी की निरंतर गलतियों के अंदर एक नज़र

OpenAI の Web クローラーの内部と FTC の連続する失敗を振り返る

OpenAI의 웹 크롤러와 FTC의 지속적인 실수 살펴보기

Uma olhada no rastreador da Web da OpenAI e os erros contínuos da FTC

OpenAI'nin Web Tarayıcısına ve FTC'nin Sürekli Yanlış Adımlarına Bir Bakış

Nhìn vào bên trong Trình thu thập dữ liệu web của OpenAI và những sai lầm liên tục của FTC

深入了解 OpenAI 的网络爬虫以及 FTC 的持续失误

Optimizing for Crawl Budget is an often-overlooked t hack that can have a big impact on larger websites. echnical SEO Generally speaking, Google is really good at indexing pages. But if your site has thousands of pages (an e-commerce site, perhaps) or has recently added a whole new section, containing hundreds of new pieces of content; or even if they’re a mid-sized site but with lots of redirects, you’re going to want to think about crawl budget. What exactly is a crawl budget? Google divides its attention among millions of websites every day. Since it doesn’t have unlimited resources, it assigns a ‘budget’ to your site that’s generally determined by its size, ‘health’, and backlink portfolio. It also takes into account the following two factors: Crawl limit or host load - how much crawling can a website handle and what are its owner's preferences? Crawl demand or crawl scheduling - which URLs are worth (re)crawling the most, based on their popularity and how often it's being updated. How and when Google decides to crawl (or re-crawl) a page is not always predictable. It could be because new links have appeared, pointing at that content; perhaps someone shared it on social media, or maybe the URL was updated in the XML sitemap. It’s a common misconception that Google crawls pages as soon as they’re published. In fact, sometimes it can take weeks, which might get in the way of your SEO efforts. This is especially true for larger sites that have 10,000 or more pages. These sites are at particular risk of having some of their most valuable content falling into the big, black Google vortex – and who wants that! The truth is: If Google doesn’t index a page, it’s not going to rank for anything. And, if it doesn’t rank, it’s not going to be found. So, if your number of pages exceeds your site’s crawl budget, there are going to be pages on your site that aren’t indexed and aren’t seen. Similarly, if you’re under your crawl budget but still have a medium to large site that doesn’t have a clear site structure, you can guarantee that it won’t be crawled as efficiently as you might like. When crawl budget becomes a problem Let’s say your site has 50,000 pages and Google is only allotted to crawl 2,000 each day. What will happen is that some pages (like the homepage) will get crawled more than others, which means it could take weeks before other pages (those that aren’t optimized as well) get noticed. To determine whether your site has a crawl budget issue, you can follow the steps below: Determine how many pages you have on your site. One way to do this would be to check the number of URLs in your XML sitemap. Go into Google Search Console. Go to “Settings” -> “Crawl stats” and take note of the average pages crawled per day. Divide the number of pages by the “Average crawled per day” number. If you end up with a number lower than 3, that’s ok. However, If you end up with a number higher than ~10 (so you have 10x more pages than the number Google crawls each day), you may have an issue. The next step will be to find out which URLs Google is currently crawling on your site by looking at your site’s server logs. Depending on your website host, you might not always be able to access your log files super easily. However, if your site is big, then I’d strongly recommend changing hosts to one that DOES let you see them. It matters. You can’t fix a site by looking at it from the outside. You need to get in there and find the problem, otherwise, it’s all guesswork. When checking how many times Google is crawling your website, it’s always safest to check multiple sources. Your site’s server logs and Google Console are the best places to start. For Google Console, follow these steps: Log in to Google Search Console and choose your website. Go to Crawl > Crawl Stats. There you can see the number of pages that Google crawls per day. While looking at your site’s server logs, you’ll also see which pages are classed as 404 – meaning ‘Page Not Found’. Don’t ignore these. As a rule, every page should either be a 200 (an ‘OK’ page) or a 301 (meaning a redirect). Anything else is known as a ‘non-indexable’ page and should be fixed as soon as possible. Otherwise, you're keeping search engines busy sifting through irrelevant pages. This is a waste of crawl budget, and the number one rule of optimizing your crawl budget is making sure it isn’t wasted! As obvious as it sounds, it’s much quicker to make fixes to existing crawl budget issues than to begin work on organically increasing them. With that in mind, here are some other common reasons for wasted crawl budget: 1. Duplicate content Pages that have exactly the same (or very similar) content, are known as duplicates. These might be copied pages, internal search results pages, or something else. Limiting duplicate content is a good idea for lots of reasons. One is the crawl budget. You don’t want search engines to waste their time, looking through multiple copies of the same page. So it's important to prevent, or at the very least minimize, the duplicate content on your site. You can do this by: Setting up website redirects for all domain variants (HTTP, HTTPS, non-WWW, and WWW). Make internal search result pages inaccessible to search engines using your robots.txt. Disabling dedicated pages for images. Be careful around your use of taxonomies such as categories and tags. Duplicate content will sometimes be necessary (for example if you have print-friendly versions of certain web pages), but there are things you can do to signal to Google that you don’t need these pages to be indexed. 2. Pages with high load times Site pages that take a long time to load have a negative impact on the crawl budget. This is because it’s a sign to search engines that your site can’t handle the request and Google will therefore de-prioritize that page. They may even adjust your crawl limit. Another reason is that a faster-loading website means Google can crawl more URLs, more quickly. Although it’s a small fix, some sites have seen their crawl budget more than double when they upgraded it to be faster. But putting crawl budget aside for a moment – slow load times (2 seconds or more) can significantly hurt your visitor’s user experience, resulting in a lower conversion rate. All in all, slow loads are a bad move for SEO. 3. Bad internal linking structure Backlinks aside, how pages within your website link to one another play an important role in crawl budget optimization. If your internal link structure isn't set up correctly, search engines may not pay enough attention to some of your pages. So make sure that your most important pages have plenty of internal links. Also avoid unbalanced linking – pages that just have one or two links, compared to others that have tens. And don’t forget about external links, either. The dream would be to have a high-authority backlink pointing to every page on your site, but that’s not realistic. Build up your backlinks, but focus more on internal linking for immediate structural benefits. 4. Avoid ‘Orphan Pages’ Orphan pages are pages that have no internal or external links pointing to them. Google has a really hard time finding orphan pages. So, if you want to get the most out of your crawl budget, make sure that there are at least two internal or external links pointing to every page on your site. Once you’ve identified an orphan page, there are multiple different ways to go about fixing it. One option is to archive the page. This way, the page, and its information is still viewable, but it’s no longer part of your live site. Another method would be to set up a redirect of the URL to a new location; ideally a relevant equivalent page or similar. Crawlers and site visitors will then be redirected to a page that you want them to see, and crawlers will index it accordingly. 5. Broken and redirecting links Broken links and long chains of redirects are dead ends for search engines. Google follows a maximum of five chained redirects in one crawl before they leave it – either entirely or to crawl later. This is why it’s strongly advised that you avoid chained redirects and keep the usage of redirects to a minimum. By fixing broken links and redirecting links, you can quickly recover the wasted crawl budget. You’ll also be significantly improving your site visitor's user experience. Redirects (and chains of redirects, in particular) cause longer page load times and consequently hurt the overall user experience. 6. Avoid Low-quality content Pages with little (or thin) content aren't interesting to search engines and do not tend to rank well. Avoid them completely, if possible. Or, at the very least, tell Google that you don’t want them to be indexed using your robots.txt file. However, you should ONLY do this if you’re sure what you’re doing. Otherwise, you could be blocking some of your most valuable content from being indexed. In conclusion, when it comes to large sites, crawl budget is one of the most important factors from a technical SEO perspective. That’s because, if you do it right, you’ll automatically be optimizing for internal linking, page speed, URL errors, low-quality content, and more, too – all very important SEO success factors. And, while smaller sites needn’t worry so much about crawl budget, the same principles of optimization will certainly help them to rank. Also published here.

Why Crawl Budget Matters for Larger Sites

Let’s talk about something that’s been driving me nuts for years: Google’s ongoing struggles with handling canonical URLs and duplicate content. It’s a problem that, despite being fundamental to search engine optimization (SEO), Google still can’t seem to get right. And while Google continues to flounder, guess who’s quietly acing it? Bing. Yes, you read that right – Bing! It’s honestly baffling. Here we are, in 2024, and Google, the self-proclaimed king of search, can’t even get the basics of web indexing straight. Let’s break this down. The Canonical URL Catastrophe A canonical URL is meant to tell search engines which version of a page is the “official” one when you have multiple URLs with similar or identical content. Seems simple enough, right? You’d think Google, with all its technological might and years of search engine dominance, would have perfected this by now. But nope. Google’s handling of canonical tags is, more often than not, a dumpster fire. Why? Because Google has a tendency to ignore your carefully placed canonical tags, or worse, it just gets them wrong. It might pick the wrong version of your page, or even worse – it might randomly declare some other URL as the canonical without any explanation. This leads to pages getting improperly indexed, with duplicate content spread across multiple URLs and absolutely no way for you to control it. Result? Your rankings plummet, your SEO strategy crumbles, and your site’s traffic tanks – all thanks to Google’s refusal to play by the canonical rulebook. But, oh, here’s where it gets good. Bing? Bing doesn’t have these problems. Bing can crawl your site, see your canonical tag, and follow it like a well-behaved child. It doesn’t second-guess you, doesn’t ignore your instructions, and doesn’t make arbitrary decisions that hurt your visibility. Bing understands that when a site owner points to a canonical URL, that’s what it should respect. It’s as if Bing actually listens to website owners, while Google seems to be playing SEO bingo with your site’s content. Duplicate Content Woes Then there’s duplicate content. Another area where Google’s “genius” just falls flat. We all know duplicate content is a big issue when it comes to SEO. When search engines find multiple copies of the same content, they’re supposed to figure out which one is the most relevant and display it in search results. Seems simple, right? Well, it’s not. Google has been notorious for misidentifying duplicate content, sometimes even penalizing sites with pages that have only slight variations. I’ve seen it happen time and again: you publish a well-researched article, and somehow, Google finds another page on your site and decides they’re too similar, tanking your rankings. Why does this happen? It’s because Google’s algorithm, despite being incredibly complex, still can’t reliably handle duplicate content in certain situations. There are cases where Google just can’t seem to distinguish between genuinely different pages and what’s essentially the same content presented slightly differently – even if those pages are clearly differentiated by their canonical tags or intended purpose. The result? Duplicate content penalties, frustrated webmasters, and a whole lot of wasted time trying to get Google to "understand" your site. Now, Bing? Bing doesn’t overcomplicate things. It doesn’t see minor content variations and freak out like Google does. It doesn’t penalize you for every slight difference between two pages that have the same general topic. Instead, Bing takes a more balanced approach to identifying and handling duplicate content. You won’t find yourself scrambling to fix duplicate content issues that Google invented in the first place. It just works. The Moral of the Story Why is Bing, a search engine that many still dismiss, consistently getting the basics right while Google is out here making life unnecessarily difficult for website owners? The answer is simple: Google is overcomplicating things, and it’s failing to acknowledge the importance of consistency and accuracy in indexing and content management. In contrast, Bing seems to have a more grounded, practical approach. Bing might not dominate the search engine market the way Google does, but at least when it comes to handling canonical URLs and duplicate content, Bing is getting it right. It’s time for Google to stop pretending that its “complex algorithms” are the answer to everything. Sometimes, simplicity works better. Maybe it’s time for Google to take a page out of Bing’s playbook and fix the very issues that have been plaguing SEO professionals for years. Because if Bing can get it right, there’s absolutely no excuse for Google to keep getting it wrong. Let’s talk about something that’s been driving me nuts for years: Google’s ongoing struggles with handling canonical URLs and duplicate content. It’s a problem that, despite being fundamental to search engine optimization (SEO), Google still can’t seem to get right. And while Google continues to flounder, guess who’s quietly acing it? Bing. Yes, you read that right – Bing! And while Google continues to flounder, guess who’s quietly acing it? Bing. Yes, you read that right – Bing! It’s honestly baffling. Here we are, in 2024, and Google, the self-proclaimed king of search, can’t even get the basics of web indexing straight. Let’s break this down. The Canonical URL Catastrophe A canonical URL is meant to tell search engines which version of a page is the “official” one when you have multiple URLs with similar or identical content. Seems simple enough, right? You’d think Google, with all its technological might and years of search engine dominance, would have perfected this by now. But nope. Google’s handling of canonical tags is, more often than not, a dumpster fire. Why? Because Google has a tendency to ignore your carefully placed canonical tags, or worse, it just gets them wrong. It might pick the wrong version of your page, or even worse – it might randomly declare some other URL as the canonical without any explanation. ignore randomly This leads to pages getting improperly indexed, with duplicate content spread across multiple URLs and absolutely no way for you to control it. Result? Your rankings plummet, your SEO strategy crumbles, and your site’s traffic tanks – all thanks to Google’s refusal to play by the canonical rulebook. SEO strategy But, oh, here’s where it gets good. Bing ? Bing doesn’t have these problems. Bing can crawl your site, see your canonical tag, and follow it like a well-behaved child. It doesn’t second-guess you, doesn’t ignore your instructions, and doesn’t make arbitrary decisions that hurt your visibility. Bing Bing understands that when a site owner points to a canonical URL, that’s what it should respect. It’s as if Bing actually listens to website owners, while Google seems to be playing SEO bingo with your site’s content. Duplicate Content Woes Then there’s duplicate content. Another area where Google’s “genius” just falls flat. We all know duplicate content is a big issue when it comes to SEO. When search engines find multiple copies of the same content, they’re supposed to figure out which one is the most relevant and display it in search results. Seems simple, right? Well, it’s not. Google has been notorious for misidentifying duplicate content, sometimes even penalizing sites with pages that have only slight variations. I’ve seen it happen time and again: you publish a well-researched article, and somehow, Google finds another page on your site and decides they’re too similar, tanking your rankings. Why does this happen? It’s because Google’s algorithm, despite being incredibly complex, still can’t reliably handle duplicate content in certain situations. There are cases where Google just can’t seem to distinguish between genuinely different pages and what’s essentially the same content presented slightly differently – even if those pages are clearly differentiated by their canonical tags or intended purpose. The result? Duplicate content penalties, frustrated webmasters, and a whole lot of wasted time trying to get Google to "understand" your site. Now, Bing? Bing doesn’t overcomplicate things. It doesn’t see minor content variations and freak out like Google does. It doesn’t penalize you for every slight difference between two pages that have the same general topic. Instead, Bing takes a more balanced approach to identifying and handling duplicate content. You won’t find yourself scrambling to fix duplicate content issues that Google invented in the first place. It just works. The Moral of the Story Why is Bing, a search engine that many still dismiss, consistently getting the basics right while Google is out here making life unnecessarily difficult for website owners? The answer is simple: Google is overcomplicating things, and it’s failing to acknowledge the importance of consistency and accuracy in indexing and content management . Google is overcomplicating things, and it’s failing to acknowledge the importance of consistency and accuracy in indexing and content management In contrast, Bing seems to have a more grounded, practical approach. Bing might not dominate the search engine market the way Google does, but at least when it comes to handling canonical URLs and duplicate content, Bing is getting it right. It’s time for Google to stop pretending that its “complex algorithms” are the answer to everything. Sometimes, simplicity works better. Maybe it’s time for Google to take a page out of Bing’s playbook and fix the very issues that have been plaguing SEO professionals for years. Because if Bing can get it right, there’s absolutely no excuse for Google to keep getting it wrong.

Why Bing Doesn’t Struggle with Canonical URLs and Duplicate Content, But Google Does!

How to Protect Small and Middle Businesses From Cyberattacks

In Kubernetes, Ingress resources are frequently used as traffic controllers, providing external access to services within the cluster. Ingress is essential for routing incoming traffic to your service; however, there may be scenarios in which you want to prevent search engines from indexing your service's content: it might be a development environment or something else. This blog post will walk you through the process of blocking your site's indexing on Kubernetes Ingress using a robots.txt file, preventing search engine bots from crawling and indexing your content. Prerequisites To proceed with the tutorial, you should have a basic grasp of Kubernetes basic objects, Ingress resources, and the official HAProxy ingress controller. You will also need access to the Kubernetes cluster and the necessary permissions to make configuration changes. Keep in mind that for this article, I assume that the HAProxy ingress controller is set as the default controller. Otherwise, if you did not select HAProxy as the default controller, you must add the ingressClassName option to all Ingress code examples. Step 1: Create an Ingress Kubernetes Resource In the first part of our journey, we'll set up a small Ingress resource to expose our service outside of the Kubernetes cluster. Pay attention: for the time being, all web crawlers will have access to the service. To apply the code below, use the command kubectl apply -f ingress.yaml. # file: ingress.yaml
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 name: hackernoon-close-site-indexing-haproxy-example
 annotations:
   cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
 rules:
   - host: hackernoon-close-site-indexing-haproxy-example.referrs.me
     http:
       paths:
         - path: /
           pathType: Prefix
           backend:
             service:
               name: hackernoon-close-site-indexing-haproxy-example-service
               port:
                 number: 80
 tls:
   - hosts:
       - hackernoon-close-site-indexing-haproxy-example.referrs.me
     secretName: hackernoon-close-site-indexing-haproxy-example Step 2: Modify the Ingress Configuration The robots.txt file is used to control how search engines index documents. Its file specifies which URLs search engine crawlers can access on your website. The most basic file that restricts access to the web service looks like this: User-agent: *
Disallow: / HAProxy does not require you to add this file to your web server or website. This can be achieved with the following configuration, which should be added to the backend section for the specific group of servers: acl robots_path path_beg /robots.txt
http-request return status 200 content-type "text/plain" lf-string "User-agent: *\nDisallow: /\n" if robots_path K8S Annotations regulate all manipulations of the HAProxy frontend/backend configuration for a single ingress resource. The full list of HAProxy annotations can be found in the official documentation on GitHub. In our case, we need to use haproxy.org/backend-config-snippet with the HAProxy snippet for blocking any indexing. To do this, edit, open your Ingress resource YAML file, and add the following annotation to the metadata section: #file: ingress.yaml
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 name: hackernoon-avoid-indexing-haproxy-example
 annotations:
   …
    haproxy.org/backend-config-snippet: |
      acl robots_path path_beg /robots.txt
      http-request return status 200 content-type "text/plain" lf-string "User-agent: *\nDisallow: /\n" if robots_path
spec:
  ... Step 3: Apply the Configuration Changes After changing the Ingress YAML file, save it and apply it to the Kubernetes cluster with the kubectl command: kubectl apply -f ingress.yaml. The Ingress controller will detect the changes and update the configuration accordingly. Step 4: Verify the Configuration Inspect the generated robots.txt file to confirm that indexing prevention is working properly. The Ingress controller generates this file based on the annotation you supply. Retrieve the external IP or domain associated with your Ingress resource and add /robots.txt to the URL. Example: $ curl hackernoon-close-site-indexing-haproxy-example.referrs.me/robots.txt
User-agent: *
Disallow: / As we can see, the answer contains a robots.txt file that prevents any search indexing. Step 5: Test Indexing Prevention To verify that search engines are not indexing your site, you can run a search for your website on popular search engines. Keep in mind that search results sometimes take some time to reflect changes, so the indexing status may not be fully updated right away. Conclusion Annotations make it easy to avoid search engine indexing while using HAProxy Kubernetes Ingress. By adding the appropriate annotation to your Ingress resource, you can prohibit search engine bots from crawling and indexing your website's content. A similar approach can be used with other ingress controllers, such as Nginx, Traefic, and others. A similar annotation can also be used for K8S Gateway API resources, which are actively replacing Ingresses. As a final note, robots.txt is a time-honored way for website creators to specify whether or not their sites should be crawled by various bots. However, it turns out that AI crawlers from large language model (LLM) companies frequently ignore the contents of robots.txt and crawl your site regardless. To avoid such situations, utilize password security, noindex, or enterprise load balancer features like HAProxy AI-crawler, which may be also configured as a K8S annotation. In Kubernetes, Ingress resources are frequently used as traffic controllers, providing external access to services within the cluster. Ingress is essential for routing incoming traffic to your service; however, there may be scenarios in which you want to prevent search engines from indexing your service's content: it might be a development environment or something else. This blog post will walk you through the process of blocking your site's indexing on Kubernetes Ingress using a robots.txt file, preventing search engine bots from crawling and indexing your content. Prerequisites To proceed with the tutorial, you should have a basic grasp of Kubernetes basic objects, Ingress resources, and the official HAProxy ingress controller . You will also need access to the Kubernetes cluster and the necessary permissions to make configuration changes. Kubernetes official HAProxy ingress controller Keep in mind that for this article, I assume that the HAProxy ingress controller is set as the default controller. Otherwise, if you did not select HAProxy as the default controller, you must add the ingressClassName option to all Ingress code examples. ingressClassName Step 1: Create an Ingress Kubernetes Resource In the first part of our journey, we'll set up a small Ingress resource to expose our service outside of the Kubernetes cluster. Pay attention: for the time being, all web crawlers will have access to the service. To apply the code below, use the command kubectl apply -f ingress.yaml . kubectl apply -f ingress.yaml # file: ingress.yaml
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 name: hackernoon-close-site-indexing-haproxy-example
 annotations:
   cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
 rules:
   - host: hackernoon-close-site-indexing-haproxy-example.referrs.me
     http:
       paths:
         - path: /
           pathType: Prefix
           backend:
             service:
               name: hackernoon-close-site-indexing-haproxy-example-service
               port:
                 number: 80
 tls:
   - hosts:
       - hackernoon-close-site-indexing-haproxy-example.referrs.me
     secretName: hackernoon-close-site-indexing-haproxy-example # file: ingress.yaml
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 name: hackernoon-close-site-indexing-haproxy-example
 annotations:
   cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
 rules:
   - host: hackernoon-close-site-indexing-haproxy-example.referrs.me
     http:
       paths:
         - path: /
           pathType: Prefix
           backend:
             service:
               name: hackernoon-close-site-indexing-haproxy-example-service
               port:
                 number: 80
 tls:
   - hosts:
       - hackernoon-close-site-indexing-haproxy-example.referrs.me
     secretName: hackernoon-close-site-indexing-haproxy-example Step 2: Modify the Ingress Configuration The robots.txt file is used to control how search engines index documents. Its file specifies which URLs search engine crawlers can access on your website. robots.txt The most basic file that restricts access to the web service looks like this: User-agent: *
Disallow: / User-agent: *
Disallow: / HAProxy does not require you to add this file to your web server or website. This can be achieved with the following configuration, which should be added to the backend section for the specific group of servers: acl robots_path path_beg /robots.txt
http-request return status 200 content-type "text/plain" lf-string "User-agent: *\nDisallow: /\n" if robots_path acl robots_path path_beg /robots.txt
http-request return status 200 content-type "text/plain" lf-string "User-agent: *\nDisallow: /\n" if robots_path K8S Annotations regulate all manipulations of the HAProxy frontend/backend configuration for a single ingress resource. The full list of HAProxy annotations can be found in the official documentation on GitHub . in the official documentation on GitHub In our case, we need to use haproxy.org/backend-config-snippet with the HAProxy snippet for blocking any indexing. To do this, edit, open your Ingress resource YAML file, and add the following annotation to the metadata section: haproxy.org/backend-config-snippet #file: ingress.yaml
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 name: hackernoon-avoid-indexing-haproxy-example
 annotations:
   …
    haproxy.org/backend-config-snippet: |
      acl robots_path path_beg /robots.txt
      http-request return status 200 content-type "text/plain" lf-string "User-agent: *\nDisallow: /\n" if robots_path
spec:
  ... #file: ingress.yaml
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 name: hackernoon-avoid-indexing-haproxy-example
 annotations:
   …
    haproxy.org/backend-config-snippet: |
      acl robots_path path_beg /robots.txt
      http-request return status 200 content-type "text/plain" lf-string "User-agent: *\nDisallow: /\n" if robots_path
spec:
  ... Step 3: Apply the Configuration Changes After changing the Ingress YAML file, save it and apply it to the Kubernetes cluster with the kubectl command: kubectl apply -f ingress.yaml . The Ingress controller will detect the changes and update the configuration accordingly. kubectl apply -f ingress.yaml Step 4: Verify the Configuration Inspect the generated robots.txt file to confirm that indexing prevention is working properly. The Ingress controller generates this file based on the annotation you supply. Retrieve the external IP or domain associated with your Ingress resource and add /robots.txt to the URL. Example: $ curl hackernoon-close-site-indexing-haproxy-example.referrs.me/robots.txt
User-agent: *
Disallow: / $ curl hackernoon-close-site-indexing-haproxy-example.referrs.me/robots.txt
User-agent: *
Disallow: / As we can see, the answer contains a robots.txt file that prevents any search indexing. robots.txt Step 5: Test Indexing Prevention To verify that search engines are not indexing your site, you can run a search for your website on popular search engines. Keep in mind that search results sometimes take some time to reflect changes, so the indexing status may not be fully updated right away. Conclusion Annotations make it easy to avoid search engine indexing while using HAProxy Kubernetes Ingress. By adding the appropriate annotation to your Ingress resource, you can prohibit search engine bots from crawling and indexing your website's content. A similar approach can be used with other ingress controllers, such as Nginx, Traefic, and others. A similar annotation can also be used for K8S Gateway API resources, which are actively replacing Ingresses. K8S Gateway API As a final note, robots.txt is a time-honored way for website creators to specify whether or not their sites should be crawled by various bots. However, it turns out that AI crawlers from large language model (LLM) companies frequently ignore the contents of robots.txt and crawl your site regardless. To avoid such situations, utilize password security, noindex, or enterprise load balancer features like HAProxy AI-crawler , which may be also configured as a K8S annotation. HAProxy AI-crawler

How to Block Search Engine Indexing in Kubernetes with HAProxy

I want to talk about a quick tweak you can do in your CDN TTL settings to radically improve your site’s performance. Direct impact on Time-To-First-Byte (TTFB) metric, but as a halo effect, pretty much every other Web Vital. You can do this in any CDN since TTL customization is a pretty standard need, and most CDN providers have easy ways to create rules for various rule configurations. I use Cloudflare for my blog’s CDN layer. Cloudflare already comes with nice defaults for optimizing the delivery of static assets like images, javascript, css files. But for HTML documents, CDNs use cache-control headers to determine how to cache, and how long to cache. Applications return this header, and it’s a way for the application (origin) to tell CDNs how to behave on certain pages. But in this optimization method, we’ll simply override all (or most) of our pages to be highly cached and served from the cache while revalidating in the background. The way this works is CDN always serves the “last” cached HTML to the reader (or crawler) from the edge network, really really really fast (in some cases double-digit milliseconds), and triggers a request to the origin server to get the “latest” version. Most applications also return proper response codes if the content hasn’t changed from the timestamp that CDN will ask if there is a new update to the content. How to Configure Custom TTL in Cloudflare To set up custom edge TTL in Cloudflare, navigate to your site page, Caching > Cache Rules page. Create a new rule, give it a name, and then set up the request path configuration. You can set multiple expressions, and exclude patterns that you know are Admin, or Rest API, or other URLs that should NOT be cached long. I use WordPress for my blog, and I exclude paths containing things like wp-admin, wp-json, cron… Then, select “Ignore cache-control header,” and use this TTL in the Edge TTL section. And finally, select how long you want to cache. Longer is better because longer means most of your site content, including long-tail content that doesn’t get consistent traffic, will also be cached at the edge. I started with 1 day, then 1 week, then I tried 1 month, but then had some pages stuck in the cache too long and dialed it back to 1 week as my sweet spot. Even if you’re not using Cloudflare, I’m sure there is an equivalent of this in your CDN provider. What Is the Impact on Page Speed? After the change, I saw a big drop (like 90% reduced load) in my server’s load. It meant CDN was doing what it was supposed to do. It’s one of the positive side effects of doing higher cache offload to CDN, to be able to scale higher traffic without needing powerful hosting resources. My Time-To-First-Byte decreased (improved) 70%, coming from shy of 500ms down to 100–160ms range 🤯 More importantly, the real user experience on the page became even more mind-blowing because things became super snappy. Click click click, bam bam bam, nothing was in a visible loading state anymore. Even if metrics didn’t move, I am super happy with this aspect of the change. 🤯🤯🤩 I got my Cloudflare Web Analytics email and noticed almost all Web Vitals moved positively at least 30% improvement. I wasn’t expecting other Web vitals like CLS and LCP to be directly impacted (or impacted as much as they did). But it makes sense. When the assets load much faster like this, the “wait time” (or blocking time) goes down, therefore, the layout shift or the largest paint goes down. SEO Impact It’s a well-known fact that Google takes your “core web vitals” into account when determining your ranking in the search results. This change has more impact on crawlers than you think. Because most of the time, crawlers’ requests are the ones that hit “cache cold” pages. It means Google (or other search engines) is reading your site holistically way more than your real users. Imagine every single article you wrote. There is no user who reads every single one of them — Google does 🙂 (and does it regularly). When a crawler tries to visit a page that nobody read in a long time, its request will have cache-miss more likely than cache-hit, so it will “wait” longer for your web server to render the page. When you put yourself in the crawler’s shoes, imagine you try to read 10,000 articles/pages on a site over a day or two period (maybe it takes longer, who knows…). Now, consider the percentage of those pages that will have to be rendered, or served from the CDN cache. The more pages Google sees “slow”, it will think your whole site is slow. This is where the real value of super-long TTLs comes in. Especially if you combine that with serve-stale-while-revalidate (SSWR) which most CDNs automatically do (if not, I’m sure there is a setting you can enable these together). SSWR, with a super-long TTL (like 7 days, or more), basically creates an infinite loop of “always cached” scenarios. And with that, your crawler traffic gets served from the cache (at cost/risk of “stale content” which is OK in the vast majority of use cases), and directly increases your site’s overall speed score and, therefore your SEO scores. Content Freshness There is one caveat though, which is content freshness. When you bump the Edge TTL up to multi-day TTLs like I did, you need to make sure your CMS/site is nicely integrated with your CDN’s cache clear systems, in the case you make updates. Two scenarios: You update existing content (like fixing a typo or changing the cover image of a post); the change should be reflected on the content’s detail page right away. You publish new content, so the new content is supposed to show up in common places like your homepage. You can use your CDN’s cache clear UI or APIs to trigger “purge” on URLs you think it’s impacted (homepage, section pages, etc.), or highly visible pages like the homepage can be configured with a lower TTL in a separate cache rule set. I use WordPress for my content management system and Cloudflare has a WordPress plugin to listen to publish/update hooks to trigger these cache clear nicely. Another way to think about this is to find the balance. What is the “staleness” you can tolerate on a page? Let’s say another article detail page showing “recent articles” or “related articles” sections to NOT show your most recent article there. As long as that time length is not something you can’t afford, cache longer to achieve better site/page performance. I want to talk about a quick tweak you can do in your CDN TTL settings to radically improve your site’s performance. Direct impact on Time-To-First-Byte (TTFB) metric, but as a halo effect, pretty much every other Web Vital. You can do this in any CDN since TTL customization is a pretty standard need, and most CDN providers have easy ways to create rules for various rule configurations. I use Cloudflare for my blog’s CDN layer. Cloudflare already comes with nice defaults for optimizing the delivery of static assets like images, javascript, css files. But for HTML documents, CDNs use cache-control headers to determine how to cache, and how long to cache. Applications return this header, and it’s a way for the application (origin) to tell CDNs how to behave on certain pages. But in this optimization method, we’ll simply override all (or most) of our pages to be highly cached and served from the cache while revalidating in the background. The way this works is CDN always serves the “last” cached HTML to the reader (or crawler) from the edge network, really really really fast (in some cases double-digit milliseconds), and triggers a request to the origin server to get the “latest” version. Most applications also return proper response codes if the content hasn’t changed from the timestamp that CDN will ask if there is a new update to the content. How to Configure Custom TTL in Cloudflare To set up custom edge TTL in Cloudflare, navigate to your site page, Caching > Cache Rules page. Create a new rule, give it a name, and then set up the request path configuration. You can set multiple expressions, and exclude patterns that you know are Admin, or Rest API, or other URLs that should NOT be cached long. I use WordPress for my blog, and I exclude paths containing things like wp-admin, wp-json, cron… Then, select “Ignore cache-control header,” and use this TTL in the Edge TTL section. And finally, select how long you want to cache. Longer is better because longer means most of your site content, including long-tail content that doesn’t get consistent traffic, will also be cached at the edge. I started with 1 day, then 1 week, then I tried 1 month, but then had some pages stuck in the cache too long and dialed it back to 1 week as my sweet spot. Even if you’re not using Cloudflare, I’m sure there is an equivalent of this in your CDN provider. What Is the Impact on Page Speed? After the change, I saw a big drop (like 90% reduced load) in my server’s load. It meant CDN was doing what it was supposed to do. It’s one of the positive side effects of doing higher cache offload to CDN, to be able to scale higher traffic without needing powerful hosting resources. My Time-To-First-Byte decreased (improved) 70%, coming from shy of 500ms down to 100–160ms range 🤯 More importantly, the real user experience on the page became even more mind-blowing because things became super snappy. Click click click, bam bam bam, nothing was in a visible loading state anymore. Even if metrics didn’t move, I am super happy with this aspect of the change. 🤯🤯🤩 I got my Cloudflare Web Analytics email and noticed almost all Web Vitals moved positively at least 30% improvement. I wasn’t expecting other Web vitals like CLS and LCP to be directly impacted (or impacted as much as they did). But it makes sense. When the assets load much faster like this, the “wait time” (or blocking time) goes down, therefore, the layout shift or the largest paint goes down. SEO Impact It’s a well-known fact that Google takes your “core web vitals” into account when determining your ranking in the search results. This change has more impact on crawlers than you think. Because most of the time, crawlers’ requests are the ones that hit “cache cold” pages. It means Google (or other search engines) is reading your site holistically way more than your real users. Imagine every single article you wrote. There is no user who reads every single one of them — Google does 🙂 (and does it regularly). When a crawler tries to visit a page that nobody read in a long time, its request will have cache-miss more likely than cache-hit, so it will “wait” longer for your web server to render the page. When you put yourself in the crawler’s shoes, imagine you try to read 10,000 articles/pages on a site over a day or two period (maybe it takes longer, who knows…). Now, consider the percentage of those pages that will have to be rendered, or served from the CDN cache. The more pages Google sees “slow”, it will think your whole site is slow. This is where the real value of super-long TTLs comes in. Especially if you combine that with serve-stale-while-revalidate (SSWR) which most CDNs automatically do (if not, I’m sure there is a setting you can enable these together). SSWR, with a super-long TTL (like 7 days, or more), basically creates an infinite loop of “always cached” scenarios. And with that, your crawler traffic gets served from the cache (at cost/risk of “stale content” which is OK in the vast majority of use cases), and directly increases your site’s overall speed score and, therefore your SEO scores. Content Freshness There is one caveat though, which is content freshness. When you bump the Edge TTL up to multi-day TTLs like I did, you need to make sure your CMS/site is nicely integrated with your CDN’s cache clear systems, in the case you make updates. Two scenarios: You update existing content (like fixing a typo or changing the cover image of a post); the change should be reflected on the content’s detail page right away. You update existing content (like fixing a typo or changing the cover image of a post); the change should be reflected on the content’s detail page right away. You publish new content, so the new content is supposed to show up in common places like your homepage. You publish new content, so the new content is supposed to show up in common places like your homepage. You can use your CDN’s cache clear UI or APIs to trigger “purge” on URLs you think it’s impacted (homepage, section pages, etc.), or highly visible pages like the homepage can be configured with a lower TTL in a separate cache rule set. I use WordPress for my content management system and Cloudflare has a WordPress plugin to listen to publish/update hooks to trigger these cache clear nicely. Another way to think about this is to find the balance. What is the “staleness” you can tolerate on a page? Let’s say another article detail page showing “recent articles” or “related articles” sections to NOT show your most recent article there. As long as that time length is not something you can’t afford, cache longer to achieve better site/page performance.

How to Tweak CDN/Edge TTL to Radically Improve Site Performance (and SEO): A Quick Guide

Navigating the LLM Landscape: A Comparative Analysis of Open-Source Models

Stories, News & Mentions about CRAWLQ

Crawlq’s Stories on HackerNoon