CloudFlare is building a faster, safer web from the ground up

Cloudflare

cloudflare

Your web scraper has just been blocked by Cloudflare again! You've looked online for solutions, but nothing seems to do the trick. Is there a secret sauce for Cloudflare bypass? You bet there is. You just haven't stumbled upon the right recipe! Embark on a guided tour through the maze of Cloudflare anti-bot technology. We're here to demystify Cloudflare's inner workings, unveil some game-changing tricks, and ensure you never hit a dead end again! What Is Cloudflare? , the cyber fortress of the Internet, offers top-notch services to boost the security and performance of websites and web applications. But you probably knew that already! Cloudflare Now, what's the secret sauce that makes Cloudflare so popular? Drumroll, please—it’s anti-bot solutions! Ever come across that page? Well, you’re the only one! That’s a result of the , a high-tech shield against the mischief of malicious bots. Cloudflare Bot Management service Here are the technologies that power that cybersecurity platform: Behavioral analysis: Picture it as Sherlock Holmes scrutinizing user behavior on the Web to decipher whether users are humans or not. Machine learning magic: Think of it as the Dumbledore's Army of bot detection, learning from a colossal dataset with billions of records to give every request a reliable bot score. Fingerprinting: Use millions of internet properties to precisely identify bot requests without compromising user privacy. The result? Bots, begone! From stopping credential stuffing to thwarting content scrapers, spammers, inventory hoarders, credit card stuffers, and even putting up a solid defense against application DDoS attacks—Cloudflare's got it covered. Cloudflare’s Bot Management stands tall as the Batman of bot management, vigilantly safeguarding websites to thwart any hint of malicious bot behavior. 🦸‍♂️ But every superhero has its kryptonite! Dive into the next section to unravel more. Tips to Bypass Cloudflare Before digging into this chapter, let's tackle a burning question. Is it even possible to bypass Cloudflare? Absolutely! No online solutions can or should restrict users from accessing public sites as they wish, even through automated software. Thus, the million-dollar question is: how to outsmart Cloudflare? 🤔 Dive into the tips below to master the art of eluding Cloudflare! Use a Reliable IP Cloudflare gives IPs a score based on their reputation. If your IP has been caught red-handed performing actions considered malicious by Cloudflare, the chances of getting blocked become high. The solution? Having a large pool of IPs to distribute your requests on. In that scenario, even if one IP gets blocked, you have many others available in your arsenal. That’s where IP rotation via proxy comes into play. Find out more on . how to rotate an IP address Leverage Browser Automation Only browsers can overcome the challenges set by Cloudflare’s Bot Management. When you navigate a page shielded by that solution, your browser uses JavaScript to tackle those challenges without you even realizing it. Requests made by an automated script through a traditional HTTP client will then hit an immediate roadblock. Why? Because HTTP clients can’t execute JavaScript! That’s something only a browser can do. Thus, your Cloudflare-ready automated script should rely on an approach for programmatically executing actions on a web page in a browser, such as . controlling a headless browser via Playwright Reverse Engineering the Cloudflare Challenges This is where the real challenge kicks in—how to outsmart the anti-bot defenses intricately embedded into a web page by Cloudflare? The answer: dive deep into their workings, unravel the puzzle through reverse engineering, and conquer. That's easier said than done 😅 Step one involves finding the request linked to the retrieval of the JavaScript challenge. This will involve encrypted code that you must somehow decipher: Suppose you managed to do that. Now, all that remains is to figure out the original code's intentions and create a strategy to fool it into thinking that the Cloudflare challenge has been overcome. On paper, this sounds like a valid approach, but in the real world, it may be a one-way ticket to a colossal headache. Don’t worry, though. Overcoming Cloudflare is possible—you just need the right tools! My Solution Works… Until It Doesn't Imagine you're an IT genius who manages to crack the code to get past Cloudflare's bot protection technologies. Bear in mind that Cloudflare is an ever-evolving beast, so no guarantees your solution will work tomorrow. To reach the goal, you've got to embrace a different approach! A real-world, ideal, intelligent solution to overcome Cloudflare would be a controllable browser that: Can overcome advanced anti-bot measures for you. Is always up-to-date because it runs in the cloud. Automatically rotates IPs and handles request attempts. Avoids fingerprinting to improve your privacy. Can resolve CAPTCHAs on its own. Allows cookie management and header customization. Ensures amazing scalability and performance by executing in the cloud. It can be controlled by any browser automation library, such as Selenium, Playwright, or Puppeteer. This isn't a mere dream, but a palpable reality. The browser you're imagining is real, developed by Bright Data, and goes by the name of ! Scraping Browser Final Thoughts Cloudflare reigns supreme in safeguarding websites against malicious bots. Here, we've delved into the technologies it uses for detecting and stopping automated software, offering you some handy tips to avoid them. But let's be real, that’s a grueling, trial-and-error, demoralizing task. Need to bypass Cloudflare once and for all? Embrace Scraping Browser from and join our mission to transform the Internet into a public space accessible to everyone, everywhere, even via automated scripts! Bright Data Until next time, keep exploring the Internet with freedom!

Cracking the Code of Cloudflare Bypass

ক্লাউডফ্লেয়ার বাইপাসের কোড ক্র্যাক করা

Den Code von Cloudflare Bypass knacken

Descifrando el código de omisión de Cloudflare

Déchiffrer le code du contournement de Cloudflare

क्लाउडफ्लेयर बाईपास के कोड को क्रैक करना

Cloudflareバイパスのコードを解読する

Cloudflare Bypass 코드 크래킹

Quebrando o código do Cloudflare Bypass

Взлом кода обхода Cloudflare

Cloudflare Bypass Kodunu Kırmak

Bẻ khóa mã bỏ qua Cloudflare

破解 Cloudflare 绕过代码

I can't go a day (or two) without building a bot of some sort, and last week was no exception. I've been a fan of the for nearly a decade now, and one of my favorite examples of it is my bot. I thought I'd use the Marvel API as a way to build bot, but this time on the platform. Here's how I did it. Marvel API random comic book cover another Cloudflare Workers The Architecture So, obviously, I'm using Cloudflare Workers, but I decided to make this project a two-step process. In my , I shared how you can "connect" one Worker to another via service bindings. Obviously, this is practical for reuse, and while it may be a bit overkill for this, I wanted to put what I learned in the last post into practice. My project's two workers are: last post A scheduled Worker that makes use of their support. This worker will be run on a schedule, call the second Worker for its data, and then post to Mastodon. Cron triggers A Worker that wraps calls to the Marvel API, specifically one to get a random character. The latter of the two is the simpler one, so let's start with that. Using the Marvel API for Character Information So, much like my 'random comic book' logic, I get a random Marvel character by: Figure out the total number of characters. I did this using their interactive tester and making note of the total number of results. As of August 2023, that number is 1562. Here's an example of that output: Then, simply call the character endpoint with a limit of one and an offset of a random number in that range. For the most part, it's all relatively simple, except Marvel requires you to sign your API requests. This was a minor sticking point for me as my previous Node.js code didn't work on Cloudflare, and I had to switch to Web Crypto, which was nicely documented here: . https://developers.cloudflare.com/workers/runtime-apis/web-crypto/#web-crypto Here's the entirety of the Worker: // Based on checking the API (in Aug 2023) to see how the max number of characters
const CHAR_TOTAL = 1562;

const getRandomInt = (min, max) => {
    return Math.floor(Math.random() * (max - min + 1)) + min;
}

async function getSuperHero(privateKey, publicKey) {

    let selected = getRandomInt(0, CHAR_TOTAL);
    let url = `https://gateway.marvel.com:443/v1/public/characters?limit=1&apikey=${publicKey}&offset=${selected}`;

    // add hash
    let ts = new Date().getTime();
    let myText = new TextEncoder().encode(ts + privateKey + publicKey);

    let hash = await crypto.subtle.digest({
        name:'MD5'
    }, myText);

    // Credit: https://developers.cloudflare.com/workers/runtime-apis/web-crypto/
    const hexString = [...new Uint8Array(hash)]
      .map(b => b.toString(16).padStart(2, '0'))
      .join('');

    url += '&hash='+encodeURIComponent(hexString)+'&ts='+ts;

    let resp = await fetch(url);
    let data = await resp.json();
    return data.data.results[0];
}

export default {
    async fetch(request, env, ctx) {
        const PRIVATE_KEY = env.MARVEL_PRIVATE_KEY;
        const PUBLIC_KEY = env.MARVEL_PUBLIC_KEY;
        let hero = await getSuperHero(PRIVATE_KEY, PUBLIC_KEY);
        console.log(`I got the hero ${hero.name}`);

        return new Response(JSON.stringify(hero), {
            headers: {
                'Content-Type':'application/json;charset=UTF-8'
            }
        });
    },
}; By the way, notice the ? Later this week, I'll show how that works in production. In case you're curious, here's what that JSON response looks like. It randomly selected one of my favorite characters, Galactus. In order to save on space, I removed items from the arrays of data. console.log many {
    "id": 1009312,
    "name": "Galactus",
    "description": "",
    "modified": "2014-09-30T16:47:03-0400",
    "thumbnail": {
        "path": "http://i.annihil.us/u/prod/marvel/i/mg/5/03/528d31a791308",
        "extension": "jpg"
    },
    "resourceURI": "http://gateway.marvel.com/v1/public/characters/1009312",
    "comics": {
        "available": 236,
        "collectionURI": "http://gateway.marvel.com/v1/public/characters/1009312/comics",
        "items": [
            {
                "resourceURI": "http://gateway.marvel.com/v1/public/comics/12638",
                "name": "Alpha Flight (1983) #10"
            },
            {
                "resourceURI": "http://gateway.marvel.com/v1/public/comics/12639",
                "name": "Alpha Flight (1983) #100"
            },
            {
                "resourceURI": "http://gateway.marvel.com/v1/public/comics/4788",
                "name": "Annihilation (2006) #1"
            },

        ],
        "returned": 20
    },
    "series": {
        "available": 112,
        "collectionURI": "http://gateway.marvel.com/v1/public/characters/1009312/series",
        "items": [
            {
                "resourceURI": "http://gateway.marvel.com/v1/public/series/2116",
                "name": "Alpha Flight (1983 - 1994)"
            },
            {
                "resourceURI": "http://gateway.marvel.com/v1/public/series/3613",
                "name": "Annihilation (2006 - 2007)"
            },
            {
                "resourceURI": "http://gateway.marvel.com/v1/public/series/1864",
                "name": "Annihilation: Heralds of Galactus (2007)"
            }
        ],
        "returned": 20
    },
    "stories": {
        "available": 259,
        "collectionURI": "http://gateway.marvel.com/v1/public/characters/1009312/stories",
        "items": [
            {
                "resourceURI": "http://gateway.marvel.com/v1/public/stories/694",
                "name": "Cover #694",
                "type": "cover"
            },
            {
                "resourceURI": "http://gateway.marvel.com/v1/public/stories/898",
                "name": "Fantastic Four (1998) #520",
                "type": "cover"
            },
            {
                "resourceURI": "http://gateway.marvel.com/v1/public/stories/899",
                "name": "1 of 5 - Galactus",
                "type": "interiorStory"
            },
            {
                "resourceURI": "http://gateway.marvel.com/v1/public/stories/922",
                "name": "Fantastic Four (1998) #518",
                "type": "cover"
            },
            {
                "resourceURI": "http://gateway.marvel.com/v1/public/stories/923",
                "name": "AVENGERS DISASSEMBLED TIE-IN! \"FOURTITUDE\" PART 2 (OF 3) With public opinion of the FF at an all-time low and with all of Manhat",
                "type": "interiorStory"
            }
        ],
        "returned": 20
    },
    "events": {
        "available": 9,
        "collectionURI": "http://gateway.marvel.com/v1/public/characters/1009312/events",
        "items": [
            {
                "resourceURI": "http://gateway.marvel.com/v1/public/events/229",
                "name": "Annihilation"
            },
            {
                "resourceURI": "http://gateway.marvel.com/v1/public/events/234",
                "name": "Avengers Disassembled"
            },
            {
                "resourceURI": "http://gateway.marvel.com/v1/public/events/318",
                "name": "Dark Reign"
            },
            {
                "resourceURI": "http://gateway.marvel.com/v1/public/events/302",
                "name": "Fear Itself"
            }
        ],
        "returned": 9
    },
    "urls": [
        {
            "type": "detail",
            "url": "http://marvel.com/comics/characters/1009312/galactus?utm_campaign=apiRef&utm_source=fe877c0bf61f995fc8540d9eac4704f1"
        },
        {
            "type": "wiki",
            "url": "http://marvel.com/universe/Galactus?utm_campaign=apiRef&utm_source=fe877c0bf61f995fc8540d9eac4704f1"
        },
        {
            "type": "comiclink",
            "url": "http://marvel.com/comics/characters/1009312/galactus?utm_campaign=apiRef&utm_source=fe877c0bf61f995fc8540d9eac4704f1"
        }
    ]
} Notice the property is both a path and an extension. Here's our lovely world devourer. thumbnail Now, on to the next Worker. The Scheduled Tooter I love that subhead. So the next Worker is responsible for running on a schedule and actually doing the Mastodon posting. The cover how these are set up and how you can test them. For the most part, this is just a plain worker. The biggest difference in the code is that you have a handler, not . Here's a barebones Worker for scheduled execution. Cloudflare docs scheduled fetch export default {
    async scheduled(event, env, ctx) {


    },
}; And your schedule is defined in the file. In this case, every two hours: wrangler.toml [triggers]
crons = ["0 */2 * * *"] The first issue I ran into was how to connect this worker to the first one. In the , you'll see it's rather simple: last blog post const backResponse = await env.backlogic.fetch(request.clone()); But, this expects an incoming request, an HTTP-driven Worker. There isn't a request in a Cron-triggered worker. I on the Cloudflare forums and got help from the most Internet nickname ever, . The fix is to simply make an empty (mostly) Request object like so: fetch asked Cyb3r-Jak3 let heroRequest = await env.randomsuperhero.fetch(new Request('http://127.0.0.1'));
let hero = await heroRequest.json(); Next, I prepare my data for my toot: /*
Generate the text for the toot.
I'm using the 'detail' link which is not always the best, better than the wiki though :( 
*/
let toot = `
Your random Marvel superhero of the moment is: ${hero.name}.
More information here: ${hero.urls[0].url}
`;

let image = `${hero.thumbnail.path}.${hero.thumbnail.extension}`; This is where I should point out that while my random comic book cover bot is , the character information is - unfortunately - a bit slim, especially for obscure characters. To be clear, I don't mean a lack of information, but a lot of 404s and no images. Honestly, I almost punted on this as a source of data, but I figured I'd let it go and see how it looks after a while "in the wild." great Now, at this point, I've got the text for my toot, as well as the image URL. I had intended to make use of the npm module I'd used in the past, , however, when I included this in my Worker, I got compatibility errors with the Workers environment. It didn't seem like an easy workaround, and I almost gave up when I thought, why not actually look at their and try using it without a wrapper? mastodon-api API documentation Creating a "toot" was incredibly simple: let data = new FormData();
data.append('status', toot);

let resp = await fetch('https://botsin.space/api/v1/statuses', {
    body:data,
    method:'post',
    headers:{
        'Authorization':`Bearer ${KEY}`
    }
}); Literally ten lines or so of code. I was pleasantly surprised. I then looked into the image aspect. Like Twitter, if you want to associate an image with a toot, you first upload the image, get the ID, and then associate it with the new message. To do this, I needed to first get the bits from the URL on Marvel's side and then send that to Mastodon. In the past, I would have simply saved the image to , but I don't think Cloudflare supports that. Instead, I did everything in memory. This took me the longest time, so hopefully, this code can help others. /tmp async function uploadMedia(url, key) {
    // first, grab the bits of the url
    let imgreq = await fetch(url);
    let blob = new Blob([await imgreq.blob()]);

    let data = new FormData();
    data.append('file', blob);

    let mediaupload = await fetch('https://botsin.space/api/v2/media', {
        body:data,
        method:'post',
        headers:{
            'Authorization':`Bearer ${key}`
        }
    });

    return await mediaupload.json();

} Woot. Okay, with that in play, here's the updated toot code (I love saying toot): let toot = `
Your random Marvel superhero of the moment is: ${hero.name}.
More information here: ${hero.urls[0].url}
`;

let image = `${hero.thumbnail.path}.${hero.thumbnail.extension}`;
let mediaOb = await uploadMedia(image, KEY);

let data = new FormData();
data.append('status', toot);
data.append('media_ids[]', mediaOb.id);

let resp = await fetch('https://botsin.space/api/v1/statuses', {
    body:data,
    method:'post',
    headers:{
        'Authorization':`Bearer ${KEY}`
    }
}); And voila - magic: Random Super Hero @myrandomsuperhero@botsin.space Your random Marvel superhero of the moment is: Alex Power.More information here: http://marvel.com/characters/1387/alex_power?utm_campaign=apiRef&utm_source=fe877c0bf61f995fc8540d9eac4704f1 (UTC) 12:01 PM • August 13, 2023 This is Alex Power, part of Power Pack, a favorite of mine when I was a teenager, and unfortunately, the detail links to a 404. :( All in all, this is kind of par for the course for Cloudflare Workers. I hit a few snags, learn how Cloudflare does things, and then it all just works. I know I said this before, but I'm absolutely digging into the Cloudflare developer experience. If you want to follow the bot, you can find the account at . The source code for generating the random hero may be found here: . The source for the cron/Mastodon Worker may be found here: https://botsin.space/@myrandomsuperhero https://github.com/cfjedimaster/cloudflareworkers-demos/tree/main/randomsuperhero https://github.com/cfjedimaster/cloudflareworkers-demos/tree/main/randomsuperherobot Also published . here

Mastodon: Building a Marvel Character Info Bot with Cloudflare Workers

This article describes how to deploy a static site. We would use (IaC) approach to describe site changes and deploy them automatically with Cloudflare Pages. Infrastructure as Code If you are not familiar with one of the tools, don't worry, all steps are described in the details. You can use this article as a guide for your own static site for 10 minutes and 100% free. Table of Content Description Pre-requirements Tips Quick Start Useful links Conclusion For a quick start, you can jump to the “Quick Start” section. Description The site will be hosted on , and the domain name will be managed by you. The site will be built using the Hugo framework. The whole process will be automated using Cloudflare pages. We will use GitHub as a source of truth for our site. Cloudflare Pages So we describe an infrastructure as code and deploy it automatically; it's a little different from the site constructor approach, there is no user interface (UI) for site management, but it's more flexible and powerful. Just follow these , and you will have your own personal site for free. 5 steps Initialize your site Choose a theme for your site Commit the changes to your repository on GitHub Assign your site to your project on Cloudflare Pages Enjoy your site To update your site, you just need to commit changes to your repository on GitHub. Then Cloudflare Pages will automatically rebuild your site. Pre-requirements GitHub account. If you don't have one, you can create it , it's free. here Cloudflare account. If you don't have one, you can create it , it's free. here Domain name. This step is . optional If you don't have a domain name, you can use a free subdomain from Cloudflare. It's up to you. Your free subdomain will look like this: . yourname.pages.dev TIPS TIP 1 If you would like to make a similar page to my , just grab my repo in 3 steps: kazakov.xyz Fork my repo kksudo/kazakov.xyz Update content on your new site. Please, see the repo structure. You need to update 2 files: - Edit the links to your social networks or remove unused ones. For the main pic, I use , but you can use this setting . To read more about Hugo config.toml, please read . ./config.toml Gravatar avatarurl = "images/avatar.jpg" here - Add a few words about yourself. about.md Follow to step “#4 Cloudflare Pages” TIP 2 To verify your changes, run and open docker run --rm -it -v $(pwd):/src -w /src -p 1313:1313 klakegg/hugo:alpine server -D http://localhost:1313/ TIP 3 You can add a new page by . docker run --rm -it -v $(pwd):/src -w /src klakegg/hugo:alpine new <SECTIONNAME>/<FILENAME>.<FORMAT> TIP 4 To exclude unnecessary dynamic files, update your .gitignore Quick Start Step 1. Initialize Your Site For our site, we will use the Hugo framework. We will use some steps from the to initialize our site. So, to start it, we need to a Hugo binary on your machine, but I prefer a docker to keep my OS clean. Hugo quick start guide install Let's create a root structure for your site : hugo new site business-card docker run --rm -it -v $(pwd):/src -w /src klakegg/hugo:alpine new site business-card Docker options: - remove the container after exit --rm - interactive mode -it - mount the current directory to directory in the container -v $(pwd):/src /src - set the working directory to in the container -w /src /src - latest docker image with Hugo, based on the Alpine Linux klakegg/hugo:alpine Tip for a lazy person like me: smart Check this step in my repository kksudo/kazakov.xyz Step 2.  Chose a Theme for Your Site To find a theme for our site, we will use an . official Hugo hub There are a lot of themes; you can choose any of them. But my favorite option is to discover it via GitHub. Because you can filter repositories by stars, forks, maintaining, etc. For example, is a simple filter. here So, let's start with the theme with more than starts on GitHub. Coder 2k+ Move to the root directory of your site. cd business-card Initialize an empty Git repository in the current directory. Skip it if your site already has a Git repository. git init -b main # Initialize the local directory as a Git repository. Clone the Coder theme into the directory, and add it to your project as a Git submodule. themes git submodule add https://github.com/luizdepra/hugo-coder themes/hugo-coder Update the site configuration file , indicating to use theme. Also, we should add some settings for this theme. Grab these settings from my file. ./config.toml hugo-coder config.toml You can find more information about the theme settings . here Note: If you would like to use a simple theme, without any settings, you can use the theme. It's a default theme for Hugo, URL. Ananke quick start Tip for a lazy person like me: smart You could grab this step from my repository kksudo/kazakov.xyz To verify changes, run and open docker run --rm -it -v $(pwd):/src -w /src -p 1313:1313 klakegg/hugo:alpine server -D http://localhost:1313/ Step 3. Commit the Changes to Your GitHub Repository Let's add some content, and change the default site settings before the first commit. Add an About page docker run --rm -it -v $(pwd):/src -w /src klakegg/hugo:alpine new about.md Add some markdown to the body of the post, but do not change the draft value, code content/about.md ---
title: "About"
date: 2023-04-11T21:14:37Z
draft: true
---

I'm a member of "I want to know everything" and "I'm happy to share what I've already learned" groups.
DevOps with love. Update as described in the theme page. Let's add some menu settings for the About page. Or grab these settings from my file, and then edit. ./config.toml hugo-coder config.toml Push our changes to GitHub origin. git add -A . # Adds the files in the local repository and stages them for commit
git commit -m "First commit" # commit all the files in your project

git remote add origin git@github.com:<owner>/<repo-name>.git
git push -u origin main Note: Read more about GitHub quick start here Tip for a lazy person like me: grab this step from my repository smart kksudo/kazakov.xyz Step 4.  Assign Your Site to Your Project on Cloudflare Pages Create an account on , and log in to your account. Cloudflare Create a new project on , and connect to your GitHub repository. Cloudflare Pages Connect Cloudflare to your GitHub repository. Install Cloudflare Pages. Grand permissions Cloudflare integration in GitHub. Setting up Cloudflare settings for the Hugo site Chose a Hugo framework Add a variable to use on the latest Hugo version, HUGO_VERSION=0.111.3 : If you would like to revert Cloudflare permissions for your repo, you should go to and change settings for Cloudflare Pages. TIP https://github.com/settings/installations So, after Cloudflare will be assigned, deployment will be triggered. And after it, Cloudflare started deployment of your site. By default, Cloudflare will build and deploy all your branches and generate a unique URL for each branch; the main URL will be also available for your default branch. So, Cloudflare shows you a lot of details about your build and deployment process. Step 5.  Enjoy Your Site Open and enjoy :) https://yourname.pages.dev Tip for a lazy person like me: smart Sorry, no tips :) Useful Links Install docker Install Hugo Join to GitHub Deploy a Hugo site CloudFlare Hugo quickstart Coder theme configuration documentation based on this article My page with code for this article GitHub repository Disclaimer, Why Did I Choose This Set of Tools? Why Hugo? Hugo it's the world’s fastest framework for building websites with more than 66 000 stars on GitHub. It's very easy to use, pretty fast, open-source, and has a lot of features. You can find more information about Hugo on the . official website Why GitHub? GitHub is a code hosting platform for version control and collaboration. It lets you and others work together on projects from anywhere. You can find more information about GitHub on the . official website Why Cloudflare? Firstly, I will explain why, I didn't choose . The last time I tried to use GitHub Pages, I faced some incipient problems. Also, I chose Cloudflare Pages because this provider is network focused. GitHub Pages I know Cloudflare as a network infrastructure and security company. I really like their products, like DNS, CDN, Firewall, etc. But I didn't know that they have a product for hosting static sites. I was surprised when I found out about it. So, I decided to use network focused provider for hosting static sites. Conclusion I hope this article was useful for you. If you have any questions, feel free to ask me in the comments. I will be happy to help you. Also, feel free to connect with me . kazakov.xyz P.S.: For this flow, we could use a lot of other approaches, services, and tools; in the section above, I described why I sue these techs. For example, you can use GitHub pages instead of Cloudflare Pages, and use another static site generator instead of Hugo. #contest-tags

Deploy Your Personal Web-Page With Hugo, Cloudflare and GitHub 100% For Free

Introduction You want to shorten the long links generated by your application because the URLs are too long to fit in an SMS. You want to keep scalability in mind and want to handle millions of shortened URLs and millions of users flowing through the links. Most importantly, you want this new service to be as cheap as possible - it can't break the bank. In this article, we will use Cloudflare Workers and KV to create a scalable URL shortener service that you can use to shorten your application links. The Stack is a Serverless platform that allows you to deploy code and
have it available globally with extremely low latency. Cloudflare Workers... Cloudflare Workers is much cheaper than competitors such as Azure Functions and AWS Lambda. ($5/10 million requests) can run code instantly without suffering from cold starts like the competitors. can automatically scale to millions of users - you are only charged per request and not the number of concurrent users. The reason why Cloudflare can do this is that it doesn't rely on the traditional serverless model of using the virtual machine architecture. Instead, it utilizes the Chrome V8 engine with the Service Workers API to run code extremely fast. If you are a nerd like me - will amaze you. reading this Alternatively, you would probably write a simple Express or Flask app, dockerize it, and deploy it through or Kubernetes. docker-compose is a key-value store that is globally distributed. We're going to use it to map from a short link to the original long URL. A URL shortener service is one of the best cases to utilize serverless functions. It does 1 simple job, so you shouldn't waste your time thinking about infrastructure, and most importantly, an expensive bill to pay at the end of the month. Workers KV Alternatively, this might be a mix of a PostgreSQL server for permanent storage and Redis to cache results. The Preparation Sign up for a . Cloudflare Account Install Cloudflare's CLI tool, . Wrangler npm install wrangler --save-dev Authenticate yourself with Wrangler by following instructions on the screen to complete OAuth with Cloudflare. npx wrangler login The Plan We want to expose 2 endpoints for the user - one to create a shortened URL, and the other to serve all shortened URLs, perform 302 redirects if the shortened link is found, and 404 otherwise. The code Step 1: Generate a boilerplate for Cloudflare Workers In a local directory, run . It's going to create the necessary files and folder structure for you to create our URL shortener. You will be prompted to select a few options, to which you will select yes to. npx wrangler init shortener Would you like to use git to manage this Worker? y No package.json found. Would you like to create one? y Would you like to use TypeScript? y Would you like to create a Worker at shortener/src/index.ts? Fetch Handler To further simplify development, since we are going to make more than 1 endpoint, we'll
install which is an Express-like router to automatically parse out the HTTP method, query params, and path params, etc. I also prefer to use to format the code so that it looks consistent. itty-router prettier cd shortener
rm package-lock.json
yarn
yarn add itty-router
yarn add --dev prettier Step 2: Create a KV namespace KV is basically the replacement for the database and cache you were going to use to make your Express URL shortener. The difference is, KV is as easy to use as the browser's localStorage. To create a KV namespace on your Cloudflare account, run the following commands. We'll need to run both because we'll need to reserve one for development as well. npx wrangler kv:namespace create SHORTENER_KV
npx wrangler kv:namespace create SHORTENER_KV --preview Note down the KV ids and edit to look like the following. When you deploy with Wrangler, it will set the correct variables and link the KV namespace to your worker. wrangler.toml name = "shortener"
main = "src/index.ts"
compatibility_date = "2022-08-11"

kv_namespaces = [
  {
    binding = "SHORTENER_KV",
    id = "<Your KV ID>",
    preview_id = "<Your KV preview ID for development>"
  }
]

[vars]
HOST_URL = "http://localhost:8787" Step 3: Make the API to shorten a URL I like to write out the skeleton of what I'm going to implement before actually implementing the logic behind it, as I feel like it will lead to better code structure. In the below snippet, I am creating an that will triage API requests to the POST and GET functions respectively. itty-router Also, I am linking the KV namespace in the environment. Finally, from my experience, it is ideal to add statements here and there to help with debugging in case my Worker goes wrong. console.log import { Router } from "itty-router";
const router = Router();

export interface Env {
  SHORTENER_NAMESPACE: KVNamespace; // the KV namespace we setup in the previous step
  HOST_URL: string;
}

router.post("/", async (request: Request, env: Env) => {
  // TODO: Post request to shorten a URL
});

router.get("/:slug", async (request: Request, env: Env) => {
  // TODO: redirect shortened URL to application URL
});

export default {
  async fetch(
    request: Request,
    env: Env,
    ctx: ExecutionContext
  ): Promise<Response> {
    return router.handle(request, env, ctx).then((res: Response) => {
      console.log("HTTP Response", request.url, res.status);
      return res;
    });
  },
}; Now, we can fill in the logic to create a shortened URL. In the below code snippet, I am generating a random slug that has not been used before and using that as the key to store the long URL that the user has passed in. Finally, I am returning the user a shortened URL that they may use to redirect to their URL. router.post("/", async (request: Request, env: Env) => {
  const json = await request.json();
  const longLink = json?.url;
  // generate a random slug - this is not cryptographically random but works in our case
  let slug = btoa(Math.random() + "").slice(0, 9);
  // generate a new slug if the previous one is already existing
  let existing = await env.SHORTENER_KV.get(slug);
  while (existing) {
    slug = btoa(Math.random() + "").slice(0, 9);
    existing = await env.SHORTENER_KV.get(slug);
  }
  // generate shortened URL and return to user
  await env.SHORTENER_KV.put(slug, longLink);
  return new Response(JSON.stringify({ url: `${env.HOST_URL}/${slug}` }), {
    headers: { "content-type": "application/json" },
  });
}); At this point, we should test our code. To run a dev server, run . Then, you can use Postman to test out the server for now. npm run start Step 4: Handle the URL redirect When the user visits the shortened URL, we would like to fetch the long URL from KV, and provide the user with a 302 redirect response, so that the browser can handle the redirection for us. If the shortened slug cannot be found, we can return a 404 telling them nothing is here. router.get("/:slug", async (request: Request, env: Env) => {
  const slug = request.params.slug;
  const redirectTo = await env.SHORTENER_KV.get(slug);
  if (redirectTo) {
    return Response.redirect(redirectTo, 302);
  }
  return new Response("URL not found", {
    status: 404,
  });
}); Hopefully by now, 8 minutes in, you have a fully working URL shortener that is deployed locally. We'll leave the last 2 minutes for deployment, which is more than enough. The deployment Simply run and Wrangler will handle the rest. By default, Cloudflare Workers will provide you a subdomain but you can also configure your own by going into the Cloudflare Dashboard under Workers. npm run deploy workers.dev - KV Namespaces:
  - SHORTENER_KV: <your KV ID>
- Vars:
  - HOST_URL: "http://localhost:8787"

Published shortener (3.83 sec)
  https://shortener.cho.workers.dev As you can see, the used is still the one we defined at the beginning, using localhost:8787. Remember to change the URL to the published URL, in my case, . (Feel free to check this out!) HOST_URL https://shortener.cho.workers.dev After you modify , run again. wrangler.toml npm run deploy Debugging with Workers One of the hardest things about programming something in a new language or system is debugging. Although can bring you very far into the debugging journey, sometimes, you will encounter issues only visible after deploying to Cloudflare. wrangler dev Luckily, Cloudflare provides a way to tail production logs - as long as you write in your worker code for debugging purposes. After that, you can tail the logs using wrangler. console.log npx wrangler tail --format=pretty Go further This example is meant to showcase the power of a serverless URL shortener. Here are a couple of enhancements I would make to the code, and I will leave it as an exercise to the reader: Input validation for POST request Authenticate via Bearer/Basic token Return a fancy UI if the link is not found. ... (let's not limit creativity here) Source Code With ~50 lines we were able to completely recreate a URL shortener. You can check out all the code used in this tutorial . That's 10 minutes of your life writing (maybe) your first Worker and also (maybe) your first URL shortener. Hope you enjoyed this! here Also published . here

How to Build a Scalable URL Shortener With Cloudflare Workers and KV Under 10 Minutes

We've recently announced that WunderGraph is now fully open source. Today, we'd like to explain how you can leverage our API Developer Framework to add Edge Caching to your GraphQL APIs without locking yourself into a specific vendor. Caching GraphQL on the Edge should be vendor-agnostic. Services like Akamai and GraphCDN / Stellate offer proprietary solutions to solving this problem.
We'll compare the different approaches and their tradeoffs. Why did we create proprietary GraphQL CDN solutions A good question to start with is why we've created proprietary GraphQL CDN solutions in the first place. Most GraphQL implementations ignore how the web works The problem with most GraphQL implementations is that they don't really use the "platform" they are operating on. By platform, I mean the web, or more specifically HTTP and the REST constraints. The web has a lot to offer if you're using it in the right way. If read requests (Queries) were using the GET verb, combined with Cache-Control Headers and ETag, Browsers, CDNs, Cache Servers like Varnish, Nginx, and many other tools could handle Caching out of the box. You wouldn't really need a service that understands GraphQL. However, the reality is that most GraphQL APIs make you send Queries over HTTP POST. Cache-Control Headers and ETags don't make sense in this case, as all participants on the web think we're trying to "manipulate" something. So, we've created Edge Caching solutions that rewrite HTTP POST to GET and are capable of invalidating in very smart ways. By analyzing Queries and Mutations they're able to build up a cache and invalidate Objects as mutations flow through the system. Limitations and Problems of GraphQL CDNs / Edge Caches A GraphQL CDN creates a secondary source of truth As discussed in the constraint by Fielding, intermediaries can be used to implement a shared cache to improve the latency of requests. The problem I see with "smart" GraphQL Edge Caches is that they look into the GraphQL Operations to figure out what to cache and what to invalidate. Some GraphQL CDN implementations even allow you to use their API to invalidate objects. Layered System What sounds very smart creates a huge problem, you're building a second source of truth. Before using the GraphQL CDN, all you've had to do is implement your resolvers. Now, you've got to think about how to invalidate the CDN. Even worse, you're now programming against a single vendor and their implementation, coupling your application to a single service provider. You can't just switch easily from one GraphQL CDN provider to another. Contrary to REST/HTTP APIs, there's no standard on how to Cache GraphQL APIs, hence every implementation will be different. Another issue is that we're creating a secondary source of truth. Imagine we're putting a GraphQL CDN in front of the GraphQL API of GitHub to cache the Issues for a repository. If we're using "smart" Cache invalidation using Mutations, we're not able to update the cache if a user is bypassing our CDN and uses the GitHub API directly. A GraphQL CDN really only works if 100% of the traffic flows through the system. Your GraphQL Edge Cache won't work on localhost or in your CI/CD Testing takes up a huge part of software development, and we definitely want to have a great Developer Experience when building our APIs. Using a proprietary, cloud-only, Caching solution means that we're not able to test it locally or from within our Continuous Integration systems. You'll be developing on your local machine without the CDN, only to find out that something behaves weirdly when you're using the CDN in production. If we were using standardized caching directives, we'd be able to use any Cache Server, like Nginx or Varnish within our CI/CD pipeline to test our APIs. Proprietary GraphQL CDNs impose a vendor lock-in problem As mentioned earlier, there's no specification for how GraphQL Caching should work. Every implementation is different, so you're always tied to a specific vendor. They might get broke, they might get acquired, they might cancel their service, and they might change their pricing model. In any case, it's a risk that needs to be managed. A GraphQL CDN can introduce cross-origin requests Depending on the setup, adding a GraphQL CDN to your architecture could mean that your browser-based applications have to make cross-origin requests. That is, when your application is running on and your GraphQL CDN runs on , the browser will always make an extra Preflight request. It's possible to cache the Preflight request, but this still means that we'd have to do at least one extra request on the initial page load. example.com example.cdn.com A GraphQL CDN might not work well for authenticated Requests with Server Side Rendering (SSR) Let's say your application requires your users to be logged in, but you still want to be able to apply to cache. Aside from that, you'd also like to implement Server-Side Rendering (SSR). In the ideal scenario, you'd have your users log into your authentication server, which sets a cookie on your apex domain so that you're logged in on all subdomains. If your CDN is running on a different domain, it's going to be impossible to server-side render a page as the browser will not send the user's cookie to the CDN domain. Luckily, some providers offer custom domains, so cookie-based authentication might still work for you. Invalidating Deeply Nested GraphQL Operations might not work Here's an example of a GraphQL Operation that invalidates easily. mutation UpdateProfile {
    updateProfile(update: {id: 1, name: "Jannik"}) {
        id
        name
        friendsCount
    }
} If you've previously Queried the user with id 1, you can invalidate all records with that id and invalidate the following Query: query {
    profile(id: 1){
        id
        name
        friendsCount
    }
} Let's make it a bit more complex and query for all your friends: query {
    me {
       friends {
            id
            name
        }
    }
} Now, you made some good friends at the last conference. Let's add them: mutation AddFriend {
    addFriends(where: {user:{hasConnection: {conferences: {id: {eq: 7}}}}}){
        id
        name
    }
} We've now added all users as friends that had visited a conference whose id equals 7. At this point, we don't get back all the data when we query again for all your friends because the cache can't know that the mutation has invalidated the cache for the query. addFriends friends At this point, you've got to start adding response tagging, surrogate keys, or analyzing the GraphQL return types to make your invalidation strategy "smarter". We'll pick up this topic again after the benefits. Benefits of using a GraphQL CDN / Edge Cache When there's a cost, there are also benefits! Using a GraphQL CDN means, you don't have to change much of your application. In an ideal scenario, you simply change the URL of the GraphQL Server and everything should work. You also don't have to deploy and configure any tooling. You're buying a ready-to-use service that just works. Despite the problems discussed earlier, a GraphQL CDN can probably improve the performance of many applications. When not to use a GraphQL CDN As discussed in the section on Cache Invalidation, building a smart CDN service that doesn't know anything about your GraphQL Backend is actually extremely hard. The problem lies in the fact that the Backend is the source of truth, but doesn't share all this information with the Cache. In fact, if you're running into such issues where invalidation becomes hard, you might want to apply caching at a different level, inside the resolvers, or even one level below, at the entity level, using the DataLoader pattern. Additionally, if your data is expected to change frequently, it might make not much sense for you to cache at the Edge. Another big underrated flaw is that the invalidation of a distributed cache is eventually consistent. This means you serve stale content after a mutation for a short period of time. If you don't respect that in your architecture it has the potential to break the business logic of clients. Client (US) -> Query all posts to fill the cache
Client (US) -> Run mutation to update the post with ID:1
System fires a web-hook to a third-party service in FRA
Client (FRA) -> Query all posts -> stale content This is not GraphQL specific but with HTTP Caching the semantics are better understood. In summary, don't use a GraphQL CDN if you need write-after-read consistency. If your requirement is to support write-after-read consistency, there are multiple solutions to the problem. One is to not cache at the Edge, but rather at the application layer. In this case, you'd trade latency for consistency, which can be a good trade. Another way of solving the problem is by distributing the state across the edge and sharding the data based on location. This model of sharding is not always possible, but if the shared state is only used across groups of users in the same location, this solution could work very well. One example of this is Cloudflare Workers + Durable Objects, which gives you a simple Key-Value store that is persisted in a specific location, meaning that all users close to that one location can have a consistent state at low latency. When a GraphQL Edge Cache makes the most sense If you've got a single GraphQL API, and this API is the only API your frontend is talking to, you fully own this API and no traffic is bypassing your Cache, then such a CDN might actually make sense. Otherwise, I doubt you get the results you're expecting, especially not with the extra costs discussed earlier. WunderGraph - An Alternative Approach to GraphQL Edge Caching without vendor lock-in We've discussed the pros and cons of GraphQL CDNs, now I'd like to propose another approach that makes you stay vendor independent. When it comes to solving problems, sometimes it's smart to be dumb. I think a Cache can be a lot smarter when it's dumb and playing by the rules of the web, and that's exactly what we're doing with WunderGraph. and can be deployed everywhere. Our solution is Open Source How does it work? I've been working with GraphQL for many years now, and I've realized that when we deploy an application that uses GraphQL APIs, I've never seen the application change the GraphQL Operations at runtime. What applications do at runtime is change the Variables, but the Operations usually stay static. So, what we've done is, we've created a GraphQL to JSON-RPC compiler, which treats GraphQL Operations like "Prepared Statements", you've probably heard of the term from using a database. When first using a SQL statement, you send it to the Database and get back a handle to "execute" it later. Subsequent requests can now execute the Statement just by sending the handle and the variables. This makes the execution a lot faster. Because we're not changing GraphQL Operations at runtime, we can actually do this "compilation" step during development. At the same time, we're replacing HTTP POST for Queries with HTTP GET, while sending the Variables as a Query Parameter. By sending GraphQL Queries via JSON-RPC with HTTP GET,
we're able to automatically enable us to use Cache-Control Headers and ETags. And that's the whole magic of WunderGraph's Caching Story. We don't build a "smart" Cache. We don't cache Objects, and we don't have to build complex invalidation logic. We simply cache the response of unique URLs. For each Operation, you can define if it should be Cached and for how long. If a response might change frequently, you can also set the Cache time to 0, but configure "stale-while-revalidate" to a non-negative number. The client will automatically send the request to the origin with the ETag, the server will either send a refreshed response or 304, not modified. It's a very dumb approach. If the cached value is expired or stale, we're asking the origin if they have an update. Depending on the configuration, that might create a few more requests to the origin, but we're also not creating a second source of truth, or have to deal with the complexity of managing cache invalidation tags for nested objects. It's similar to using Change Data Capture (CDC) as a source for Subscriptions vs. simple Polling. CDC can get extremely complex to get right, while simple server-side Polling might work just fine most of the time. It's simple and robust. There's really not much we've invented here, all this is standard Caching behavior and any service, like Cloudflare, fastly, Varnish or Nginx supports it out of the box. There's a standard for how all the participants of the web treat Cache-Control and ETag headers, we've simply implemented this standard. By removing GraphQL from the equation, we've made it compatible with the web. If you build tools for the web, you should respect how the web is built, otherwise, you're creating more problems than you're solving. Additional Benefits of the WunderGraph GraphQL Caching Approach It's not just CDNs and Servers who understand the Cache-Control and ETag Headers. Browsers also automatically cache and invalidate your responses, without adding a single line of code. Additionally, because we've removed GraphQL from the runtime, we've automatically reduced the attack surface of our application. If our frontend doesn't change GraphQL Queries at runtime, why expose an API that allows you to do so? Limitations of the JSON-RPC Caching Approach One of the limitations of this approach is that we're no longer able to use regular GraphQL Clients, as they'd expect us to send GraphQL Operations over HTTP POST, and Subscriptions over WebSockets. That's not an issue for new projects, but might be a blocker for existing applications. Btw. we've got an internal RFC to add a compatibility mode, allowing e.g. Apollo client / urql, etc. to work with WunderGraph through an Adapter. If you're interested, please let us know. That said, using plain JSON-RPC would be very inconvenient. That's why we're not just compiling GraphQL to JSON RPC, but also generating fully type-safe clients. One such client-side integration is the , making it super easy to use WunderGraph with NextJS, including Server-Side Rendering, Authentication, File Uploads, etc... NextJS package Another limitation, as of now, is that you have to self-host WunderGraph. We're working on a hosted Serverless solution, but as of now, you'd have to deploy and run it yourself. While it might not be super convenient, this also comes with an advantage: WunderGraph is Apache 2.0 licensed, you can run it anywhere. How can you deploy WunderGraph Now that we've discussed the pros and cons of the WunderGraph approach to caching, let's see how we can deploy WunderGraph to achieve good Caching results. First, you don't have to deploy WunderGraph globally. It's possible to run it close to your origin, e.g. the (micro-) services that you'd like to use. In this scenario, it's possible to run WunderGraph as an Ingress to your other services. The architecture of this scenario looks like this: Client -> WunderGraph Server -> Origin Deploy WunderGraph with Nginx or Varnish as an additional Caching Layer If you're deploying WunderGraph close to the origin, it will automatically add the required Cache-Control & ETag Headers. This setup might already be enough for your scenario. However, in some scenarios, you'd like to add another layer of Caching Servers. This could be e.g. a cluster of Nginx or Varnish servers, placed in front of your WunderGraph server. Architecture updated with this scenario: Client -> Cache Server -> WunderGraph Server -> Origin Deploy WunderGraph with Cloudflare or fastly as an Edge Cache Depending on where your users are, a centralized Caching Layer might not be sufficient for your use case. In this scenario, you can use services like Cloudflare (Workers) or fastly, to add a globally distributed GraphQL CDN / Edge Cache. What's important to note here is that you're not tied to a specific solution. As mentioned before, we're using standardized Cache-Control directives, supported by all Cache Server solutions, so you're not tying yourself to a specific vendor. Updated Architecture: Client -> CDN -> WunderGraph Server -> Origin Deploy WunderGraph directly on the Edge, using fly.io Another option is to deploy WunderGraph directly on the Edge, removing the need for an additional Caching Layer. Services like allow you to deploy Containers as close to your users as possible. The architecture of this scenario looks like the following: fly.io Client -> WunderGraph Server (on the Edge) -> Origin Deploying a service on the Edge is not always beneficial There's one important point I'd like to make that applies to all solutions mentioned in this post. If we're deploying a workload on the Edge, this service will be very close to the users, which is generally a good thing. At the same time, depending on where on the "Edge" we're processing a request, there might be a random latency between ~0-300ms to your origin server(s) per roundtrip. If we have to do multiple roundtrips to fetch the data for a single client request, this latency can add up. As you can see, it's not always beneficial to the overall performance to have logic running on the edge. There's one more thing! This solution works for GraphQL, REST, gRPC, and other Protocols as well! You've heard that right. WunderGraph is not just about GraphQL origins. We support a wide array of upstream protocols, like REST, gRPC (coming soon!), and Databases like PostgreSQL, MySQL, Planetscale, etc... WunderGraph ingests all your services and creates a "Virtual Graph", a GraphQL Schema representing all your services. The Caching layer we've described above does not just work for GraphQL origins, but all services that you've added to your Virtual Graph. This means, you're able to apply one unified layer of Authentication, Authorization, and of course Caching to all your APIs. From our experience, it's rarely the case that you're connecting a frontend to a single GraphQL server. Instead, you're probably going to connect it to many services, which is what we're trying to simplify, and caching is one part of the story. Conclusion We've discussed various options to improve application performance with different kinds of Caching systems. There are ready-to-use solutions that come with simplicity but also lock you into specific vendors. With WunderGraph, we're trying to offer an Open Source alternative that's based on standards and implemented by many tools and vendors, so you're able to choose the best-of-breed solution for your situation. If you're looking into adding Caching to your stack, we'd love to talk! or . Join us on Discord connect on Twitter Also published . here

How to Leverage Open Source GraphQL CDN/Edge Cache With Cloudflare, Fastly, and fly.io

Writers, your just got a backend lift! If you’ve published a story on April 20th, or since, you will probably see a larger number of reads on your stories. The reason for this is that we’ve started collecting data from . Why did we do this? For more accurate readership reports! story stats Cloudflare Analytics and so using Google Analytics API (as we previously were) to measure readership per story is problematic.  Cloudflare has the advantage of being so integral to how our site’s content is delivered, that their servers’ get a chance to log the traffic before a client could ever block it. counts a successful request to a URL delivering content of type HTML. In this way, we gain a straight forward view into story traffic. 58% of tech-savvy audiences block Google Analytics Cloudflare’s approach “This doesn't mean you can go to your advertisers and start demanding more ad revenue. It should, however, mean that you now have a more accurate picture into the actual resource demands required to run your site.” - Cloudflare’s Matthew Prince Why not?  Well, one reason is that Cloudflare includes more bot and crawler traffic in their analytics. Thankfully, we can filter page views by what Cloudflare calls a botManagementDecision such that the traffic has been deemed to be “likely human” in origin.  Are you a ‘likely human’ reading this story? Then you will count towards the reads at the top of this story page and writers will see that in . Here is the filter used in the GraphQL query: their story stats "pageviewsFilter": {
        "AND": [
          {
            "botManagementDecision": "likely_human"
          },
          {
            "datetime_geq": startDate,
            "datetime_leq": endDate
          },
          {
            "requestSource": "eyeball"
          },
          {
            "AND": [
              {
                "edgeResponseStatus": 200,
                "edgeResponseContentTypeName": "html"
              }
            ]
          }
        ] This was my first time using GraphQL and while it looks like there is sort of living documentation supported, I had a hard time figuring out how to get the data I wanted.  Most of their documentation was focused on firewalls and threat detection. I ended up opening up to see what requests were being sent as I changed filters while viewing their graphs on their own site. Thankfully, I was able to paste those into , an open source client to help you write queries, and verify I was getting the same page reads seen in the charts. my browser’s network tab GraphiQL I really like the idea of using these server logs over downloading and executing javascript on the client.  It is more efficient and less intrusive.  It would be nice to remove the Google Analytics code from our site, but we still rely on it for tracking the time spent on pages and some click events. And keeping it on site just may help with search traffic ;-) Perhaps we’ll roll our own system there in the future. Currently we still only fetch stats once a day, but I’ll be looking into fetching these stats more often so stay tuned for that. Okay, so that’s the update, enjoy! You should be seeing a lot less of these guys on your story now 🙂 Head on over to today and see for yourself! https://app.hackernoon.com/stats

HackerNoon Story Stats Update: Reads Powered By Cloudflare

Actualización de estadísticas de HackerNoon Story: lecturas impulsadas por Cloudflare

Mise à jour des statistiques de l'histoire de HackerNoon : lectures optimisées par Cloudflare

HackerNoon Story Stats Update: Reads Power by Cloudflare

HackerNoon ストーリー統計の更新: Cloudflare による読み取り

Atualização das estatísticas da história do HackerNoon: leituras alimentadas pela Cloudflare

Cập nhật số liệu thống kê về câu chuyện của HackerNoon: Số lượt đọc được cung cấp bởi Cloudflare

HackerNoon 故事统计更新：由 Cloudflare 提供支持的读取

Last week, I had a problem. I wanted to start blogging using . But for SEO reasons, I wanted to put my new blog at , instead of either a completely different domain, or a subdomain like blog.cloak.ist. Ghost cloak.ist/blog That's because I want to build up SEO value for my principal domain, . Google sees subdomains as being a whole other thing. So it's muuuuch better to use if you can. cloak.ist yoursite.com/blog I spent 3 days figuring out how to set up Ghost at cloak.ist/blog. It was pretty painful. So I've decided to celebrate my blog finally existing by writing a guide to how I set it up. I hope it helps you out! Before we get started - You need to own a domain and be using Cloudflare to manage its DNS - You need to be able to buy a $5/month droplet on DigitalOcean Step 1: create a DigitalOcean 1-Click Ghost Droplet Ghost has a hosted option which you can pay for. But I'm going cheap – and also I'm pretty sure, from reading a few things online, that their hosted option won't work with this guide. So head on over to DigitalOcean and . Choose the $5/month droplet. set up a new droplet with their 1-click option Once the droplet is set up, don't go and do anything in it just yet. Just make a note of the droplet's ipv4 address . Step 2: point a subdomain at your droplet's IP You need to set Ghost up at a subdomain at your domain, like . blog.yoursite.com This is so that we can later tell to get all content from that subdomain. But don't worry, we'll also have the option of telling Cloudflare to restrict access to that subdomain to you and Cloudflare, so that you don't have a competing location for your blog living out there. yoursite.com/blog So go to Cloudflare and create a new A record, with Name set to whatever you want the subdomain to be (e.g. Name will be blog if you want to use blog.yoursite.com), and IPv4 address set to the droplet's IP that you have noted. Click the orange cloud to change it from Proxied to DNS Only! This caused me a of pain. VERY IMPORTANT: lot Step 3: drink some tea The next step will be to go back to your droplet and set up Ghost on it. But if your subdomain hasn't propagated yet (in other words, if the internet hasn't yet realised that you added the A record from the previous step), the set up will fail. DNS should propagate pretty fast, but please, please: Have a little break to give DNS time to propagate, hence the tea. I recommend chamomile.On returning from your delicious tea session, check DNS with a before going onto the next step to make sure that your subdomain is pointed at your droplet IP. service like this If you left the orange cloud enabled in the previous step, it's very likely that your Ghost set up will fail and you'll see Cloudflare IP addresses when you do the DNS check. (Not doing this step properly meant that I went into a frenzy of destroying and then remaking droplets. I went through around 5 of them before I got it right.) Step 4: set up Ghost Use . this guide to set up Ghost on your droplet When you're asked for the URL of your Ghost droplet, enter: https: //yoursubdomain.yoursite.com All of the rest of the set up should be straightforward – use the standard options throughout. If you get an error, it could be that your DNS didn't propagate yet so Ghost was unable to issue an SSL certificate for your subdomain. I'd recommend going back to Step 1 and starting again if that happens – alternatively, if your error looks different, have a google around to see if you can fix it. Quite a lot of people seem to have had issues with DigitalOcean's '1-click' Ghost installation... At this point, you should also check that your blog is live at https://yoursubdomain.yoursite.com. Keep your SSH session open! Step 5: put your blog at /blog We're going to put your blog at a subdirectory on the subdomain now, we move it to living at your root domain instead of the subdomain. before Run the following commands while you're still in your SSH session: sudo -i -u ghost-mgr cd / /www/ghost ghost config url https: ghost restart // This will let you configure Ghost var // This will move you into the Ghost installation folder to make changes //yoursubdirectory.yoursite.com/blog // This will change the URL of your Ghost blog. You can obviously change /blog to whatever you want // IMPORTANT! This will restart your Ghost instance so that the URL change comes into effect. At this point, check that https://yoursubdirectory.yoursite.com/blog is working. If you see a 404, you probably need to run ghost restart. Also, you could now set yourself up as the admin of the blog by going to https://yoursubdirectory.yoursite.com/blog/ghost. This is not a necessary step though – you can do it later. Step 6: set up a Cloudflare worker We're going to set up a Cloudflare worker at your subdirectory https://yoursite.com/blog which is going to act as a reverse proxy – so it'll basically serve your Ghost blog at that URL. Go to Cloudflare -> Workers'Add Route' -> yoursite.com/blog*'Add Route' -> www.yoursite.com/blog* if you want to make sure you'Manage Workers''Create a Worker' At this stage, you're going to want to replace everything in the code editor with the following script, after : changing the config object to reflect your set up config = { : , : , : ,
} { url = URL(request.url) targetPath = url.pathname response = fetch( ) (
  	targetPath.includes( ) || 
    targetPath.includes( ) ||
    targetPath.includes( ) ||
    targetPath.includes( )
  ) { response
  } body = response.text() body = body.split(config.subdomain).join(config.root)

  response = Response(body, response) response
}

addEventListener( , (event) => {
  event.respondWith(handleRequest(event.request))
}) // Config for our subdomain (where the Ghost blog currently lives) // and our root domain (where we want it to live) // and our blogPath (the subdirectory we want it at) // Change these to suit your case! const subdomain "yoursubdirectory.yourdomain.com" root "yourdomain.com" blogPath "blog" // Function that processes requests to the URL the worker is at async ( ) function handleRequest request // Grab the request URL's pathname, we'll use it later const new const // Change request URLs to go through to the subdomain let await `https:// ` ${config.subdomain} ${targetPath} // Ghost loads assets like JS and CSS from 3 subdirectories // We don't need to change these requests at all // So if we're getting stuff from those subdirectories, // we return the response of the fetch request from above // immediately. if '/blog/favicon.png' `/ /assets/` ${config.blogPath} `/ /public/` ${config.blogPath} `/ /content/` ${config.blogPath} return // In other cases - which will usually be pages of the // Ghost blog - we want to find any reference to our subdomain // and replace it with our root domain. // This is so that things like our canonical URLs and links are // set up correctly, so we NEVER see our subdirectory in the code. // First we get the body of the response from above let await // Then we search in the body to replace the subdomain everywhere // with the root domain. new return "fetch" I've written comments explaining what this code is doing. Paste this code in, give the Worker a slightly clearer name – e.g. yoursite-blog – hit Save and Deploy, and then go back to the main Workers page: Here, you'll want to hit Edit on the Route you created earlier, and set the Worker on it to the one you just made. Step 7: test it out At this point, if all has gone well, you should have a fully working blog at yoursite.com/blog (or whichever subdirectory you chose). If you don't, check through the instructions to see if there's anything you missed. Step 8 (optional): Restrict access to the subdomain blog So to make things cleaner, you might want to do one last thing: use a Cloudflare Firewall rule to make access to yoursubdirectory.yoursite.com impossible for everyone apart from: 1) you (because you're going to need to go there still to use the CMS) 2) Cloudflare Workers. I feel like this may be a good thing to do for SEO, since otherwise you have a blog with exactly the same content and with competing canonical URLs. (I could be wrong here though – please let me know via the live chat if so!) First, we'll create an IP list for Cloudflare IP addressesGo to Configurations at your main Cloudflare menu: Select Lists, 'Create new list': Paste everything from into your new list and hit SaveGo to your domain from the main Cloudflare menu, then Firewall -> Firewall Rules -> Create a Firewall RuleSet your rule up like this: this list - My Ghost subdomain is cloak.ist – you'll want to replace this in the URI Full filter. - You'll want to change the list filter to use your list of Cloudflare IPs. - Finally, you'll want to use filters like the last 3 to add your own IP addresses. (I have 3 here because I move between different locations a lot.) - ⚠️ You'll need to do this to ensure you have access to editing the Ghost blog! That's it! At this point you should have: - A working blog at yoursite.com/blog......which has correct links to itself, and correct canonical URLs at your root domain (yoursite.com) - A subdomain where you can go to manage your blog, which is only accessible to you and Cloudflare Hopefully this will be much better for your SEO than if you'd had to put your Ghost blog at the subdomain! If you have any questions at all, feel free to tweet @cloakist . Also published at: https://cloak.ist/blog/ how-to-put-a-ghost-blog-at-a-subdirectory-using-cloudflare-workers/

How to set up a Ghost blog on a subdirectory via Cloudflare Workers

When you buy a Domain Name from a registrar (e.g. GoDaddy) to host a website, there is high probability that you would also use their DNS zone hosting services too. However, a secure website requires an SSL certificate, which is not free. Therefore, some people utilize Cloudflare services to get a free SSL bundled with their DNS services. You might have many reasons to choose Cloudflare as your preferred DNS service provider, but you realize very soon that it has started posing new challenges due to some inaccurate configurations. In fact, there is nothing wrong at the Cloudflare-side. It's merely due to lack of our understanding and insufficient documentation for novice users. Imagine a scenario where you subscribed to a web-based application that is hosted by a third-party (e.g. Okta, Zoho). It simply means that you do not have direct access to their web-server instance. You would assume that a simple CNAME record configuration in Cloudflare's DNS setting should be enough to start serving your website. It's not that straight forward. Firstly, you must understand that the Cloudflare is running an HTTP proxy server (i.e. kind of a web-broker) that has two sides to be managed. Being a web-proxy, the Cloudflare allows two different options to you. First option is a Proxy-mode (i.e. traffic intercepted by the Cloudflare) and another one is a DNS-only mode (i.e. traffic sent directly to  web origin servers without changing anything). On one side, it is accepting web request from your user's web browser (e.g. Chrome, Safari) that expects a secure traffic (i.e. HTTPS) therefore the SSL certificates used by Cloudflare must be compatible and acceptable to your user's web browser. This SSL certificate is a major hurdle many times in the configuration. On the other side, the Cloudflare must contact the web-application's origin servers that might be expecting a secure traffic (i.e. HTTPS) therefore SSL certificates acceptable to your origin servers must be used by Cloudflare in a proxy-mode else choose DNS-only mode. Many times people choose to use DNS-only, but the website is not served properly yet. Perhaps, additional configuration is required in Page Rules to route the traffic in right manner to origin servers. Sometimes the problem is due to type of SSL certificate you are using. Also, have some patience while you migrate from some other DNS provider because DNS settings propagation time could also frustrate sometimes. Typically, it takes approx. 24 ~ 48 hours to get DNS settings working properly after you make changes. Acquire some knowledge related to SSL certificates verification method as well, if you are a novice. Some companies (e.g. Okta) have simplified configuration of DNS settings using Cloudflare. I enjoy using Cloudflare due to its analytics capabilities and simple to use user interface.

How To Link a Third-Party Hosted Website using Cloudflare Services

Two companies that design and manufacture wedding dresses are suing Cloudflare for copyright infringement after it failed to terminate service to 99 'repeat infringer' websites. Allure Bridals and Justin Alexander claim that they sent 7,000 DMCA complaints to the CDN company but aside from passing the notices on, Cloudflare failed to take more meaningful action. When copyright holders feel they have exhausted all options to have websites stop their allegedly-infringing activities, there is a growing trend to move further up the chain. Sites now regularly have copyright complaints filed against their hosting companies and domain registries, for example, demanding that they take action to prevent contentious behavior. Since millions of websites now use Cloudflare’s services, that makes the CDN provider a prime candidate for pressure. A new case filed yesterday in a Tennessee district court provides yet another example. American Clothing Express Inc., which does business as Allure Bridals and Justin Alexander, designs and manufactures wedding dresses. As part of the companies’ sales and marketing efforts, they claim to spend hundreds of thousands of dollars per year on photoshoots featuring models wearing their creations. According to the companies, however, the resulting photographic images are also being deployed by unauthorized overseas websites (sample below) in an effort to drive customers to unaffiliated bridal stores in local markets selling “cheap imitation” dresses. The plaintiffs state that they lack a meaningful remedy against such sites, noting that the majority are hosted on servers in China, other locations in South East Asia, or on offshore servers that advertise their non-compliance with United States’ copyright laws. “Complaints sent by Plaintiffs, or their agents, to the Infringing Website Defendants, or to the entities hosting them in these far-away jurisdictions, largely fall on deaf ears. Domestic judgments obtained against the Infringing Website Defendants are often unenforceable against them in their home jurisdictions,” the complaint reads. The filing lists 99 websites (represented by Does 1-200) falling into these categories that all have something in common – they are or have been customers of US-based Cloudflare. As a result, the plaintiffs have resorted to filing infringement notices with the CDN company, hoping it will take action to restrict the availability of the infringing images. Indeed, over the past three years the companies claim that they sent several thousand infringement notifications to Cloudflare which included the URLs of pages on the allegedly infringing sites where unlicensed images were being used. The complaint acknowledges that Cloudflare forwarded the complaints to its customers and their hosts but due to the nature of the clients, the hosting providers mostly ignored the takedown demands. The complaint targets the operators of the 99 sample sites with claims of direct copyright infringement but additionally, due to Cloudflare’s involvement, the CDN company itself is accused of contributory copyright infringement. “CloudFlare had actual knowledge of the specific infringing activity at issue here because anti-counterfeiting vendors retained by Plaintiffs delivered more than seven thousand notifications to CloudFlare of the ongoing infringement being prosecuted herein over the course of three years,” the complaint reads. In common with a in California involving another bridal company, the plaintiffs in this matter also state that Cloudflare should have taken more permanent action when they realized that complaints were being made against the same customers time and again, as illustrated by the sample in the image below. similar on-going case “CloudFlare could have stopped this infringement being perpetrated through its CDN by simply terminating the accounts of repeat infringers,” the complaint continues. “CloudFlare has never terminated a repeat infringer in response to notifications sent by Plaintiffs or other bridal manufacturers. Consequently, an exceedingly disproportionate amount of websites infringing Plaintiffs’ copyrights are optimized by CloudFlare, as opposed to other providers of CDNs, due to CloudFlare’s well-known policy of refusing to terminate repeat infringers.” While the plaintiffs don’t mention Cloudflare’s competitors by name, the complaint alleges that in response to similar copyright infringement notices, other CDN providers told their clients that if the images weren’t removed, their entire website accounts would be terminated. The term ‘repeat infringer’ is becoming increasingly common in United States copyright infringement cases. In December 2019, Cox Communications was hit with a after a Virginia federal jury determined that the ISP didn’t do enough to stop repeat infringers. Cox was found to be contributorily and vicariously liable for the alleged pirating activities of its subscribers on more than 10,000 copyrighted works. $1 billion copyright infringement verdict For comparison, Allure Bridals and Justin Alexander state that Cloudflare is liable for contributory copyright infringement relating to more than 5,000 infringing images published on 99 different websites. Overall, Cloudflare serves many thousands of pirate sites, making the outcome of this and similar cases of particular interest. In respect of the “willful and intentional” direct infringement claims against the 99 websites themselves, Allure Bridals and Justin Alexander request actual or statutory damages, injunctive relief to prevent the ongoing infringements, and the destruction of all copies of copyright works made in violation of the bridal companies’ rights. The contributory copyright infringement claim against Cloudflare asserts that the CDN company assisted the direct infringers by storing copies of the infringing images on servers in the United States, improving the performance of the infringing websites, while concealing their true locations. As a result, Cloudflare’s behavior is also described as “willful and intentional”, with the plaintiffs demanding a similar injunction in addition to actual or statutory damages. The complaint can be obtained here ( pdf ) Originally published as “ Cloudflare Sued For Failing to Terminate 99 ‘Repeat Copyright Infringing’ Sites " with the Attribution-NonCommercial 3.0 Unported (CC BY-NC 3.0) license

Cloudflare Sued For Failing to Terminate 99 ‘Repeat Copyright Infringing’ Sites

On July 18th, 2020, caused an outage for about 27 minutes affecting businesses worldwide. a configuration error in Cloudflare’s backbone network The error configuration sent all traffic across the backbone to Cloudflare’s Atlanta node and unfortunately “overwhelmed” the router. Cloudflare later mentioned they saw traffic drop by about 50% across the network. Although not entirely catastrophic thanks to the architecture design, but many locations with heavy internet usage were affected including San Jose, Seattle, Los Angeles, Chicago, Washington, DC, London, Amsterdam, Frankfurt, Paris, Stockholm, Moscow, São Paulo, and the list goes on. How our system handled it Cloudflare is one of the many CDNs mlytics offer via Power-Ups (CDN marketplace) on the platform. When the outage occurred, the AI Load Balancing swapped out Cloudflare with the next best-performing CDN (varies in different regions) for requests. The chart below is the AI Load Balancing optimization chart, it shows which CDNs were used (swapped over) during a certain timeframe. In this case, we’re using a demo site with Akamai and Cloudflare installed. As shown, Akamai had several query spikes in the states due to the Cloudflare performance drop – the system is actively swapping over to Akamai. Frequent swaps were made from Cloudflare to Akamai in the US This had successfully helped many of our customers mitigated the outage, and we received no complaints over the course. On the chart, we’re also seeing a latency spike and availability drop for Cloudflare in the same time frame. This helps illustrate a clear picture of what happened when aligning this with the chart above. Pulse It shows a latency spike and availability drop for Cloudflare on the Pulse chart We believe all the CDN providers are doing their best to deliver the best performance possible, but unfortunately, things do happen. And our job at mlytics is to help our customers minimize the exposure of such risk and maximize the uptime at all times. Again, outages DO happen This is a wake-up call reminding all businesses will eventually have to deal with outages if offering any form of web services. Thus, a solid cloud redundancy and disaster recovery plan in place to prevent any event from taking down your service is imperative. Imagine losing money and customers hate you… horrible right? But stopping users from doing what they were doing (especially if they enjoy it so much) is your express train ticket to PR hell. Small or medium businesses might have room for mistakes, but some of the bigger companies affected by the outage such as Discord, Feedly, Politico, Shopify, and League of Legends might not be so lucky. These are some of the most used online services today, and imagine millions of users got disconnected without being informed. We've talked about in the past, and there are just so many factors that can bring a website off the grid. A website without a backup plan is exactly like sailing a cruise without liferafts – you’re betting you won’t need it. website downtime causes and solutions Previously published at https://mlytics.com/blog/how-we-helped-our-users-mitigated-the-cloudflare-outage/

Helping Our Users Mitigate The 2020 Cloudflare Outage - How We Did It

In 2018, four of Japan's largest manga publishers filed a motion at a Tokyo court demanding that Cloudflare stop providing services to several 'pirate' sites, including Mangamura replacement Hoshinoromi. The companies now reveal that a settlement has been reached with Cloudflare to "stop the replication" of the sites on its Japan-based servers, if a court declares them illegal. As the largest CDN and DDoS mitigation service on the planet, Cloudflare provides services to millions of websites. A tiny proportion of these sites are on the radars of entertainment and publishing companies since they either directly offer or link to unlicensed copies of copyrighted works. As a result, Cloudflare is under almost continual pressure to cease doing business with these entities. As first reported in September 2019, Japan-based publishers Shueisha, Kadowaka, Kodansha, and Shogakukan sued the ‘pirate’ site Hoshinoromi in a New York federal court. The platform, which positioned itself as a replacement for self-shuttered pirate site Mangamura, was accused of “willful and massive infringement” of the publishers’ copyrights. here on TF That case is still ongoing and according to a filing late last week , the publishers are having considerable difficulty identifying and serving the defendants, so need an extension. Cloudflare was mentioned in that lawsuit too and it now transpires that the same publishers previously targeted the CDN company in a Japan court back in 2018. ( pdf ) In common with other lawsuits in Japan, details are hazy. However, according to a joint statement issued late last week, Shueisha, Kadowaka, Kodansha, and Shogakukan filed a motion at the Tokyo District Court in August 2018 with a demand that Cloudflare should stop providing services to several ‘pirate’ platforms, Hoshinoromi included. Due to caching, that amounted to Cloudflare delivering infringing content to the public, they argued. For reasons that appear related to the ongoing case in the United States, they have waited until now to reveal a settlement of sorts with Cloudflare. It was reportedly reached in June 2019 and seems to hinge on whether a court determines that the ‘pirate’ sites in question are copyright-infringing and therefore illegal. The publishers’ statement indicates that when the ‘pirate’ sites using Cloudflare are viewed by users in Japan, most of those users will be accessing them via Cloudflare’s Japan-located servers. So, if the Tokyo District Court rules that the sites are illegal, Cloudflare has reportedly agreed to “stop the replication of the sites to Cloudflare’s servers in Japan.” At least in part, the announcement is designed to be a warning to other ‘pirate’ sites that may be considering using Cloudflare’s services to improve uptime and general accessibility. Whether it will make much of a difference on the ground remains to be seen, however. While this particular matter appears to be settled, last December by Takeshobo, another major publisher based in Japan that distributes dozens of manga publications, many under the Bamboo Comics label. Cloudflare was sued The publisher said it was forced to sue Cloudflare because takedown notices sent to the CDN company in respect of a nameless ‘pirate’ site were effectively ignored, allowing infringing material to stay online via Cloudflare’s services. Progress in that particular case is unknown but the settlement with Shueisha, Kadowaka, Kodansha, and Shogakukan may provide a possible solution for Takeshobo. Cloudflare is obviously extremely cautious when faced with similar lawsuits, always insisting that as a service provider it is not responsible for the activities of its users. Last week, however, the effects of a ruling handed down in December by a German court saw Cloudflare under the threat of serious fines. disconnect pirate music platform DDL-Music Originally published as “ Cloudflare Agrees to Stop Caching Pirate Content in Japan, If Court Declares Sites Illegal ” with the Attribution-NonCommercial 3.0 Unported (CC BY-NC 3.0) license

Cloudflare Agrees to Stop Caching Pirate Content in Japan, If Court Declares Sites Illegal

With the advent of Cloudflare’s new , it is now easy to download and inspect a zone’s access logs by a single RayID (Cloudflare’s fancy term for a request ID), a defined time period, or from the last downloaded. In our case, we ship all of our other logs to so everyone can analyze and parse them in any way imaginable! Enterprise Log Share (ELS) REST API RayID Sumo Logic Unfortunately, there seems to be no cut-and-dry way to integrate the two. Thankfully, Lambda functions are now able to be run on a scheduled interval using — sort of like you would a cron job on a server. AWS CloudWatch Scheduled Events There are a couple of clear benefits of using a Lambda function instead of a job on a physical/virtual server you run yourself: Execution of a Lambda function , and it may actually cost nothing if you remain within the free tier AWS hands out every month Low cost: costs next to nothing In my case, I used the to allow me to quickly get up and running, invoke a function locally (which is key to developing and iterating quickly), and it makes deployments simple. I also recommend checking out , too. Ease of development: Serverless framework apex Lambda functions are This is important for things which require speed, such as parsing and ingesting logs for a high-traffic site like ours. Speed: fast. Write the code, test and deploy — that’s it! No Chef, no Docker, no nothing… just write your function, ship it and it is live. No servers to build/maintain: How It Works First, you’re going to want to to ingest the Cloudflare logs. Set the source category, host, etc. at SumoLogic to make the configuration of the Lambda function easier. create a new HTTP source Next, create a new Serverless function: $ mkdir -p ~/cloudflareToSumoLogic $ cd ~/cloudflareToSumoLogic $ serverless create \--name cloudflareToSumoLogic \--template aws-nodejs \--path ~/cloudflareToSumoLogic Let’s walk through the configuration file: serverless.yml functions: cloudflareToSumoLogic: # This is the name of the Lambda function **_\# This is the name of the handler; this is the "main"_    \# _function called by Lambda_**  
handler: index.handler

**_\# This function was written to be run every one minute  
# Any adjustments will require code changes..._**  
events:  
  - schedule: rate(1 minute)

**_\# Required environment variables:_**  
environment:  
  **\# Grab this URL from the Sumo Logic source you just created**  
  SUMO\_ENDPOINT: ...  
  **\# Get these values from the Cloudflare admin UI**  
  CLOUDFLARE\_ZONE\_ID: ...  
  CLOUDFLARE\_AUTH\_EMAIL: ...  
  CLOUDFLARE\_AUTH\_KEY: ... Finally, you’re going to need the function: Now, try it out! $ serverless invoke local -f cloudflareToSumoLogic startTime: 2017-02-11T15:07:00.000ZendTime: 2017-02-11T15:08:00.000Z cloudflareOpts: { method: 'GET',hostname: 'api.cloudflare.com',path: '/client/v4/zones/.../logs/requests?start=1486825620&end=1486825680',headers:{ 'X-Auth-Email': ' ..','X-Auth-Key': '...' } } . res.statusCode:  200 res.headers:  { date: 'Sat, 11 Feb 2017 15:38:40 GMT','content-type': 'application/json','transfer-encoding': 'chunked',connection: 'close','set-cookie': [ '__cfduid=...; expires=Sun, 11-Feb-18 15:38:39 GMT; path=/; domain=.cloudflare.com; HttpOnly' ],vary: 'Accept-Encoding','x-cf-request-id': '00000000-0000-0000-0000-000000000000','strict-transport-security': 'max-age=31536000','served-in-seconds': '0.355',server: 'cloudflare-nginx','cf-ray': '...-ORD' } Log events: 659 Everything look good? It’s time to deploy: $ serverless deploy Serverless: Packaging service...Serverless: Uploading CloudFormation file to S3...Serverless: Uploading service .zip file to S3 (9.45 KB)...Serverless: Updating Stack...Serverless: Checking Stack update progress........................Serverless: Stack update finished...Serverless: Removing old service versions... Service Informationservice: cloudflareToSumoLogicregion: us-east-1api keys:Noneendpoints:Nonefunctions:cloudflareToSumoLogic Analyzing Ingested Logs in Sumo Logic Now that your logs are being ingested every minute, you can run the following query and you should be able to start seeing results like the screenshot above: _sourceCategory=...| json auto| toLong(num(timestamp)) as timestamp| timestamp as _messageTime| fields - timestamp| fields - _raw Replace with the HTTP source category that you created earlier for your Cloudflare logs. This snippet parses the JSON lines automatically, translates the log timestamp to a time object, replaces the with the log timestamp, then remove the raw JSON object and its ingest timestamp. _sourceCategory timestamp _messageTime

How to Ship Cloudflare logs to Sumo Logic with Lambda

I am building an open-source, highly scalable, and low-cost . URL shortener with Cloudflare KV By low-cost I mean, just $5 per month on Cloudflare worker and a free plan for basic requirement Idea At DaySchedule, we had many customers upvote a URL shortener feature with auto-expirable links to book an appointment instead of sharing their original URL. So, I decided to try Cloudflare for this project - By leveraging Cloudflare KV's capabilities, the goal was to develop a solution that offered not only URL shortening but also customization options, analytics, and scalability, all while maintaining a significantly lower cost compared to our established services on AWS. Features Expirable links using option on KV cacheTtl Custom domain using Cloudflare for SaaS Analytics using Worker analytics API I used by for API routing, and it's blazing fast and offers Cloudflare pages template to build Edge projects quickly. hono.dev @yusukebe Here is an example of from their docs - server.ts import { Hono } from 'hono'
const app = new Hono()

app.get('/', (c) => c.text('Hono!'))

export default app Controller The API is all I need to create, update, and manage the short links. /links const links = new Hono<{ Bindings: Bindings }>();

links.post(
  '/',
  validator('json', (value, c) => {
    const parsed = linkSchema.safeParse(value);
    if (!parsed.success) {
      return c.json(parsed, 400);
    }
    return parsed.data as ShortLink;
  }),
  async (ctx) => {
    const data = ctx.req.valid('json');
    // Set expire_at if given, or fallback to 1 month expiry
    const expireAt = data.expire_at
      ? dayjs(data.expire_at).unix()
      : dayjs().add(1, 'month').unix();
    if (expireAt <= dayjs().unix()) {
      return ctx.json(
        { message: 'The expire_at must be greater then current date time' },
        400
      );
    }

    let key = data.key || nanoid(7);
    let exists = await ctx.env.SHORTLINKS.get(key);
    while (exists) {
      key = nanoid(7);
      exists = await ctx.env.SHORTLINKS.get(key);
    }
    await ctx.env.SHORTLINKS.put(key, JSON.stringify(data), {
      expiration: expireAt,
    });
    return ctx.json(
      { ...data, key: key, short_url: `https://idm.in/${key}` },
      200
    );
  }
); Code explanations - This code is request to create short links. Here's a breakdown of the code to explain what I am doing: POST It creates an instance of the Hono object. const links = new Hono<{ Bindings: Bindings }>(): Validation with zod - The function is used as a middleware for validating the incoming JSON payload defined in . validator('json', (value, c) => {...}): linkSchema Set expiration: Checks if an expiry date is provided in the payload. If not, it sets an expiry date one month from the current date and time. Generates a unique key for the link using or uses the provided key from the payload if available. nanoid(7) Stores the link data (converted to a string) in the Cloudflare KV ( ) with an expiration time based on the calculated expiry date. JSON SHORTLINKS Demo The demo is available on , it's free to use and open-sourced on Github to . IDM build your custom URL shortener You can clone the repo to deploy on your Cloudflare account. Don’t forget to on GitHub to show your support :-) Star the repository Also published here.

How I Built a Low-Cost URL Shortener With Cloudflare KV

Every now and then, I get a dumb little idea. And too often, I turn those dumb ideas into little web toys. About five years ago, I discovered , which--in my limited understanding--is a deterministic way to guess what would come after some input. A bit like autocomplete. For example, if I type "I like,” I'm more likely to type "cats" after that than "yard work.” It's fairly complex (see the Wikipedia link above for more details) and perhaps a bit like GenAI. For me, I just think it's neat. Markov chains tiny Five years ago, I took the excellent npm package, a list of Cure songs, and built a generator for... well, Cure songs: . I was thinking about this post and wondered if I could do something with horror movies. Why horror movies? Maybe because it's 200 degrees here, and I'm pining for October and Halloween. Either way, I built it! And here's how I did it. titlegen Generating Random Cure Song Titles with Markov Chain really Getting the Data In order for my demo to work, I needed data for the Markov chain. For my data, I used the simple . This gives you access to loads of movie and TV data, and while I've seen folks use it before, this was the first time I tried it, and honestly, I was really impressed with it. To get my data, I hit their endpoint with these arguments: really TMDB API discover The horror genre (27), per the docs. No video (I believe that means direct-to-video) or adult movies. Original language as English The API can only return 20 results per call but supports paging. I decided to use again because I'm enjoying the platform and knew I could get something up super quick. I want to point out that I'm also using , and Glitch absolutely supports server-side code. I just felt more comfortable doing my server-side code in Cloudflare and my front-end in Glitch. Cloudflare Workers Glitch Ok, so here's the entire worker: // Horror!
const GENRE = '27';
// cache time of 6 hours
const CACHE = 60 * 60 * 6;

async function getHorrorMovies(key,page=1) {

	let resp = await fetch(`https://api.themoviedb.org/3/discover/movie?include_adult=false&include_video=false&with_genres=${GENRE}&with_original_language=en&page=${page}`, {
		headers: {
			'Authorization':`Bearer ${key}` 
		}
	});
  
  return (await resp.json()).results;
  
}

export default {
	async fetch(request, env, ctx) {

		const APIKEY = env.MOVIEAPI;

		let titles = await env.horrormovies.get('horrormovies');
		if(!titles) {
			let horrorMovies = [];
			const totalPages = 20;
			for(let i=0;i<totalPages;i++) {
			let movies = await getHorrorMovies(APIKEY,i+1);
			horrorMovies = [...horrorMovies, ...movies]
			}
			console.log(`Fetched ${horrorMovies.length} horror movies.`);
			titles = horrorMovies.map(m => m.title);
			await env.horrormovies.put('horrormovies', JSON.stringify(titles), { expirationTtl: CACHE });

		} else titles = JSON.parse(titles);

		return new Response(JSON.stringify(titles), {
			headers: {
				'Access-Control-Allow-Origin': '*',
				'Access-Control-Allow-Methods': 'GET,HEAD,POST,OPTIONS',
				'Content-Type':'application/json;charset=UTF-8'
			}
		});

	},
}; On top, I've got a function that wraps calls to the TMDB API. My worker hits the KV cache (something I'll be blogging about more on Monday, for now, just think of it as a simple key/value caching system) if possible, and if not, grabs 400 movies from the API, filtering out to the just the titles. I cache for 6 hours so the worker can return quicker. Finally, I return the data along with CORS headers so I can use it from another server. You can hit this endpoint here: https://horrormovies.raymondcamden.workers.dev/ Presenting the Data For the front end, I used . I set up my HTML, JavaScript, and CSS there. You can view all the code at the , but I'll focus on the JavaScript. Glitch project On page load, I fetch my titles from the Cloudflare Worker and then use titlegen to initialize the ability to generate titles. As I said, their utility package is super simple. Here's the entire JavaScript file: let $title, generator, $regenBtn;

document.addEventListener('DOMContentLoaded', async () => {
  
  let titlesReq = await fetch('https://horrormovies.raymondcamden.workers.dev/');
  let titles = await titlesReq.json();

  generator = titlegen.create();
  generator.feed(titles);

  $title = document.querySelector('#title');
  $regenBtn = document.querySelector('#regenBtn');

  doTitle();
  
  $regenBtn.addEventListener('click', doTitle);
});

function doTitle() {
  $title.innerText = generator.next();  
} As you can see, it's one line to initialize titlegen, one to input the data, and then just running to get a new title. next() And that's literally it. Play with the full version here: https://glitch.com/edit/?embedable=true#!/random-horror-movie?previewSize=100&attributionHidden=false&sidebarCollapsed=false&path=README.md&previewFirst=false Obviously, it doesn't always work, but sometimes the "silly" results are funny as hell: Final Destination 5: The Addams Family 2 You Should Have Eyes H.P. Lovecraft's Dracula I hope you've enjoyed this 100% useless bit of code today! Note that the font I used, while excellent, apparently doesn't support numbers.

I Know What You Did Last Summer (With Glitch and Cloudflare)

If you have ever been on a job hunt for a software developer position, you would have come across the so-called LeetCode style interviews. Despite the fact that most of us don’t have to at our actual job, that is how most coding/technical interviews are conducted at big tech companies like Google and Microsoft. Yes, even at the likes of (except for Stripe, because they are cool) and startups. invert a Binary Tree unicorns In this post, I’ll be writing about the thought process of how I came about building and deploying a very simple JavaScript app for free with . If you simply want to deploy and use the app on your own, check out the repository . Cloudflare Worker here TL;DR How to get the Daily LeetCoding Challenge question using LeetCode API Creating a Todoist task using Todoist API Schedule our app to run daily using Cloudflare Worker ScheduledEvent API How to test a Cloudflare Worker Cron trigger locally with Miniflare Problem Statement For a year now, I have been trying to make it a habit to solve the Daily LeetCoding Challenge (which I’m still struggling with). As I am using as my main productivity tool of choice, I have a daily task that looks just like this: Todoist As a lazy person, having to check every time I want to practice is too much of a hassle. So then I thought, why not just sync the Daily LeetCoding Challenge to my Todoist every day? leetcode.com Requirements Let’s start by defining what I want the app to do: Get Daily LeetCoding Challenge question Ability to create a new task on my Todoist account Sync new Daily LeetCoding Challenge question on time The app has to sync on time for free every day Let’s start! LeetCode API Like any sane person would do, the first thing I did was to do some research. By research, I meant I started to Google for information. The first thing I did was to immediately Google for “leetcode API”, looking for the official API documentation. Official API documentation To my surprise, there wasn’t any official LeetCode API documentation available. While there is a couple of unofficial LeetCode API repositories on GitHub, I would rather not use any unofficial API due to reliability concerns (poorly maintained, outdated, etc.). The Good Ol’ DevTool inspection The second thing that immediately came to my mind was to inspect the network request being made while visiting the site . https://leetcode.com/problemset/all/ With this, I was able to figure out the exact API called to query for the Daily LeetCoding Challenge — done. Here’s the GraphQL request body: # HTTP POST to https://leetcode.com/graphql
query questionOfToday {
    activeDailyCodingChallengeQuestion {
        date
        userStatus
        link
        question {
            acRate
            difficulty
            freqBar
            frontendQuestionId: questionFrontendId
            isFavor
            paidOnly: isPaidOnly
            status
            title
            titleSlug
            hasVideoSolution
            hasSolution
            topicTags {
                name
                id
                slug
            }
        }
    }
} You can use the command below to try it out yourself: curl curl --request POST \
  --url https://leetcode.com/graphql \
  --header 'Content-Type: application/json' \
  --data '{"query":"query questionOfToday {\n\tactiveDailyCodingChallengeQuestion {\n\t\tdate\n\t\tuserStatus\n\t\tlink\n\t\tquestion {\n\t\t\tacRate\n\t\t\tdifficulty\n\t\t\tfreqBar\n\t\t\tfrontendQuestionId: questionFrontendId\n\t\t\tisFavor\n\t\t\tpaidOnly: isPaidOnly\n\t\t\tstatus\n\t\t\ttitle\n\t\t\ttitleSlug\n\t\t\thasVideoSolution\n\t\t\thasSolution\n\t\t\ttopicTags {\n\t\t\t\tname\n\t\t\t\tid\n\t\t\t\tslug\n\t\t\t}\n\t\t}\n\t}\n}\n","operationName":"questionOfToday"}' Enough talking, let’s start to write some code that does exactly what we went through: // Just some constants
const LEETCODE_API_ENDPOINT = 'https://leetcode.com/graphql'
const DAILY_CODING_CHALLENGE_QUERY = `
query questionOfToday {
	activeDailyCodingChallengeQuestion {
		date
		userStatus
		link
		question {
			acRate
			difficulty
			freqBar
			frontendQuestionId: questionFrontendId
			isFavor
			paidOnly: isPaidOnly
			status
			title
			titleSlug
			hasVideoSolution
			hasSolution
			topicTags {
				name
				id
				slug
			}
		}
	}
}`

// We can pass the JSON response as an object to our createTodoistTask later.
const fetchDailyCodingChallenge = async () => {
    console.log(`Fetching daily coding challenge from LeetCode API.`)

    const init = {
        method: 'POST',
        headers: { 'Content-Type': 'application/json' },
        body: JSON.stringify({ query: DAILY_CODING_CHALLENGE_QUERY }),
    }

    const response = await fetch(LEETCODE_API_ENDPOINT, init)
    return response.json()
} Task “Get Daily LeetCoding Challenge question” — checked. Todoist API Like what we did in the previous step, I was able to find the for Todoist. Typically, the first section that I always look for in API documentation is the section, especially when you want to perform create/update/delete operations on an app. official API documentation Authorization In short, authorization was pretty straightforward for Todoist: Get your API token Whenever you make a request, attach to your HTTP request header Authorization: Bearer xxx-your-todoist-api-token-xxx Here’s an example of what the curl command to create a new task on Todoist would look like: curl --request POST \
 --url 'https://api.todoist.com/rest/v1/tasks?=' \
 --header 'Authorization: Bearer xxx-your-todoist-api-token-xxx' \
 --header 'Content-Type: application/json' \
 --data '{
"content": "Buy a jar of peanut butter",
"due_string": "Today"
}' Code Writing a function that does what we said is relatively easy, it looks something like this: const TODOIST_API_ENDPOINT = "https://api.todoist.com/rest/v1";

// Passing in the `question` object from fetchDailyCodingChallenge function
const createTodoistTask = async (question) => {
    const questionInfo = question.data.activeDailyCodingChallengeQuestion;

    const questionTitle = questionInfo.question.title;
    const questionDifficulty = questionInfo.question.difficulty;
    const questionLink = `https://leetcode.com${questionInfo.link}`;

    console.log(`Creating Todoist task with title ${questionTitle}.`);

    const body = {
        content: `[${questionTitle}](${questionLink})`,
        description: `Difficulty: ${questionDifficulty}`,
        due_string: "Today",
        priority: 4,
    };

    const init = {
        method: "POST",
        body: JSON.stringify(body),
        headers: {
            "Content-Type": "application/json",
            Authorization: `Bearer ${TODOIST_API_TOKEN}`, // Set at environment variable
        },
    };

    const response = await fetch(`${TODOIST_API_ENDPOINT}/tasks`, init);
    return response.json();
}; Task “Create a new task on my Todoist account” — done. Cloudflare Worker And we’re down to our one final task — running/automating the 2 tasks above every day, for free. The first thing that came to my mind was Cron job. So, I immediately started looking for free solutions on the Internet. After spending a couple of hours doing some homework, I came across Cloudflare Worker, and I figured to give them a try. It runs on V8 JavaScript, not Node.js This is probably one of the most common misconceptions . As the worker’s environment is not in Node.js, a lot of packages (e.g. ) that are running on Node.js would simply not work. about Cloudflare Worker npm install some-node-package Tip: Check out the supported packages and libraries . here Lucky for us, we only need to use the JavaScript built-in API. fetch More code Starting a Cloudflare Worker project is dead simple ( ), basically: reference Install the Wrangler CLI using npm install -g @cloudflare/wrangler Run wrangler generate <your-project-name> <worker-template> The entry point is function. For our use case, we will be using the API where we simply have to change our event from to addEventListener ScheduledEvent "fetch" "scheduled" Let’s stitch everything together: // Move the constants to const.js

const syncLeetCodeCodingChallenge = async (event) => {
    const question = await fetchDailyCodingChallenge();
    await createTodoistTask(question);
};

addEventListener("scheduled", (event) => {
    // Change 'fetch' to 'scheduled'
    event.waitUntil(syncLeetCodeCodingChallenge(event));
}); Next, we would simply need to modify the wrangler.toml as below: name = "<your-project-name>"
type = "webpack"

...

[triggers]
crons = ["1 0 * * *"] With the setting above, the worker will run every day at and sync the Daily LeetCoding Challenge to your Todoist. 00:01 UTC That’s all! Moving on to testing. How to test Cloudflare workers locally In order to try out the Cron triggers locally, we would need to install the . After installation, you may run your worker using: Miniflare CLI # At terminal 1
miniflare

# At terminal 2
curl "http://localhost:8787/.mf/scheduled" If you see a new task is created on your Todoist, you have made it! Deployment No side project is ever done without hosting it. To deploy the app on your own immediately, check out the and use the button. If you’re interested in the know-how: project repository “Deploy with Workers” Create a Cloudflare account. Add using . You may find the newly added secret under 'Cloudflare Worker' → 'Settings' → 'Variables'. You can get your Todoist API token . TODOIST_API_TOKEN wrangler secret put TODOIST_API_TOKEN here Optional: This is only required for . Add into your GitHub repository secrets. You can create your API token from using the template. Wrangler actions CF_API_TOKEN https://dash.cloudflare.com/profile/api-tokens Edit Cloudflare Workers Finally, to publish any new changes to your Cloudflare Worker, run wrangler publish And we are finally done! Closing Thoughts Finally, there’s a lot more that we could have done, e.g.: Handling unanswered questions from previous days Making the app configurable/customizable for users Add tags to our task Allowing users to create a random LeetCode question as a task based on question tag I’m going to leave these features out for another day. While there is a lot of hate on coding interviews as such, I personally look at it this way — by learning some brain teasers, you probably get paid a lot more, so why not? It is really a matter of perspective. If you happen to enjoy doing them, that is even better. Personally, I don’t find as much joy doing LeetCode questions. Rather, I work on LeetCode problems as if I am lifting weights at the gym. While I don’t enjoy lifting weights, I do like reaping the benefits of it. That’s all for today. Let me know what are you building with Cloudflare Worker!

How to Keep Track of Daily LeetCode Challenges With Todoist and Cloudflare Worker

Cloudflare, Inc. (NYSE: NET), the security, performance, and reliability company helping to build a better Internet, today announced it will help power an Internet with zero carbon emissions. Now, not only will one of the world’s largest and most interconnected networks operate on 100 percent renewable energy, but Cloudflare will also remove all historical carbon its global network has emitted since its founding by 2025. The Internet is responsible for approximately 1 billion tons of greenhouse gas emissions per year—roughly comparable to the annual CO2 emissions of the entire aviation industry worldwide. Now any Internet property on Cloudflare’s network—which today already consists of 17 percent of the web—will automatically reduce emissions and seamlessly contribute to a greener, more sustainable world. Today, Cloudflare is also releasing sustainable edge computing services and energy-efficient features to help organisations of any size build sustainability directly into their own applications, websites, and networks. Now businesses can choose to intelligently route traffic through data centres powered by renewable energy using Green Compute on Cloudflare Workers. Customers can also understand and quantify their individual footprints and approximate savings from Cloudflare’s network in comparison to Internet averages with the Carbon Impact Report. Furthermore, organisations that deploy websites on Cloudflare Pages will automatically operate on renewable energy and receive a green certification from The Green Web Foundation displayed through a verified badge. “Internet usage has skyrocketed in the last couple of years, growing over 80 percent on our network, and is showing no signs of slowing down. ,” said Matthew Prince, co-founder and CEO of Cloudflare. For every search you do, online order you make, and video call you join, there is an environmental impact from the physical data centres, servers, and other physical infrastructure the Internet relies on “We want to give the Internet a clean slate to help build toward a future where it’s easy for anyone to create, build, and run their applications on green energy while remaining secure, fast, and reliable.” As Internet usage continues to rise, so does the energy that’s required to deliver online services. The electricity used to power the Internet’s infrastructure helps move content that sits in faraway servers, passes it through networks, and allows for it to land on a user’s device. While turning to cloud platforms to deliver a service or document is more environmentally efficient than paper printouts, for example, each Internet-powered property has a climate impact that can be mitigated. Organisations need the insights, tools, and control to lower their supply chain carbon emissions without sacrificing the security, speed, or reliability of their applications and corporate networks. Cloudflare is providing environmentally efficient solutions, giving organisations a seamless way to: : Using Cloudflare’s distributed global network to drive a better climate outcome, businesses and developers can elect to route a portion of their compute workloads through facilities on Cloudflare’s edge powered by 100 percent renewable energy with Green Compute on Cloudflare Workers. Power compute with renewable energy 2. : The millions of customers powered by Cloudflare can effortlessly track and view their approximate carbon savings from their usage of Cloudflare services in comparison to Internet averages for the same volumes. With the Carbon Impact Report, all customers can view customised data on their individual footprints on Cloudflare’s network to quantify, report on, and set sustainability goals, as well as view recommendations on actions they can take to reduce their carbon impact. Quantify their individual footprint and carbon savings from Cloudflare’s network 3. : All websites are automatically powered by 100 percent renewable energy with Cloudflare Pages, now in partnership with The Green Web Foundation. Organisations and developers default to deploy certified green web pages and can elect to display a verified badge. Receive green certification on their websites by default 4. : Search engines will be notified by Cloudflare when content on a page has changed, to eliminate unnecessary crawls and energy. This option will be available in the future for all Cloudflare customers, to help deliver fresher indexes while wasting fewer resources from search engines. Reduce unnecessary energy from search engines indexing the web Efficiency has been at the core of Cloudflare’s network from its founding. In contrast to legacy models that transport certain activity to specific data centres halfway across the world, each Cloudflare data centre delivers full functions of security, performance, and reliability to intelligently serve users based on location. The coupling of comprehensive services and Cloudflare’s network edge—spanning more than 200 cities operating within milliseconds of the Internet-connected population—simultaneously reduces energy to transport while providing optimal speed and security. Cloudflare continues to iterate on the optimal efficiency of its network, and has achieved more than 50 percent improvement in requests per watt by deploying Arm® Neoverse™-based processors in its edge servers. Today, the company is furthering its efficiencies with renewable energy-powered edge computing and services for its customers to build and scale sustainability. “The industry is facing a massive opportunity to accommodate the increasing demand on cloud computing in a sustainable way,” said Chris Bergey, Senior Vice President and General Manager, Infrastructure Line of Business, Arm. “Energy efficiency is part of Arm’s DNA, and we are pleased to see Cloudflare leverage the performance-per-watt benefits of Neoverse as they continue to build on and prioritise the overall efficiencies of its vast global network.” “Cloudflare’s recent climate disclosures and commitments are encouraging, especially given how much traffic flows through their network. Every provider should be at least this transparent when it comes to accounting for the environmental impact of their services”, said Chris Adams, Co-director of The Green Web Foundation. “We see a growing number of users relying on CDNs to host their sites, and they are often confused when their sites no longer show as green, because they’re not using a green CDN. It’s good to see another more sustainable option available to users, and one that is independently verified.”

Cloudflare Helps Reduce The Environmental Impact of Internet Carbon Emissions

Adult movie company AMA Multimedia has obtained a DMCA subpoena from a Washington court to help it track down individuals who uploaded content to various 'tube' sites. The subpoena orders Cloudflare to hand over the identities of uploaders and potentially site operators too but given the way the content seems to be delivered, it remains a question whether the former will be possible. Before taking direct legal action against alleged copyright infringers, it helps if the identities of those people are known to the potential plaintiff. One method to obtain this information is to file an application for a DMCA subpoena. Commonly filed against domain registries and Cloudflare, DMCA subpoenas can require such companies to give up the names of their allegedly-infringing customers, who are often the operators of ‘pirate’ sites. The process of obtaining a subpoena is attractive and relatively easy since the applications are rarely subjected to much scrutiny and can yield useful results. Last week adult company AMA Multimedia (better known for its Casting Couch X and ) filed an application at a Washington court demanding that Cloudflare provide identifying information of customers said to have infringed the company’s copyrights. various other brands According to AMA, it previously asked Cloudflare to remove or disable access to around three dozen URLs, mostly JPG images and direct content links, on domains including the 12 million visits per month Pornmilo.com and the 15 million visits per month HLSMP4.com. With that content apparently still intact, AMA asked the court for permission to demand information from Cloudflare to identify the alleged infringers. “For the period January 1, 2016 through the present, produce all documents and account records that identify the person(s) or entities that caused the infringement of the material described in the attached Exhibit B DMCA notifications to the DMCA Agent for Cloudflare, Inc. and/or who unlawfully uploaded AMA Multimedia LLC’s copyrighted works at the URLs listed in the notifications, including but not limited to identification by names, email addresses, IP addresses, user history, posting history, physical addresses, telephone numbers, and any other identifying information,” the subpoena to Cloudflare reads. In respect of the phrase “person(s) or entities that caused the infringement”, that could mean the operators of the various listed domains – pornmilo.com, javbeautiful.com, 3fu.xyz, 4fu.xyz, hlsmp4.com, o0-1.com, o0-2.com, o0-3.com, o0-4.com, and o0-5.com. However, when it comes to identifying the underlying infringers, that could be more tricky. When one visits Pornmilo.com, the platform gives the initial impression of being a YouTube-like site, presumably one that hosts its own content. On closer inspection, however, the site claims not to host any video content at all. Indeed, it appears that the videos are embedded having been supplied by Fembed, a service that advertises itself as an “All-in-one Video Platform Designed by webmasters, for webmasters.” Essentially, people can host their video files and serve them on another site, with or without revenue-generating advertising. on Fembed Fembed.com isn’t mentioned in the DMCA subpoena but it appears to be connected to HLSMP4.com, which is mentioned multiple times. Furthermore, javbeautiful.com, 3fu.xyz, 4fu.xyz and indeed all the other domains redirect to Fembed.com, so it’s possible that they have the same owners. AMA seem pretty keen to find out exactly who they are. That being said, it is far from clear how Cloudflare itself can establish who uploaded the infringing content on HLSMP4, Fembed, and the other sites so it can hand that information to AMA. At this early stage that may not concern AMA too much and it’s possible that outcome is already being anticipated. Nevertheless, the DMCA subpoena has the ability to get closer to the targets in a cheap and relatively easy fashion. The DMCA subpoena documents can be found here and here (pdf, NSFW) Originally published as “ ‘Casting Couch’ Movie Company Orders Cloudflare to Unmask Tube Site Pirates ” with the Attribution-NonCommercial 3.0 Unported (CC BY-NC 3.0) license

The Subpoena Orders Cloudflare To Hand Over The Identities of Uploaders 

CDN company Cloudflare is displaying an extremely rare 'Error 451' to German visitors who attempt to access a music piracy site. The message currently affecting DDL-Music.to states that the site has been rendered "Unavailable for Legal Reasons'. Contrary to Cloudflare's own error code guide, no explanatory legal demand specifics have been published. As one of the most important infrastructure companies on the entire Internet, Cloudflare often finds itself embroiled in copyright disputes of other entities’ making. Along with millions of other ‘normal’ websites, ‘pirate’ sites often use Cloudflare’s services. The CDN company has a stated aim of remaining a neutral intermediary but entertainment industry groups see the company as helping to facilitate piracy by assisting sites to more efficiently stay online while rendering their true locations harder to find. While sites like The Pirate Bay, for example, are using Cloudflare’s services trouble-free at this very moment, the same cannot be said of Germany-focused music piracy site DDL-Music.to. This morning, informed TorrentFreak that visitors to the platform utilizing German IP addresses were being blocked by Cloudflare. While these kinds of claims are relatively common given the sometimes unreliable nature of pirate sites, in this case clarity is being provided by the CDN company itself, which clearly states that it has taken action. Tarnkappe Tests carried out for TF using the German-based IP address listed above returned a rare “Error HTTP 451” which, as the image shows, is deployed when a site is “Unavailable For Legal Reasons.” Tests carried out by us from IP addresses elsewhere in Europe accessed the site without issue, perhaps suggesting that any “legal reason” may be specific to German territory. This is the first time that we’ve seen an Error HTTP 451 displayed by Cloudflare when attempting to access a pirate site. The special HTTP status code by the Internet Engineering Task Force (IETF) back in 2015 as a more informative alternative to the “403 Forbidden” code utilized by some ISPs engaged in blocking. was approved “A really good Error 451 message would tell their customers how to challenge a block, how long the block’s expected to last, where the relevant legal documents are and which legal authority imposed the blocking order,” the 451 Unavailable project at the time. wrote A nod towards providing this additional information is actually mentioned in Cloudflare’s own 4XX error code documentation that was updated only this month. “Server is unable to deliver the resource due to legal actions,” the against Error 451 read. “The response should include an explanation in the response body with details of the legal demand.” notes As the above screenshot of the Cloudflare error shows, no explanation for the blocking is provided. Why that is the case is currently unknown since Cloudflare has yet to respond to our request for comment. However, DDL-Music’s status as an obvious pirate site (and one with a history of being targeted by copyright groups) leads us towards a theory that Cloudflare may have been placed under pressure. With that in mind, it’s perhaps helpful to look at the recent history surrounding ISP blockades of pirate sites in Germany and how they have played out, mostly without direct intervention from the courts. In March 2019, it had implemented a blockade of pirate platform Boerse.to following a notification from local music rights group GEMA, an action duplicated by its service provider 1&1. Around the same time, Vodafone also began blocking DDL-Music. Kabel Deutschland is also known to block access to the music platform. Vodafone told TorrentFreak The companies believe that established by the Federal Court of Justice require them to do so, without a specific court order. principles On the same basis, Vodafone previously blocked access to several other ‘pirate’ sites including , and . S.to, BS.to Libgen Kinox.to Whether Cloudflare is following a similar pattern of behavior with its restriction of DDL-Music is currently unknown but we will update this article as soon as the company responds. Originally published as “ Cloudflare Blocks Access to Pirate Site For “Legal Reasons”, Displays Rare 451 Error ” with the Attribution-NonCommercial 3.0 Unported (CC BY-NC 3.0) license

Stories, News & Mentions about CLOUDFLARE

Cloudflare’s Stories on HackerNoon