CloudFlare is building a faster, safer web from the ground up

Cloudflare

Your web scraper has just been blocked by Cloudflare again! You've looked online for solutions, but nothing seems to do the trick. Is there a secret sauce for Cloudflare bypass? You bet there is. You just haven't stumbled upon the right recipe! Embark on a guided tour through the maze of Cloudflare anti-bot technology. We're here to demystify Cloudflare's inner workings, unveil some game-changing tricks, and ensure you never hit a dead end again! What Is Cloudflare? , the cyber fortress of the Internet, offers top-notch services to boost the security and performance of websites and web applications. But you probably knew that already! Cloudflare Now, what's the secret sauce that makes Cloudflare so popular? Drumroll, please—it’s anti-bot solutions! Ever come across that page? Well, you’re the only one! That’s a result of the , a high-tech shield against the mischief of malicious bots. Cloudflare Bot Management service Here are the technologies that power that cybersecurity platform: Behavioral analysis: Picture it as Sherlock Holmes scrutinizing user behavior on the Web to decipher whether users are humans or not. Machine learning magic: Think of it as the Dumbledore's Army of bot detection, learning from a colossal dataset with billions of records to give every request a reliable bot score. Fingerprinting: Use millions of internet properties to precisely identify bot requests without compromising user privacy. The result? Bots, begone! From stopping credential stuffing to thwarting content scrapers, spammers, inventory hoarders, credit card stuffers, and even putting up a solid defense against application DDoS attacks—Cloudflare's got it covered. Cloudflare’s Bot Management stands tall as the Batman of bot management, vigilantly safeguarding websites to thwart any hint of malicious bot behavior. 🦸‍♂️ But every superhero has its kryptonite! Dive into the next section to unravel more. Tips to Bypass Cloudflare Before digging into this chapter, let's tackle a burning question. Is it even possible to bypass Cloudflare? Absolutely! No online solutions can or should restrict users from accessing public sites as they wish, even through automated software. Thus, the million-dollar question is: how to outsmart Cloudflare? 🤔 Dive into the tips below to master the art of eluding Cloudflare! Use a Reliable IP Cloudflare gives IPs a score based on their reputation. If your IP has been caught red-handed performing actions considered malicious by Cloudflare, the chances of getting blocked become high. The solution? Having a large pool of IPs to distribute your requests on. In that scenario, even if one IP gets blocked, you have many others available in your arsenal. That’s where IP rotation via proxy comes into play. Find out more on . how to rotate an IP address Leverage Browser Automation Only browsers can overcome the challenges set by Cloudflare’s Bot Management. When you navigate a page shielded by that solution, your browser uses JavaScript to tackle those challenges without you even realizing it. Requests made by an automated script through a traditional HTTP client will then hit an immediate roadblock. Why? Because HTTP clients can’t execute JavaScript! That’s something only a browser can do. Thus, your Cloudflare-ready automated script should rely on an approach for programmatically executing actions on a web page in a browser, such as . controlling a headless browser via Playwright Reverse Engineering the Cloudflare Challenges This is where the real challenge kicks in—how to outsmart the anti-bot defenses intricately embedded into a web page by Cloudflare? The answer: dive deep into their workings, unravel the puzzle through reverse engineering, and conquer. That's easier said than done 😅 Step one involves finding the request linked to the retrieval of the JavaScript challenge. This will involve encrypted code that you must somehow decipher: Suppose you managed to do that. Now, all that remains is to figure out the original code's intentions and create a strategy to fool it into thinking that the Cloudflare challenge has been overcome. On paper, this sounds like a valid approach, but in the real world, it may be a one-way ticket to a colossal headache. Don’t worry, though. Overcoming Cloudflare is possible—you just need the right tools! My Solution Works… Until It Doesn't Imagine you're an IT genius who manages to crack the code to get past Cloudflare's bot protection technologies. Bear in mind that Cloudflare is an ever-evolving beast, so no guarantees your solution will work tomorrow. To reach the goal, you've got to embrace a different approach! A real-world, ideal, intelligent solution to overcome Cloudflare would be a controllable browser that: Can overcome advanced anti-bot measures for you. Is always up-to-date because it runs in the cloud. Automatically rotates IPs and handles request attempts. Avoids fingerprinting to improve your privacy. Can resolve CAPTCHAs on its own. Allows cookie management and header customization. Ensures amazing scalability and performance by executing in the cloud. It can be controlled by any browser automation library, such as Selenium, Playwright, or Puppeteer. This isn't a mere dream, but a palpable reality. The browser you're imagining is real, developed by Bright Data, and goes by the name of ! Scraping Browser Final Thoughts Cloudflare reigns supreme in safeguarding websites against malicious bots. Here, we've delved into the technologies it uses for detecting and stopping automated software, offering you some handy tips to avoid them. But let's be real, that’s a grueling, trial-and-error, demoralizing task. Need to bypass Cloudflare once and for all? Embrace Scraping Browser from and join our mission to transform the Internet into a public space accessible to everyone, everywhere, even via automated scripts! Bright Data Until next time, keep exploring the Internet with freedom!

Cracking the Code of Cloudflare Bypass

ক্লাউডফ্লেয়ার বাইপাসের কোড ক্র্যাক করা

Den Code von Cloudflare Bypass knacken

Descifrando el código de omisión de Cloudflare

Déchiffrer le code du contournement de Cloudflare

क्लाउडफ्लेयर बाईपास के कोड को क्रैक करना

Cloudflareバイパスのコードを解読する

Cloudflare Bypass 코드 크래킹

Quebrando o código do Cloudflare Bypass

Взлом кода обхода Cloudflare

Cloudflare Bypass Kodunu Kırmak

Bẻ khóa mã bỏ qua Cloudflare

破解 Cloudflare 绕过代码

I can't go a day (or two) without building a bot of some sort, and last week was no exception. I've been a fan of the for nearly a decade now, and one of my favorite examples of it is my bot. I thought I'd use the Marvel API as a way to build bot, but this time on the platform. Here's how I did it. Marvel API random comic book cover another Cloudflare Workers The Architecture So, obviously, I'm using Cloudflare Workers, but I decided to make this project a two-step process. In my , I shared how you can "connect" one Worker to another via service bindings. Obviously, this is practical for reuse, and while it may be a bit overkill for this, I wanted to put what I learned in the last post into practice. My project's two workers are: last post A scheduled Worker that makes use of their support. This worker will be run on a schedule, call the second Worker for its data, and then post to Mastodon. Cron triggers A Worker that wraps calls to the Marvel API, specifically one to get a random character. The latter of the two is the simpler one, so let's start with that. Using the Marvel API for Character Information So, much like my 'random comic book' logic, I get a random Marvel character by: Figure out the total number of characters. I did this using their interactive tester and making note of the total number of results. As of August 2023, that number is 1562. Here's an example of that output: Then, simply call the character endpoint with a limit of one and an offset of a random number in that range. For the most part, it's all relatively simple, except Marvel requires you to sign your API requests. This was a minor sticking point for me as my previous Node.js code didn't work on Cloudflare, and I had to switch to Web Crypto, which was nicely documented here: . https://developers.cloudflare.com/workers/runtime-apis/web-crypto/#web-crypto Here's the entirety of the Worker: // Based on checking the API (in Aug 2023) to see how the max number of characters
const CHAR_TOTAL = 1562;

const getRandomInt = (min, max) => {
    return Math.floor(Math.random() * (max - min + 1)) + min;
}

async function getSuperHero(privateKey, publicKey) {

    let selected = getRandomInt(0, CHAR_TOTAL);
    let url = `https://gateway.marvel.com:443/v1/public/characters?limit=1&apikey=${publicKey}&offset=${selected}`;

    // add hash
    let ts = new Date().getTime();
    let myText = new TextEncoder().encode(ts + privateKey + publicKey);

    let hash = await crypto.subtle.digest({
        name:'MD5'
    }, myText);

    // Credit: https://developers.cloudflare.com/workers/runtime-apis/web-crypto/
    const hexString = [...new Uint8Array(hash)]
      .map(b => b.toString(16).padStart(2, '0'))
      .join('');

    url += '&hash='+encodeURIComponent(hexString)+'&ts='+ts;

    let resp = await fetch(url);
    let data = await resp.json();
    return data.data.results[0];
}

export default {
    async fetch(request, env, ctx) {
        const PRIVATE_KEY = env.MARVEL_PRIVATE_KEY;
        const PUBLIC_KEY = env.MARVEL_PUBLIC_KEY;
        let hero = await getSuperHero(PRIVATE_KEY, PUBLIC_KEY);
        console.log(`I got the hero ${hero.name}`);

        return new Response(JSON.stringify(hero), {
            headers: {
                'Content-Type':'application/json;charset=UTF-8'
            }
        });
    },
}; By the way, notice the ? Later this week, I'll show how that works in production. In case you're curious, here's what that JSON response looks like. It randomly selected one of my favorite characters, Galactus. In order to save on space, I removed items from the arrays of data. console.log many {
    "id": 1009312,
    "name": "Galactus",
    "description": "",
    "modified": "2014-09-30T16:47:03-0400",
    "thumbnail": {
        "path": "http://i.annihil.us/u/prod/marvel/i/mg/5/03/528d31a791308",
        "extension": "jpg"
    },
    "resourceURI": "http://gateway.marvel.com/v1/public/characters/1009312",
    "comics": {
        "available": 236,
        "collectionURI": "http://gateway.marvel.com/v1/public/characters/1009312/comics",
        "items": [
            {
                "resourceURI": "http://gateway.marvel.com/v1/public/comics/12638",
                "name": "Alpha Flight (1983) #10"
            },
            {
                "resourceURI": "http://gateway.marvel.com/v1/public/comics/12639",
                "name": "Alpha Flight (1983) #100"
            },
            {
                "resourceURI": "http://gateway.marvel.com/v1/public/comics/4788",
                "name": "Annihilation (2006) #1"
            },

        ],
        "returned": 20
    },
    "series": {
        "available": 112,
        "collectionURI": "http://gateway.marvel.com/v1/public/characters/1009312/series",
        "items": [
            {
                "resourceURI": "http://gateway.marvel.com/v1/public/series/2116",
                "name": "Alpha Flight (1983 - 1994)"
            },
            {
                "resourceURI": "http://gateway.marvel.com/v1/public/series/3613",
                "name": "Annihilation (2006 - 2007)"
            },
            {
                "resourceURI": "http://gateway.marvel.com/v1/public/series/1864",
                "name": "Annihilation: Heralds of Galactus (2007)"
            }
        ],
        "returned": 20
    },
    "stories": {
        "available": 259,
        "collectionURI": "http://gateway.marvel.com/v1/public/characters/1009312/stories",
        "items": [
            {
                "resourceURI": "http://gateway.marvel.com/v1/public/stories/694",
                "name": "Cover #694",
                "type": "cover"
            },
            {
                "resourceURI": "http://gateway.marvel.com/v1/public/stories/898",
                "name": "Fantastic Four (1998) #520",
                "type": "cover"
            },
            {
                "resourceURI": "http://gateway.marvel.com/v1/public/stories/899",
                "name": "1 of 5 - Galactus",
                "type": "interiorStory"
            },
            {
                "resourceURI": "http://gateway.marvel.com/v1/public/stories/922",
                "name": "Fantastic Four (1998) #518",
                "type": "cover"
            },
            {
                "resourceURI": "http://gateway.marvel.com/v1/public/stories/923",
                "name": "AVENGERS DISASSEMBLED TIE-IN! \"FOURTITUDE\" PART 2 (OF 3) With public opinion of the FF at an all-time low and with all of Manhat",
                "type": "interiorStory"
            }
        ],
        "returned": 20
    },
    "events": {
        "available": 9,
        "collectionURI": "http://gateway.marvel.com/v1/public/characters/1009312/events",
        "items": [
            {
                "resourceURI": "http://gateway.marvel.com/v1/public/events/229",
                "name": "Annihilation"
            },
            {
                "resourceURI": "http://gateway.marvel.com/v1/public/events/234",
                "name": "Avengers Disassembled"
            },
            {
                "resourceURI": "http://gateway.marvel.com/v1/public/events/318",
                "name": "Dark Reign"
            },
            {
                "resourceURI": "http://gateway.marvel.com/v1/public/events/302",
                "name": "Fear Itself"
            }
        ],
        "returned": 9
    },
    "urls": [
        {
            "type": "detail",
            "url": "http://marvel.com/comics/characters/1009312/galactus?utm_campaign=apiRef&utm_source=fe877c0bf61f995fc8540d9eac4704f1"
        },
        {
            "type": "wiki",
            "url": "http://marvel.com/universe/Galactus?utm_campaign=apiRef&utm_source=fe877c0bf61f995fc8540d9eac4704f1"
        },
        {
            "type": "comiclink",
            "url": "http://marvel.com/comics/characters/1009312/galactus?utm_campaign=apiRef&utm_source=fe877c0bf61f995fc8540d9eac4704f1"
        }
    ]
} Notice the property is both a path and an extension. Here's our lovely world devourer. thumbnail Now, on to the next Worker. The Scheduled Tooter I love that subhead. So the next Worker is responsible for running on a schedule and actually doing the Mastodon posting. The cover how these are set up and how you can test them. For the most part, this is just a plain worker. The biggest difference in the code is that you have a handler, not . Here's a barebones Worker for scheduled execution. Cloudflare docs scheduled fetch export default {
    async scheduled(event, env, ctx) {


    },
}; And your schedule is defined in the file. In this case, every two hours: wrangler.toml [triggers]
crons = ["0 */2 * * *"] The first issue I ran into was how to connect this worker to the first one. In the , you'll see it's rather simple: last blog post const backResponse = await env.backlogic.fetch(request.clone()); But, this expects an incoming request, an HTTP-driven Worker. There isn't a request in a Cron-triggered worker. I on the Cloudflare forums and got help from the most Internet nickname ever, . The fix is to simply make an empty (mostly) Request object like so: fetch asked Cyb3r-Jak3 let heroRequest = await env.randomsuperhero.fetch(new Request('http://127.0.0.1'));
let hero = await heroRequest.json(); Next, I prepare my data for my toot: /*
Generate the text for the toot.
I'm using the 'detail' link which is not always the best, better than the wiki though :( 
*/
let toot = `
Your random Marvel superhero of the moment is: ${hero.name}.
More information here: ${hero.urls[0].url}
`;

let image = `${hero.thumbnail.path}.${hero.thumbnail.extension}`; This is where I should point out that while my random comic book cover bot is , the character information is - unfortunately - a bit slim, especially for obscure characters. To be clear, I don't mean a lack of information, but a lot of 404s and no images. Honestly, I almost punted on this as a source of data, but I figured I'd let it go and see how it looks after a while "in the wild." great Now, at this point, I've got the text for my toot, as well as the image URL. I had intended to make use of the npm module I'd used in the past, , however, when I included this in my Worker, I got compatibility errors with the Workers environment. It didn't seem like an easy workaround, and I almost gave up when I thought, why not actually look at their and try using it without a wrapper? mastodon-api API documentation Creating a "toot" was incredibly simple: let data = new FormData();
data.append('status', toot);

let resp = await fetch('https://botsin.space/api/v1/statuses', {
    body:data,
    method:'post',
    headers:{
        'Authorization':`Bearer ${KEY}`
    }
}); Literally ten lines or so of code. I was pleasantly surprised. I then looked into the image aspect. Like Twitter, if you want to associate an image with a toot, you first upload the image, get the ID, and then associate it with the new message. To do this, I needed to first get the bits from the URL on Marvel's side and then send that to Mastodon. In the past, I would have simply saved the image to , but I don't think Cloudflare supports that. Instead, I did everything in memory. This took me the longest time, so hopefully, this code can help others. /tmp async function uploadMedia(url, key) {
    // first, grab the bits of the url
    let imgreq = await fetch(url);
    let blob = new Blob([await imgreq.blob()]);

    let data = new FormData();
    data.append('file', blob);

    let mediaupload = await fetch('https://botsin.space/api/v2/media', {
        body:data,
        method:'post',
        headers:{
            'Authorization':`Bearer ${key}`
        }
    });

    return await mediaupload.json();

} Woot. Okay, with that in play, here's the updated toot code (I love saying toot): let toot = `
Your random Marvel superhero of the moment is: ${hero.name}.
More information here: ${hero.urls[0].url}
`;

let image = `${hero.thumbnail.path}.${hero.thumbnail.extension}`;
let mediaOb = await uploadMedia(image, KEY);

let data = new FormData();
data.append('status', toot);
data.append('media_ids[]', mediaOb.id);

let resp = await fetch('https://botsin.space/api/v1/statuses', {
    body:data,
    method:'post',
    headers:{
        'Authorization':`Bearer ${KEY}`
    }
}); And voila - magic: Random Super Hero @myrandomsuperhero@botsin.space Your random Marvel superhero of the moment is: Alex Power.More information here: http://marvel.com/characters/1387/alex_power?utm_campaign=apiRef&utm_source=fe877c0bf61f995fc8540d9eac4704f1 (UTC) 12:01 PM • August 13, 2023 This is Alex Power, part of Power Pack, a favorite of mine when I was a teenager, and unfortunately, the detail links to a 404. :( All in all, this is kind of par for the course for Cloudflare Workers. I hit a few snags, learn how Cloudflare does things, and then it all just works. I know I said this before, but I'm absolutely digging into the Cloudflare developer experience. If you want to follow the bot, you can find the account at . The source code for generating the random hero may be found here: . The source for the cron/Mastodon Worker may be found here: https://botsin.space/@myrandomsuperhero https://github.com/cfjedimaster/cloudflareworkers-demos/tree/main/randomsuperhero https://github.com/cfjedimaster/cloudflareworkers-demos/tree/main/randomsuperherobot Also published . here

Mastodon: Building a Marvel Character Info Bot with Cloudflare Workers

This article describes how to deploy a static site. We would use (IaC) approach to describe site changes and deploy them automatically with Cloudflare Pages. Infrastructure as Code If you are not familiar with one of the tools, don't worry, all steps are described in the details. You can use this article as a guide for your own static site for 10 minutes and 100% free. Table of Content Description Pre-requirements Tips Quick Start Useful links Conclusion For a quick start, you can jump to the “Quick Start” section. Description The site will be hosted on , and the domain name will be managed by you. The site will be built using the Hugo framework. The whole process will be automated using Cloudflare pages. We will use GitHub as a source of truth for our site. Cloudflare Pages So we describe an infrastructure as code and deploy it automatically; it's a little different from the site constructor approach, there is no user interface (UI) for site management, but it's more flexible and powerful. Just follow these , and you will have your own personal site for free. 5 steps Initialize your site Choose a theme for your site Commit the changes to your repository on GitHub Assign your site to your project on Cloudflare Pages Enjoy your site To update your site, you just need to commit changes to your repository on GitHub. Then Cloudflare Pages will automatically rebuild your site. Pre-requirements GitHub account. If you don't have one, you can create it , it's free. here Cloudflare account. If you don't have one, you can create it , it's free. here Domain name. This step is . optional If you don't have a domain name, you can use a free subdomain from Cloudflare. It's up to you. Your free subdomain will look like this: . yourname.pages.dev TIPS TIP 1 If you would like to make a similar page to my , just grab my repo in 3 steps: kazakov.xyz Fork my repo kksudo/kazakov.xyz Update content on your new site. Please, see the repo structure. You need to update 2 files: - Edit the links to your social networks or remove unused ones. For the main pic, I use , but you can use this setting . To read more about Hugo config.toml, please read . ./config.toml Gravatar avatarurl = "images/avatar.jpg" here - Add a few words about yourself. about.md Follow to step “#4 Cloudflare Pages” TIP 2 To verify your changes, run and open docker run --rm -it -v $(pwd):/src -w /src -p 1313:1313 klakegg/hugo:alpine server -D http://localhost:1313/ TIP 3 You can add a new page by . docker run --rm -it -v $(pwd):/src -w /src klakegg/hugo:alpine new <SECTIONNAME>/<FILENAME>.<FORMAT> TIP 4 To exclude unnecessary dynamic files, update your .gitignore Quick Start Step 1. Initialize Your Site For our site, we will use the Hugo framework. We will use some steps from the to initialize our site. So, to start it, we need to a Hugo binary on your machine, but I prefer a docker to keep my OS clean. Hugo quick start guide install Let's create a root structure for your site : hugo new site business-card docker run --rm -it -v $(pwd):/src -w /src klakegg/hugo:alpine new site business-card Docker options: - remove the container after exit --rm - interactive mode -it - mount the current directory to directory in the container -v $(pwd):/src /src - set the working directory to in the container -w /src /src - latest docker image with Hugo, based on the Alpine Linux klakegg/hugo:alpine Tip for a lazy person like me: smart Check this step in my repository kksudo/kazakov.xyz Step 2.  Chose a Theme for Your Site To find a theme for our site, we will use an . official Hugo hub There are a lot of themes; you can choose any of them. But my favorite option is to discover it via GitHub. Because you can filter repositories by stars, forks, maintaining, etc. For example, is a simple filter. here So, let's start with the theme with more than starts on GitHub. Coder 2k+ Move to the root directory of your site. cd business-card Initialize an empty Git repository in the current directory. Skip it if your site already has a Git repository. git init -b main # Initialize the local directory as a Git repository. Clone the Coder theme into the directory, and add it to your project as a Git submodule. themes git submodule add https://github.com/luizdepra/hugo-coder themes/hugo-coder Update the site configuration file , indicating to use theme. Also, we should add some settings for this theme. Grab these settings from my file. ./config.toml hugo-coder config.toml You can find more information about the theme settings . here Note: If you would like to use a simple theme, without any settings, you can use the theme. It's a default theme for Hugo, URL. Ananke quick start Tip for a lazy person like me: smart You could grab this step from my repository kksudo/kazakov.xyz To verify changes, run and open docker run --rm -it -v $(pwd):/src -w /src -p 1313:1313 klakegg/hugo:alpine server -D http://localhost:1313/ Step 3. Commit the Changes to Your GitHub Repository Let's add some content, and change the default site settings before the first commit. Add an About page docker run --rm -it -v $(pwd):/src -w /src klakegg/hugo:alpine new about.md Add some markdown to the body of the post, but do not change the draft value, code content/about.md ---
title: "About"
date: 2023-04-11T21:14:37Z
draft: true
---

I'm a member of "I want to know everything" and "I'm happy to share what I've already learned" groups.
DevOps with love. Update as described in the theme page. Let's add some menu settings for the About page. Or grab these settings from my file, and then edit. ./config.toml hugo-coder config.toml Push our changes to GitHub origin. git add -A . # Adds the files in the local repository and stages them for commit
git commit -m "First commit" # commit all the files in your project

git remote add origin git@github.com:<owner>/<repo-name>.git
git push -u origin main Note: Read more about GitHub quick start here Tip for a lazy person like me: grab this step from my repository smart kksudo/kazakov.xyz Step 4.  Assign Your Site to Your Project on Cloudflare Pages Create an account on , and log in to your account. Cloudflare Create a new project on , and connect to your GitHub repository. Cloudflare Pages Connect Cloudflare to your GitHub repository. Install Cloudflare Pages. Grand permissions Cloudflare integration in GitHub. Setting up Cloudflare settings for the Hugo site Chose a Hugo framework Add a variable to use on the latest Hugo version, HUGO_VERSION=0.111.3 : If you would like to revert Cloudflare permissions for your repo, you should go to and change settings for Cloudflare Pages. TIP https://github.com/settings/installations So, after Cloudflare will be assigned, deployment will be triggered. And after it, Cloudflare started deployment of your site. By default, Cloudflare will build and deploy all your branches and generate a unique URL for each branch; the main URL will be also available for your default branch. So, Cloudflare shows you a lot of details about your build and deployment process. Step 5.  Enjoy Your Site Open and enjoy :) https://yourname.pages.dev Tip for a lazy person like me: smart Sorry, no tips :) Useful Links Install docker Install Hugo Join to GitHub Deploy a Hugo site CloudFlare Hugo quickstart Coder theme configuration documentation based on this article My page with code for this article GitHub repository Disclaimer, Why Did I Choose This Set of Tools? Why Hugo? Hugo it's the world’s fastest framework for building websites with more than 66 000 stars on GitHub. It's very easy to use, pretty fast, open-source, and has a lot of features. You can find more information about Hugo on the . official website Why GitHub? GitHub is a code hosting platform for version control and collaboration. It lets you and others work together on projects from anywhere. You can find more information about GitHub on the . official website Why Cloudflare? Firstly, I will explain why, I didn't choose . The last time I tried to use GitHub Pages, I faced some incipient problems. Also, I chose Cloudflare Pages because this provider is network focused. GitHub Pages I know Cloudflare as a network infrastructure and security company. I really like their products, like DNS, CDN, Firewall, etc. But I didn't know that they have a product for hosting static sites. I was surprised when I found out about it. So, I decided to use network focused provider for hosting static sites. Conclusion I hope this article was useful for you. If you have any questions, feel free to ask me in the comments. I will be happy to help you. Also, feel free to connect with me . kazakov.xyz P.S.: For this flow, we could use a lot of other approaches, services, and tools; in the section above, I described why I sue these techs. For example, you can use GitHub pages instead of Cloudflare Pages, and use another static site generator instead of Hugo. #contest-tags

Deploy Your Personal Web-Page With Hugo, Cloudflare and GitHub 100% For Free

Introduction You want to shorten the long links generated by your application because the URLs are too long to fit in an SMS. You want to keep scalability in mind and want to handle millions of shortened URLs and millions of users flowing through the links. Most importantly, you want this new service to be as cheap as possible - it can't break the bank. In this article, we will use Cloudflare Workers and KV to create a scalable URL shortener service that you can use to shorten your application links. The Stack is a Serverless platform that allows you to deploy code and
have it available globally with extremely low latency. Cloudflare Workers... Cloudflare Workers is much cheaper than competitors such as Azure Functions and AWS Lambda. ($5/10 million requests) can run code instantly without suffering from cold starts like the competitors. can automatically scale to millions of users - you are only charged per request and not the number of concurrent users. The reason why Cloudflare can do this is that it doesn't rely on the traditional serverless model of using the virtual machine architecture. Instead, it utilizes the Chrome V8 engine with the Service Workers API to run code extremely fast. If you are a nerd like me - will amaze you. reading this Alternatively, you would probably write a simple Express or Flask app, dockerize it, and deploy it through or Kubernetes. docker-compose is a key-value store that is globally distributed. We're going to use it to map from a short link to the original long URL. A URL shortener service is one of the best cases to utilize serverless functions. It does 1 simple job, so you shouldn't waste your time thinking about infrastructure, and most importantly, an expensive bill to pay at the end of the month. Workers KV Alternatively, this might be a mix of a PostgreSQL server for permanent storage and Redis to cache results. The Preparation Sign up for a . Cloudflare Account Install Cloudflare's CLI tool, . Wrangler npm install wrangler --save-dev Authenticate yourself with Wrangler by following instructions on the screen to complete OAuth with Cloudflare. npx wrangler login The Plan We want to expose 2 endpoints for the user - one to create a shortened URL, and the other to serve all shortened URLs, perform 302 redirects if the shortened link is found, and 404 otherwise. The code Step 1: Generate a boilerplate for Cloudflare Workers In a local directory, run . It's going to create the necessary files and folder structure for you to create our URL shortener. You will be prompted to select a few options, to which you will select yes to. npx wrangler init shortener Would you like to use git to manage this Worker? y No package.json found. Would you like to create one? y Would you like to use TypeScript? y Would you like to create a Worker at shortener/src/index.ts? Fetch Handler To further simplify development, since we are going to make more than 1 endpoint, we'll
install which is an Express-like router to automatically parse out the HTTP method, query params, and path params, etc. I also prefer to use to format the code so that it looks consistent. itty-router prettier cd shortener
rm package-lock.json
yarn
yarn add itty-router
yarn add --dev prettier Step 2: Create a KV namespace KV is basically the replacement for the database and cache you were going to use to make your Express URL shortener. The difference is, KV is as easy to use as the browser's localStorage. To create a KV namespace on your Cloudflare account, run the following commands. We'll need to run both because we'll need to reserve one for development as well. npx wrangler kv:namespace create SHORTENER_KV
npx wrangler kv:namespace create SHORTENER_KV --preview Note down the KV ids and edit to look like the following. When you deploy with Wrangler, it will set the correct variables and link the KV namespace to your worker. wrangler.toml name = "shortener"
main = "src/index.ts"
compatibility_date = "2022-08-11"

kv_namespaces = [
  {
    binding = "SHORTENER_KV",
    id = "<Your KV ID>",
    preview_id = "<Your KV preview ID for development>"
  }
]

[vars]
HOST_URL = "http://localhost:8787" Step 3: Make the API to shorten a URL I like to write out the skeleton of what I'm going to implement before actually implementing the logic behind it, as I feel like it will lead to better code structure. In the below snippet, I am creating an that will triage API requests to the POST and GET functions respectively. itty-router Also, I am linking the KV namespace in the environment. Finally, from my experience, it is ideal to add statements here and there to help with debugging in case my Worker goes wrong. console.log import { Router } from "itty-router";
const router = Router();

export interface Env {
  SHORTENER_NAMESPACE: KVNamespace; // the KV namespace we setup in the previous step
  HOST_URL: string;
}

router.post("/", async (request: Request, env: Env) => {
  // TODO: Post request to shorten a URL
});

router.get("/:slug", async (request: Request, env: Env) => {
  // TODO: redirect shortened URL to application URL
});

export default {
  async fetch(
    request: Request,
    env: Env,
    ctx: ExecutionContext
  ): Promise<Response> {
    return router.handle(request, env, ctx).then((res: Response) => {
      console.log("HTTP Response", request.url, res.status);
      return res;
    });
  },
}; Now, we can fill in the logic to create a shortened URL. In the below code snippet, I am generating a random slug that has not been used before and using that as the key to store the long URL that the user has passed in. Finally, I am returning the user a shortened URL that they may use to redirect to their URL. router.post("/", async (request: Request, env: Env) => {
  const json = await request.json();
  const longLink = json?.url;
  // generate a random slug - this is not cryptographically random but works in our case
  let slug = btoa(Math.random() + "").slice(0, 9);
  // generate a new slug if the previous one is already existing
  let existing = await env.SHORTENER_KV.get(slug);
  while (existing) {
    slug = btoa(Math.random() + "").slice(0, 9);
    existing = await env.SHORTENER_KV.get(slug);
  }
  // generate shortened URL and return to user
  await env.SHORTENER_KV.put(slug, longLink);
  return new Response(JSON.stringify({ url: `${env.HOST_URL}/${slug}` }), {
    headers: { "content-type": "application/json" },
  });
}); At this point, we should test our code. To run a dev server, run . Then, you can use Postman to test out the server for now. npm run start Step 4: Handle the URL redirect When the user visits the shortened URL, we would like to fetch the long URL from KV, and provide the user with a 302 redirect response, so that the browser can handle the redirection for us. If the shortened slug cannot be found, we can return a 404 telling them nothing is here. router.get("/:slug", async (request: Request, env: Env) => {
  const slug = request.params.slug;
  const redirectTo = await env.SHORTENER_KV.get(slug);
  if (redirectTo) {
    return Response.redirect(redirectTo, 302);
  }
  return new Response("URL not found", {
    status: 404,
  });
}); Hopefully by now, 8 minutes in, you have a fully working URL shortener that is deployed locally. We'll leave the last 2 minutes for deployment, which is more than enough. The deployment Simply run and Wrangler will handle the rest. By default, Cloudflare Workers will provide you a subdomain but you can also configure your own by going into the Cloudflare Dashboard under Workers. npm run deploy workers.dev - KV Namespaces:
  - SHORTENER_KV: <your KV ID>
- Vars:
  - HOST_URL: "http://localhost:8787"

Published shortener (3.83 sec)
  https://shortener.cho.workers.dev As you can see, the used is still the one we defined at the beginning, using localhost:8787. Remember to change the URL to the published URL, in my case, . (Feel free to check this out!) HOST_URL https://shortener.cho.workers.dev After you modify , run again. wrangler.toml npm run deploy Debugging with Workers One of the hardest things about programming something in a new language or system is debugging. Although can bring you very far into the debugging journey, sometimes, you will encounter issues only visible after deploying to Cloudflare. wrangler dev Luckily, Cloudflare provides a way to tail production logs - as long as you write in your worker code for debugging purposes. After that, you can tail the logs using wrangler. console.log npx wrangler tail --format=pretty Go further This example is meant to showcase the power of a serverless URL shortener. Here are a couple of enhancements I would make to the code, and I will leave it as an exercise to the reader: Input validation for POST request Authenticate via Bearer/Basic token Return a fancy UI if the link is not found. ... (let's not limit creativity here) Source Code With ~50 lines we were able to completely recreate a URL shortener. You can check out all the code used in this tutorial . That's 10 minutes of your life writing (maybe) your first Worker and also (maybe) your first URL shortener. Hope you enjoyed this! here Also published . here

How to Build a Scalable URL Shortener With Cloudflare Workers and KV Under 10 Minutes

We've recently announced that WunderGraph is now fully open source. Today, we'd like to explain how you can leverage our API Developer Framework to add Edge Caching to your GraphQL APIs without locking yourself into a specific vendor. Caching GraphQL on the Edge should be vendor-agnostic. Services like Akamai and GraphCDN / Stellate offer proprietary solutions to solving this problem.
We'll compare the different approaches and their tradeoffs. Why did we create proprietary GraphQL CDN solutions A good question to start with is why we've created proprietary GraphQL CDN solutions in the first place. Most GraphQL implementations ignore how the web works The problem with most GraphQL implementations is that they don't really use the "platform" they are operating on. By platform, I mean the web, or more specifically HTTP and the REST constraints. The web has a lot to offer if you're using it in the right way. If read requests (Queries) were using the GET verb, combined with Cache-Control Headers and ETag, Browsers, CDNs, Cache Servers like Varnish, Nginx, and many other tools could handle Caching out of the box. You wouldn't really need a service that understands GraphQL. However, the reality is that most GraphQL APIs make you send Queries over HTTP POST. Cache-Control Headers and ETags don't make sense in this case, as all participants on the web think we're trying to "manipulate" something. So, we've created Edge Caching solutions that rewrite HTTP POST to GET and are capable of invalidating in very smart ways. By analyzing Queries and Mutations they're able to build up a cache and invalidate Objects as mutations flow through the system. Limitations and Problems of GraphQL CDNs / Edge Caches A GraphQL CDN creates a secondary source of truth As discussed in the constraint by Fielding, intermediaries can be used to implement a shared cache to improve the latency of requests. The problem I see with "smart" GraphQL Edge Caches is that they look into the GraphQL Operations to figure out what to cache and what to invalidate. Some GraphQL CDN implementations even allow you to use their API to invalidate objects. Layered System What sounds very smart creates a huge problem, you're building a second source of truth. Before using the GraphQL CDN, all you've had to do is implement your resolvers. Now, you've got to think about how to invalidate the CDN. Even worse, you're now programming against a single vendor and their implementation, coupling your application to a single service provider. You can't just switch easily from one GraphQL CDN provider to another. Contrary to REST/HTTP APIs, there's no standard on how to Cache GraphQL APIs, hence every implementation will be different. Another issue is that we're creating a secondary source of truth. Imagine we're putting a GraphQL CDN in front of the GraphQL API of GitHub to cache the Issues for a repository. If we're using "smart" Cache invalidation using Mutations, we're not able to update the cache if a user is bypassing our CDN and uses the GitHub API directly. A GraphQL CDN really only works if 100% of the traffic flows through the system. Your GraphQL Edge Cache won't work on localhost or in your CI/CD Testing takes up a huge part of software development, and we definitely want to have a great Developer Experience when building our APIs. Using a proprietary, cloud-only, Caching solution means that we're not able to test it locally or from within our Continuous Integration systems. You'll be developing on your local machine without the CDN, only to find out that something behaves weirdly when you're using the CDN in production. If we were using standardized caching directives, we'd be able to use any Cache Server, like Nginx or Varnish within our CI/CD pipeline to test our APIs. Proprietary GraphQL CDNs impose a vendor lock-in problem As mentioned earlier, there's no specification for how GraphQL Caching should work. Every implementation is different, so you're always tied to a specific vendor. They might get broke, they might get acquired, they might cancel their service, and they might change their pricing model. In any case, it's a risk that needs to be managed. A GraphQL CDN can introduce cross-origin requests Depending on the setup, adding a GraphQL CDN to your architecture could mean that your browser-based applications have to make cross-origin requests. That is, when your application is running on and your GraphQL CDN runs on , the browser will always make an extra Preflight request. It's possible to cache the Preflight request, but this still means that we'd have to do at least one extra request on the initial page load. example.com example.cdn.com A GraphQL CDN might not work well for authenticated Requests with Server Side Rendering (SSR) Let's say your application requires your users to be logged in, but you still want to be able to apply to cache. Aside from that, you'd also like to implement Server-Side Rendering (SSR). In the ideal scenario, you'd have your users log into your authentication server, which sets a cookie on your apex domain so that you're logged in on all subdomains. If your CDN is running on a different domain, it's going to be impossible to server-side render a page as the browser will not send the user's cookie to the CDN domain. Luckily, some providers offer custom domains, so cookie-based authentication might still work for you. Invalidating Deeply Nested GraphQL Operations might not work Here's an example of a GraphQL Operation that invalidates easily. mutation UpdateProfile {
    updateProfile(update: {id: 1, name: "Jannik"}) {
        id
        name
        friendsCount
    }
} If you've previously Queried the user with id 1, you can invalidate all records with that id and invalidate the following Query: query {
    profile(id: 1){
        id
        name
        friendsCount
    }
} Let's make it a bit more complex and query for all your friends: query {
    me {
       friends {
            id
            name
        }
    }
} Now, you made some good friends at the last conference. Let's add them: mutation AddFriend {
    addFriends(where: {user:{hasConnection: {conferences: {id: {eq: 7}}}}}){
        id
        name
    }
} We've now added all users as friends that had visited a conference whose id equals 7. At this point, we don't get back all the data when we query again for all your friends because the cache can't know that the mutation has invalidated the cache for the query. addFriends friends At this point, you've got to start adding response tagging, surrogate keys, or analyzing the GraphQL return types to make your invalidation strategy "smarter". We'll pick up this topic again after the benefits. Benefits of using a GraphQL CDN / Edge Cache When there's a cost, there are also benefits! Using a GraphQL CDN means, you don't have to change much of your application. In an ideal scenario, you simply change the URL of the GraphQL Server and everything should work. You also don't have to deploy and configure any tooling. You're buying a ready-to-use service that just works. Despite the problems discussed earlier, a GraphQL CDN can probably improve the performance of many applications. When not to use a GraphQL CDN As discussed in the section on Cache Invalidation, building a smart CDN service that doesn't know anything about your GraphQL Backend is actually extremely hard. The problem lies in the fact that the Backend is the source of truth, but doesn't share all this information with the Cache. In fact, if you're running into such issues where invalidation becomes hard, you might want to apply caching at a different level, inside the resolvers, or even one level below, at the entity level, using the DataLoader pattern. Additionally, if your data is expected to change frequently, it might make not much sense for you to cache at the Edge. Another big underrated flaw is that the invalidation of a distributed cache is eventually consistent. This means you serve stale content after a mutation for a short period of time. If you don't respect that in your architecture it has the potential to break the business logic of clients. Client (US) -> Query all posts to fill the cache
Client (US) -> Run mutation to update the post with ID:1
System fires a web-hook to a third-party service in FRA
Client (FRA) -> Query all posts -> stale content This is not GraphQL specific but with HTTP Caching the semantics are better understood. In summary, don't use a GraphQL CDN if you need write-after-read consistency. If your requirement is to support write-after-read consistency, there are multiple solutions to the problem. One is to not cache at the Edge, but rather at the application layer. In this case, you'd trade latency for consistency, which can be a good trade. Another way of solving the problem is by distributing the state across the edge and sharding the data based on location. This model of sharding is not always possible, but if the shared state is only used across groups of users in the same location, this solution could work very well. One example of this is Cloudflare Workers + Durable Objects, which gives you a simple Key-Value store that is persisted in a specific location, meaning that all users close to that one location can have a consistent state at low latency. When a GraphQL Edge Cache makes the most sense If you've got a single GraphQL API, and this API is the only API your frontend is talking to, you fully own this API and no traffic is bypassing your Cache, then such a CDN might actually make sense. Otherwise, I doubt you get the results you're expecting, especially not with the extra costs discussed earlier. WunderGraph - An Alternative Approach to GraphQL Edge Caching without vendor lock-in We've discussed the pros and cons of GraphQL CDNs, now I'd like to propose another approach that makes you stay vendor independent. When it comes to solving problems, sometimes it's smart to be dumb. I think a Cache can be a lot smarter when it's dumb and playing by the rules of the web, and that's exactly what we're doing with WunderGraph. and can be deployed everywhere. Our solution is Open Source How does it work? I've been working with GraphQL for many years now, and I've realized that when we deploy an application that uses GraphQL APIs, I've never seen the application change the GraphQL Operations at runtime. What applications do at runtime is change the Variables, but the Operations usually stay static. So, what we've done is, we've created a GraphQL to JSON-RPC compiler, which treats GraphQL Operations like "Prepared Statements", you've probably heard of the term from using a database. When first using a SQL statement, you send it to the Database and get back a handle to "execute" it later. Subsequent requests can now execute the Statement just by sending the handle and the variables. This makes the execution a lot faster. Because we're not changing GraphQL Operations at runtime, we can actually do this "compilation" step during development. At the same time, we're replacing HTTP POST for Queries with HTTP GET, while sending the Variables as a Query Parameter. By sending GraphQL Queries via JSON-RPC with HTTP GET,
we're able to automatically enable us to use Cache-Control Headers and ETags. And that's the whole magic of WunderGraph's Caching Story. We don't build a "smart" Cache. We don't cache Objects, and we don't have to build complex invalidation logic. We simply cache the response of unique URLs. For each Operation, you can define if it should be Cached and for how long. If a response might change frequently, you can also set the Cache time to 0, but configure "stale-while-revalidate" to a non-negative number. The client will automatically send the request to the origin with the ETag, the server will either send a refreshed response or 304, not modified. It's a very dumb approach. If the cached value is expired or stale, we're asking the origin if they have an update. Depending on the configuration, that might create a few more requests to the origin, but we're also not creating a second source of truth, or have to deal with the complexity of managing cache invalidation tags for nested objects. It's similar to using Change Data Capture (CDC) as a source for Subscriptions vs. simple Polling. CDC can get extremely complex to get right, while simple server-side Polling might work just fine most of the time. It's simple and robust. There's really not much we've invented here, all this is standard Caching behavior and any service, like Cloudflare, fastly, Varnish or Nginx supports it out of the box. There's a standard for how all the participants of the web treat Cache-Control and ETag headers, we've simply implemented this standard. By removing GraphQL from the equation, we've made it compatible with the web. If you build tools for the web, you should respect how the web is built, otherwise, you're creating more problems than you're solving. Additional Benefits of the WunderGraph GraphQL Caching Approach It's not just CDNs and Servers who understand the Cache-Control and ETag Headers. Browsers also automatically cache and invalidate your responses, without adding a single line of code. Additionally, because we've removed GraphQL from the runtime, we've automatically reduced the attack surface of our application. If our frontend doesn't change GraphQL Queries at runtime, why expose an API that allows you to do so? Limitations of the JSON-RPC Caching Approach One of the limitations of this approach is that we're no longer able to use regular GraphQL Clients, as they'd expect us to send GraphQL Operations over HTTP POST, and Subscriptions over WebSockets. That's not an issue for new projects, but might be a blocker for existing applications. Btw. we've got an internal RFC to add a compatibility mode, allowing e.g. Apollo client / urql, etc. to work with WunderGraph through an Adapter. If you're interested, please let us know. That said, using plain JSON-RPC would be very inconvenient. That's why we're not just compiling GraphQL to JSON RPC, but also generating fully type-safe clients. One such client-side integration is the , making it super easy to use WunderGraph with NextJS, including Server-Side Rendering, Authentication, File Uploads, etc... NextJS package Another limitation, as of now, is that you have to self-host WunderGraph. We're working on a hosted Serverless solution, but as of now, you'd have to deploy and run it yourself. While it might not be super convenient, this also comes with an advantage: WunderGraph is Apache 2.0 licensed, you can run it anywhere. How can you deploy WunderGraph Now that we've discussed the pros and cons of the WunderGraph approach to caching, let's see how we can deploy WunderGraph to achieve good Caching results. First, you don't have to deploy WunderGraph globally. It's possible to run it close to your origin, e.g. the (micro-) services that you'd like to use. In this scenario, it's possible to run WunderGraph as an Ingress to your other services. The architecture of this scenario looks like this: Client -> WunderGraph Server -> Origin Deploy WunderGraph with Nginx or Varnish as an additional Caching Layer If you're deploying WunderGraph close to the origin, it will automatically add the required Cache-Control & ETag Headers. This setup might already be enough for your scenario. However, in some scenarios, you'd like to add another layer of Caching Servers. This could be e.g. a cluster of Nginx or Varnish servers, placed in front of your WunderGraph server. Architecture updated with this scenario: Client -> Cache Server -> WunderGraph Server -> Origin Deploy WunderGraph with Cloudflare or fastly as an Edge Cache Depending on where your users are, a centralized Caching Layer might not be sufficient for your use case. In this scenario, you can use services like Cloudflare (Workers) or fastly, to add a globally distributed GraphQL CDN / Edge Cache. What's important to note here is that you're not tied to a specific solution. As mentioned before, we're using standardized Cache-Control directives, supported by all Cache Server solutions, so you're not tying yourself to a specific vendor. Updated Architecture: Client -> CDN -> WunderGraph Server -> Origin Deploy WunderGraph directly on the Edge, using fly.io Another option is to deploy WunderGraph directly on the Edge, removing the need for an additional Caching Layer. Services like allow you to deploy Containers as close to your users as possible. The architecture of this scenario looks like the following: fly.io Client -> WunderGraph Server (on the Edge) -> Origin Deploying a service on the Edge is not always beneficial There's one important point I'd like to make that applies to all solutions mentioned in this post. If we're deploying a workload on the Edge, this service will be very close to the users, which is generally a good thing. At the same time, depending on where on the "Edge" we're processing a request, there might be a random latency between ~0-300ms to your origin server(s) per roundtrip. If we have to do multiple roundtrips to fetch the data for a single client request, this latency can add up. As you can see, it's not always beneficial to the overall performance to have logic running on the edge. There's one more thing! This solution works for GraphQL, REST, gRPC, and other Protocols as well! You've heard that right. WunderGraph is not just about GraphQL origins. We support a wide array of upstream protocols, like REST, gRPC (coming soon!), and Databases like PostgreSQL, MySQL, Planetscale, etc... WunderGraph ingests all your services and creates a "Virtual Graph", a GraphQL Schema representing all your services. The Caching layer we've described above does not just work for GraphQL origins, but all services that you've added to your Virtual Graph. This means, you're able to apply one unified layer of Authentication, Authorization, and of course Caching to all your APIs. From our experience, it's rarely the case that you're connecting a frontend to a single GraphQL server. Instead, you're probably going to connect it to many services, which is what we're trying to simplify, and caching is one part of the story. Conclusion We've discussed various options to improve application performance with different kinds of Caching systems. There are ready-to-use solutions that come with simplicity but also lock you into specific vendors. With WunderGraph, we're trying to offer an Open Source alternative that's based on standards and implemented by many tools and vendors, so you're able to choose the best-of-breed solution for your situation. If you're looking into adding Caching to your stack, we'd love to talk! or . Join us on Discord connect on Twitter Also published . here

How to Leverage Open Source GraphQL CDN/Edge Cache With Cloudflare, Fastly, and fly.io

Writers, your just got a backend lift! If you’ve published a story on April 20th, or since, you will probably see a larger number of reads on your stories. The reason for this is that we’ve started collecting data from . Why did we do this? For more accurate readership reports! story stats Cloudflare Analytics and so using Google Analytics API (as we previously were) to measure readership per story is problematic.  Cloudflare has the advantage of being so integral to how our site’s content is delivered, that their servers’ get a chance to log the traffic before a client could ever block it. counts a successful request to a URL delivering content of type HTML. In this way, we gain a straight forward view into story traffic. 58% of tech-savvy audiences block Google Analytics Cloudflare’s approach “This doesn't mean you can go to your advertisers and start demanding more ad revenue. It should, however, mean that you now have a more accurate picture into the actual resource demands required to run your site.” - Cloudflare’s Matthew Prince Why not?  Well, one reason is that Cloudflare includes more bot and crawler traffic in their analytics. Thankfully, we can filter page views by what Cloudflare calls a botManagementDecision such that the traffic has been deemed to be “likely human” in origin.  Are you a ‘likely human’ reading this story? Then you will count towards the reads at the top of this story page and writers will see that in . Here is the filter used in the GraphQL query: their story stats "pageviewsFilter": {
        "AND": [
          {
            "botManagementDecision": "likely_human"
          },
          {
            "datetime_geq": startDate,
            "datetime_leq": endDate
          },
          {
            "requestSource": "eyeball"
          },
          {
            "AND": [
              {
                "edgeResponseStatus": 200,
                "edgeResponseContentTypeName": "html"
              }
            ]
          }
        ] This was my first time using GraphQL and while it looks like there is sort of living documentation supported, I had a hard time figuring out how to get the data I wanted.  Most of their documentation was focused on firewalls and threat detection. I ended up opening up to see what requests were being sent as I changed filters while viewing their graphs on their own site. Thankfully, I was able to paste those into , an open source client to help you write queries, and verify I was getting the same page reads seen in the charts. my browser’s network tab GraphiQL I really like the idea of using these server logs over downloading and executing javascript on the client.  It is more efficient and less intrusive.  It would be nice to remove the Google Analytics code from our site, but we still rely on it for tracking the time spent on pages and some click events. And keeping it on site just may help with search traffic ;-) Perhaps we’ll roll our own system there in the future. Currently we still only fetch stats once a day, but I’ll be looking into fetching these stats more often so stay tuned for that. Okay, so that’s the update, enjoy! You should be seeing a lot less of these guys on your story now 🙂 Head on over to today and see for yourself! https://app.hackernoon.com/stats

HackerNoon Story Stats Update: Reads Powered By Cloudflare

Actualización de estadísticas de HackerNoon Story: lecturas impulsadas por Cloudflare

Mise à jour des statistiques de l'histoire de HackerNoon : lectures optimisées par Cloudflare

HackerNoon Story Stats Update: Reads Power by Cloudflare

HackerNoon ストーリー統計の更新: Cloudflare による読み取り

Atualização das estatísticas da história do HackerNoon: leituras alimentadas pela Cloudflare

Cập nhật số liệu thống kê về câu chuyện của HackerNoon: Số lượt đọc được cung cấp bởi Cloudflare

HackerNoon 故事统计更新：由 Cloudflare 提供支持的读取

Last week, I had a problem. I wanted to start blogging using . But for SEO reasons, I wanted to put my new blog at , instead of either a completely different domain, or a subdomain like blog.cloak.ist. Ghost cloak.ist/blog That's because I want to build up SEO value for my principal domain, . Google sees subdomains as being a whole other thing. So it's muuuuch better to use if you can. cloak.ist yoursite.com/blog I spent 3 days figuring out how to set up Ghost at cloak.ist/blog. It was pretty painful. So I've decided to celebrate my blog finally existing by writing a guide to how I set it up. I hope it helps you out! Before we get started - You need to own a domain and be using Cloudflare to manage its DNS - You need to be able to buy a $5/month droplet on DigitalOcean Step 1: create a DigitalOcean 1-Click Ghost Droplet Ghost has a hosted option which you can pay for. But I'm going cheap – and also I'm pretty sure, from reading a few things online, that their hosted option won't work with this guide. So head on over to DigitalOcean and . Choose the $5/month droplet. set up a new droplet with their 1-click option Once the droplet is set up, don't go and do anything in it just yet. Just make a note of the droplet's ipv4 address . Step 2: point a subdomain at your droplet's IP You need to set Ghost up at a subdomain at your domain, like . blog.yoursite.com This is so that we can later tell to get all content from that subdomain. But don't worry, we'll also have the option of telling Cloudflare to restrict access to that subdomain to you and Cloudflare, so that you don't have a competing location for your blog living out there. yoursite.com/blog So go to Cloudflare and create a new A record, with Name set to whatever you want the subdomain to be (e.g. Name will be blog if you want to use blog.yoursite.com), and IPv4 address set to the droplet's IP that you have noted. Click the orange cloud to change it from Proxied to DNS Only! This caused me a of pain. VERY IMPORTANT: lot Step 3: drink some tea The next step will be to go back to your droplet and set up Ghost on it. But if your subdomain hasn't propagated yet (in other words, if the internet hasn't yet realised that you added the A record from the previous step), the set up will fail. DNS should propagate pretty fast, but please, please: Have a little break to give DNS time to propagate, hence the tea. I recommend chamomile.On returning from your delicious tea session, check DNS with a before going onto the next step to make sure that your subdomain is pointed at your droplet IP. service like this If you left the orange cloud enabled in the previous step, it's very likely that your Ghost set up will fail and you'll see Cloudflare IP addresses when you do the DNS check. (Not doing this step properly meant that I went into a frenzy of destroying and then remaking droplets. I went through around 5 of them before I got it right.) Step 4: set up Ghost Use . this guide to set up Ghost on your droplet When you're asked for the URL of your Ghost droplet, enter: https: //yoursubdomain.yoursite.com All of the rest of the set up should be straightforward – use the standard options throughout. If you get an error, it could be that your DNS didn't propagate yet so Ghost was unable to issue an SSL certificate for your subdomain. I'd recommend going back to Step 1 and starting again if that happens – alternatively, if your error looks different, have a google around to see if you can fix it. Quite a lot of people seem to have had issues with DigitalOcean's '1-click' Ghost installation... At this point, you should also check that your blog is live at https://yoursubdomain.yoursite.com. Keep your SSH session open! Step 5: put your blog at /blog We're going to put your blog at a subdirectory on the subdomain now, we move it to living at your root domain instead of the subdomain. before Run the following commands while you're still in your SSH session: sudo -i -u ghost-mgr cd / /www/ghost ghost config url https: ghost restart // This will let you configure Ghost var // This will move you into the Ghost installation folder to make changes //yoursubdirectory.yoursite.com/blog // This will change the URL of your Ghost blog. You can obviously change /blog to whatever you want // IMPORTANT! This will restart your Ghost instance so that the URL change comes into effect. At this point, check that https://yoursubdirectory.yoursite.com/blog is working. If you see a 404, you probably need to run ghost restart. Also, you could now set yourself up as the admin of the blog by going to https://yoursubdirectory.yoursite.com/blog/ghost. This is not a necessary step though – you can do it later. Step 6: set up a Cloudflare worker We're going to set up a Cloudflare worker at your subdirectory https://yoursite.com/blog which is going to act as a reverse proxy – so it'll basically serve your Ghost blog at that URL. Go to Cloudflare -> Workers'Add Route' -> yoursite.com/blog*'Add Route' -> www.yoursite.com/blog* if you want to make sure you'Manage Workers''Create a Worker' At this stage, you're going to want to replace everything in the code editor with the following script, after : changing the config object to reflect your set up config = { : , : , : ,
} { url = URL(request.url) targetPath = url.pathname response = fetch( ) (
  	targetPath.includes( ) || 
    targetPath.includes( ) ||
    targetPath.includes( ) ||
    targetPath.includes( )
  ) { response
  } body = response.text() body = body.split(config.subdomain).join(config.root)

  response = Response(body, response) response
}

addEventListener( , (event) => {
  event.respondWith(handleRequest(event.request))
}) // Config for our subdomain (where the Ghost blog currently lives) // and our root domain (where we want it to live) // and our blogPath (the subdirectory we want it at) // Change these to suit your case! const subdomain "yoursubdirectory.yourdomain.com" root "yourdomain.com" blogPath "blog" // Function that processes requests to the URL the worker is at async ( ) function handleRequest request // Grab the request URL's pathname, we'll use it later const new const // Change request URLs to go through to the subdomain let await `https:// ` ${config.subdomain} ${targetPath} // Ghost loads assets like JS and CSS from 3 subdirectories // We don't need to change these requests at all // So if we're getting stuff from those subdirectories, // we return the response of the fetch request from above // immediately. if '/blog/favicon.png' `/ /assets/` ${config.blogPath} `/ /public/` ${config.blogPath} `/ /content/` ${config.blogPath} return // In other cases - which will usually be pages of the // Ghost blog - we want to find any reference to our subdomain // and replace it with our root domain. // This is so that things like our canonical URLs and links are // set up correctly, so we NEVER see our subdirectory in the code. // First we get the body of the response from above let await // Then we search in the body to replace the subdomain everywhere // with the root domain. new return "fetch" I've written comments explaining what this code is doing. Paste this code in, give the Worker a slightly clearer name – e.g. yoursite-blog – hit Save and Deploy, and then go back to the main Workers page: Here, you'll want to hit Edit on the Route you created earlier, and set the Worker on it to the one you just made. Step 7: test it out At this point, if all has gone well, you should have a fully working blog at yoursite.com/blog (or whichever subdirectory you chose). If you don't, check through the instructions to see if there's anything you missed. Step 8 (optional): Restrict access to the subdomain blog So to make things cleaner, you might want to do one last thing: use a Cloudflare Firewall rule to make access to yoursubdirectory.yoursite.com impossible for everyone apart from: 1) you (because you're going to need to go there still to use the CMS) 2) Cloudflare Workers. I feel like this may be a good thing to do for SEO, since otherwise you have a blog with exactly the same content and with competing canonical URLs. (I could be wrong here though – please let me know via the live chat if so!) First, we'll create an IP list for Cloudflare IP addressesGo to Configurations at your main Cloudflare menu: Select Lists, 'Create new list': Paste everything from into your new list and hit SaveGo to your domain from the main Cloudflare menu, then Firewall -> Firewall Rules -> Create a Firewall RuleSet your rule up like this: this list - My Ghost subdomain is cloak.ist – you'll want to replace this in the URI Full filter. - You'll want to change the list filter to use your list of Cloudflare IPs. - Finally, you'll want to use filters like the last 3 to add your own IP addresses. (I have 3 here because I move between different locations a lot.) - ⚠️ You'll need to do this to ensure you have access to editing the Ghost blog! That's it! At this point you should have: - A working blog at yoursite.com/blog......which has correct links to itself, and correct canonical URLs at your root domain (yoursite.com) - A subdomain where you can go to manage your blog, which is only accessible to you and Cloudflare Hopefully this will be much better for your SEO than if you'd had to put your Ghost blog at the subdomain! If you have any questions at all, feel free to tweet @cloakist . Also published at: https://cloak.ist/blog/ how-to-put-a-ghost-blog-at-a-subdirectory-using-cloudflare-workers/

How to set up a Ghost blog on a subdirectory via Cloudflare Workers

When you buy a Domain Name from a registrar (e.g. GoDaddy) to host a website, there is high probability that you would also use their DNS zone hosting services too. However, a secure website requires an SSL certificate, which is not free. Therefore, some people utilize Cloudflare services to get a free SSL bundled with their DNS services. You might have many reasons to choose Cloudflare as your preferred DNS service provider, but you realize very soon that it has started posing new challenges due to some inaccurate configurations. In fact, there is nothing wrong at the Cloudflare-side. It's merely due to lack of our understanding and insufficient documentation for novice users. Imagine a scenario where you subscribed to a web-based application that is hosted by a third-party (e.g. Okta, Zoho). It simply means that you do not have direct access to their web-server instance. You would assume that a simple CNAME record configuration in Cloudflare's DNS setting should be enough to start serving your website. It's not that straight forward. Firstly, you must understand that the Cloudflare is running an HTTP proxy server (i.e. kind of a web-broker) that has two sides to be managed. Being a web-proxy, the Cloudflare allows two different options to you. First option is a Proxy-mode (i.e. traffic intercepted by the Cloudflare) and another one is a DNS-only mode (i.e. traffic sent directly to  web origin servers without changing anything). On one side, it is accepting web request from your user's web browser (e.g. Chrome, Safari) that expects a secure traffic (i.e. HTTPS) therefore the SSL certificates used by Cloudflare must be compatible and acceptable to your user's web browser. This SSL certificate is a major hurdle many times in the configuration. On the other side, the Cloudflare must contact the web-application's origin servers that might be expecting a secure traffic (i.e. HTTPS) therefore SSL certificates acceptable to your origin servers must be used by Cloudflare in a proxy-mode else choose DNS-only mode. Many times people choose to use DNS-only, but the website is not served properly yet. Perhaps, additional configuration is required in Page Rules to route the traffic in right manner to origin servers. Sometimes the problem is due to type of SSL certificate you are using. Also, have some patience while you migrate from some other DNS provider because DNS settings propagation time could also frustrate sometimes. Typically, it takes approx. 24 ~ 48 hours to get DNS settings working properly after you make changes. Acquire some knowledge related to SSL certificates verification method as well, if you are a novice. Some companies (e.g. Okta) have simplified configuration of DNS settings using Cloudflare. I enjoy using Cloudflare due to its analytics capabilities and simple to use user interface.

CLOUDFLARE

cloudflare.com

-- employees

Since n.d.

COMPANY RANKING

CLOUDFLARE

EVERGREEN INDEX #2849

Representing this company? Click here to claim and customize this page!

Cloudflare WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

CLOUDFLARE

cloudflare.com

-- employees

Since n.d.

COMPANY RANKING

CLOUDFLARE

EVERGREEN INDEX #2849

RELATED COMPANIESRANK

STORIES ABOUT CLOUDFLARE

Representing this company? Click here to claim and customize this page!

Cloudflare WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!