Test Results Prove Some Free CDNs are Faster than Premium CDNs

Evidently premium CDNs are NOT always faster than free CDNs. While testing whether Free CDNs performed better than premium CDNs, I discovered many free CDNs are actually performing faster than Amazon CloudFront. Cloudflare is starting to look good too. One is Cloudinary, the other is WordPress’s Jetpack CDN. In this article, I go over my CDN experiment showing how Jetpack CDN was faster than Amazon CloudFront in nearly every case except for one. Before I present the simplified summary, here are some links relevant to this article. . My testing approach to CDNs . Python script for measuring CDNs On Twitter, or . I tweet analysis about software performance email list , based only on hot cache results: As a very simplified summary Jetpack CDN was faster than CloudFront by 25% (0.332 vs 0.245 seconds, or a difference of 87 milliseconds). Even after controlling for image size, CloudFront was only faster by 6% (0.199 vs 0.214 seconds, or a difference of only 15 milliseconds). Basically within a margin of error. Jetpack CDN had better performance for several reasons: The nice trick of compressing images 2–3x smaller (~700 KB image to ~236 KB). Much more common DNS ( vs ), which allows for a significantly faster DNS lookup. In one case, 0.11 vs 0.008 seconds. i0.wp.com d20zaq59cm4c4j.cloudfront.net CloudFront typically degrades a file from a hot cache to warm cache very rapidly, whereas Jetpack CDN doesn’t do this until at least several days. What follows is a much more detailed technical analysis. I had to do 2-speed tests. The 1st speed test was not apples-to-apples because Jetpack CDN surprisingly compressed the image, leading to a crushing result. The 2nd speed test, I controlled for file size, and CloudFront came out ahead very slightly by 6%. In the first speed test, Jetpack CDN surprisingly crushed CloudFront. I was shocked. I investigated and found out . Well…that explained a lot. Jetpack CDN magically compressed the ~700 KB image down to ~219 KB I had to do a second speed test making sure CloudFront was delivering the same image as Jetpack CDN. CloudFront did a lot better in the hot cache. Although it still sucked at the other two metrics. There were several reasons for Jetpack CDN performing so well: Now there is an instance where …you “get what you pay for”…while this free Jetpack CDN has great performance, it is limited in many other ways. You can’t host videos. You can’t host a whole website off Jetpack CDN (while this blog itself is entirely CloudFront). do You can’t update a file on Jetpack CDN easily unless you wholesale rename the file (very annoying). The amazing performance applies mainly to images. JS and CSS won’t have the same amazing image compression, although you see Jetpack CDN perform on par with CloudFront. See the conclusion for my thoughts on the . will Jetpack CDN vs CloudFront decision It should be noted that every free CDN performs as well as Jetpack CDN. I found 2 free CDNs that performed really poorly. not By the way, I’ve been blogging a series of experiments on free CDNs and comparing them against CloudFront. You can get updates by following me on , or via . Twitter SpeedTestDemon email list Why is it interesting to test Jetpack CDN (previously known as Photon CDN)? Because it’s an extremely popular CDN with millions of users. It was able to get so many users due to it being a first-class feature in WordPress’s Jetpack, which itself has 5 million users. If you’ve got Jetpack, it is basically 1 click to enable Jetpack CDN, and all your blog’s images get served from the Jetpack CDN. And not to mention free! Jetpack CDN is a free CDN. So for that reason, it’s popular. It used to be called Photon CDN before they did a name change. But how “good” is free? Is it as good as a CDN you have to pay for, like Amazon’s CloudFront? Why Would one use CloudFront as the Gold Standard in CDN speed tests? Well Amazon AWS is in general the gold standard for anything in the cloud. Amazon AWS has the largest market share for cloud providers (32% with Azure at 19%). They were the first to market with cloud services, and they’ve had many years to develop CloudFront into a top-of-the-line service. AWS CloudFront has a generous free tier as well, along with only $0.08 / GB network-out. If you’re a normal website, you might incur a total aggregate cost of $0.50/month (which is nothing). Brief Summary of the Metrics Being Measured It’s not as simple as running a curl on the same file from the two CDNs. Most simplistically: CDNs are just a caching layer. There are 3 states that a cache can be in: cold cache, hot cache, and warm cache. We are going to measure all 3 (because we have it automated so why not). If you are interested, you can read this much more detailed article about my . CDN speed testing methodology If you’re interested in trying to replicate our results, see the where we discuss the exact steps we took. Appendix I didn’t expect any of this when I first started. There is actually quite a bit of nuance when analyzing the curl measurements between Jetpack CDN and CloudFront. Finding #1: Jetpack CDN’s DNS lookup is significantly faster than CloudFront’s DNS lookup. To those new to measuring request latency, DNS is actually a VERY important detail that can seem minor. While DNS is normally very fast because it’s usually cached, if your DNS lookup is on a low-trafficked website, it’s highly unlikely to be cached in most routers. This analysis was done off of . speed test #2’s warm cache test I can only speculate why CloudFront took a whole 0.111 seconds compared to Jetpack CDN’s almost imperceptible 0.005 seconds. Compare these two DNS lookups: CloudFront “ ” and Jetpack CDN “ ”. d20zaq59cm4c4j.cloudfront.net i0.wp.com Jetpack CDN’s looks like a shared domain that many other people are also using (there’s millions of JetPack users). This means that it’s likely that nearby routers will have that DNS lookup cached. i0.wp.com Unlike Jetpack CDN, CloudFront’s domain is highly unique and customized to this test instance. It’s highly unlikely nearby routers will have the DNS lookup cached, resulting in a significantly longer DNS lookup. As you can see, it’s fairly interesting that there are indeed cases where DNS lookups can take a very significant part of the time (0.111 seconds out of 0.445 seconds is nearly 25%!). So don’t discount DNS lookups in your measurement times. Finding #2: Generally, the Data Transfer, TCP handshake and SSL handshake times were the same between CloudFront and Jetpack CDN. In general, I found the “generic metrics” speeds were usually almost the same between CloudFront and Jetpack CDN. Probably because these operations “Data Transfer”, “SSL handshake”, “TCP handshake” are the same for every file regardless of file size or file types, no matter if you’re requesting a 10 GB video or just a 10 KB HTML file. I have an essay explaining what each of these metrics mean if you’re interested in learning more: . guide to curl metrics The only time the “Data Transfer” was different was when Jetpack CDN compressed the image to a 2x lower size than CloudFront (see Finding #3). Finding #3: Jetpack CDN had a surprisingly strong image compression algorithm. I found that uploading a 703 KB image was compressed to 219 KB on WordPress’s image CDN, while CloudFront compressed it to only around 300–400 KB. The bottom line is Jetpack CDN is able to have 2x to 3x faster Data Transfer times than CloudFront. Remember Data Transfer time is specifically regarding how long the download takes (once TCP, SSL, and server is prepared to start sending bytes). This is quite impressive. You’ll see Finding #3 referenced several times during analysis of Speed Test #1 when I didn’t account for this factor. Finding #4: Jetpack CDN’s warm cache times were significantly stronger than CloudFront’s warm cache. This analysis was done off of . speed test #2’s warm cache test Impressively, Jetpack CDN’s warm cache time was nearly the same as its hot cache time. This indicates that Jetpack CDN keeps the image in RAM for quite a long time. That’s good! For the user, that is, because it’s a faster response time. This is going to be more expensive for Jetpack CDN because RAM is the most expensive hardware resource by far (way more than SSD). CloudFront on the other hand…was nearly 2x slower than its hot cache time. Taking a closer look at where exactly CloudFront lost so much time, it was in two metrics: “Request Latency” and “DNS lookup”. With regards to “Request Latency” (0.135 seconds compared to 0.020 seconds), it seems clear that CloudFront degrades its hot cache rather quickly, where the image file probably got shunted over to NVMe or SSD. Well, Amazon has a reputation among software engineers for being cheap ;), we really should not be surprised they are trying to save on hardware costs where they think they can get away with it. CloudFront beats out Jetpack CDN when it comes to the cold cache test. However, for the 2nd download, Jetpack CDN is slightly faster (5%) than CloudFront. When we did the measurement again a couple of other times, Jetpack CDN was beating out CloudFront handily on cached downloads. Sometimes it was 2x, sometimes 3x. measurement and analysis (speed test #1). Cold cache CloudFront is 10% slower than Jetpack CDN for a new file. Note: I am aware there is no “Data Transfer” row. I had a mistake with this test run where I accidentally made curl not download the file. CloudFront’s Data Transfer is probably going to be slower than Jetpack CDN’s Data Transfer (see later sections). I don’t need to belabor this point. Turns out yes, CloudFront is 8% slower than Jetpack CDN for cold caches. I’m not really entirely certain why. The main deltas were in DNS lookup and SSL handshake. The DNS lookup delta (0.048 vs 0.002 seconds) might surprise some people who think DNS lookups are normally always very fast. This is not true. They are not always fast. The big exceptions are low-trafficked DNS names ( ). see more analysis in “finding #1” I’m really not certain why CloudFront took so much longer in SSL handshake. It’s kind of out of left-field and perhaps is just randomness. Request Latency was the same for both CloudFront and Jetpack CDN. That’s good news because I would expect both CDNs to take the same amount of time for a file that’s not in cache at all. For some reason, re-running the cold cache test in “speed test #2” showed CloudFront much slower than Jetpack CDN in “Request Latency”. measurement and analysis (speed test #1). Hot cache Jetpack CDN is 25% faster than CloudFront! (0.332 vs 0.256 seconds) (Reminder: these times are an average of 10 curl calls). Where did Jetpack CDN save so much time? We can see CloudFront is neck-and-neck with Jetpack CDN for most of the metrics except for “Data Transfer” which is the download time. This makes sense as CloudFront is trying to send a 703 KB file while Jetpack CDN is only around ~250 KB. See analysis in section “Finding #3”. And Data Transfer was literally the only difference. What’s even more impressive is you could argue the “hot cache” times is one of the most important metrics for a CDN. If your site has a moderate amount of traffic, chances are pretty good the CDN will be in a hot cache state. Thus, Jetpack CDN’s hot cache time is 25% better than CloudFront. Very impressive work from Automattic, the company behind WordPress! To even out the odds between Jetpack CDN and CloudFront, we will be conducting a second speed test. You can jump down to the later on in the article. second-speed test section measurement (speed test #1). Warm cache Jetpack CDN is nearly 2x faster than CloudFront. Reminder: the warm cache is a measurement time taken 30 minutes after the hot cache time. You should also see the 2nd speed test’s warm cache analysis, which differs slightly from the 1st speed test’s warm cache analysis. TCP handshake, SSL handshake was the same. See . analysis in section “Finding #2” Request Latency was also nearly the same. This differs from the 2nd-speed test where I’m seeing CloudFront’s Request Latency spike for some reason. At any rate, it makes sense Request Latency should be the same as both CDNs probably have the file in RAM and that’s pretty much the best you can do, no matter who you are. The big difference in time went to “Data Transfer” where Jetpack CDN was…nearly 3x faster than CloudFront? 0.089 seconds vs 0.228 seconds. Already touched on this earlier. Jetpack CDN did a great job of compressing the image down to around 219 KB while CloudFront is still trying to transfer ~700 KB image. See . analysis in section “Finding #3” Conclusion and Analysis. Speed Test #1 Jetpack CDN beat out CloudFront in hot cache, cold cache, and warm cache, due mainly to the image automatically compressed to 219 KB and CloudFront still trying to transfer a 720 KB PNG. What if we even the odds and make CloudFront and Jetpack CDN transfer the same file? Surely CloudFront should beat out Jetpack CDN now? It’s not that simple. See the next section for Speed Test #2. In brief: CloudFront’s Data Transfer metric improved to be on par with Jetpack CDN. This was expected because CloudFront and Jetpack CDN are now transferring the same file (previously, was transferring ~700 KB compared to ~200 KB image). CloudFront’s Hot Cache times also improved from -25% to +7% speed over Jetpack CDN (due to improved Data Transfer). However, CloudFront’s Cold Cache and Warm Cache still lags behind Jetpack CDN. Impressively, Jetpack CDN is still 2x faster in warm cache time, due to DNS lookup and Request Latency. measurement (speed test #2) Cold cache Seems Cloudfront is still slower than Jetpack CDN when downloading a new file. Note: I am aware there is no “Data Transfer” row. I had a mistake with this test run where I accidentally made curl not download the file. In my opinion, not worth re-running the test since I don’t expect Data Transfer numbers to be extremely different. It’s interesting…I had run the cold cache test a few times and CloudFront is still slower than Jetpack CDN for new files. Even after ensuring the images downloaded the same size. That’s unfortunate. The main reason for the large time discrepancy was in Request Latency — all the other metrics were close enough (except for DNS lookup — see ). I honestly have no clue why CloudFront should take much longer (about 50%) than Jetpack CDN for fetching the image from the origin server — the images are the same size in both cases. finding #2 section measurement and analysis (speed test #2) Hot cache CloudFront and Jetpack CDN neck-to-neck in almost all the individual metrics. (Reminder: these times are an average of 10 curl calls). CloudFront beats out Jetpack CDN by 7%. Oh good, so the premium CDN is finally outperforming the free CDN. But in absolute difference, only about 16 milliseconds. Every single metric here shows CloudFront and Jetpack CDN spending approximately the same time. For some metrics, CloudFront was faster, and vice versa for some other metrics. CloudFront just ended up edging out overall. The relative difference was only 7%, not much of a big deal. And possibly within the margin of error. See “ “ explaining why “Data Transfer”, “TCP handshake”, and “SSL handshake” times were similar between CloudFront and Jetpack CDN. Finding #2 measurement and analysis (speed test #2) Warm cache Reminder: the warm cache is a measurement time taken 30 minutes after the hot cache time. This was an extremely interesting test. Jetpack CDN is 2x faster than CloudFront. That’s not a small difference. How? Where did the time go? The first interesting bit was that Jetpack CDN DNS lookup was way faster than CloudFront. In fact, CloudFront’s 0.111 seconds is nearly 25% of the overall time of 0.445 seconds. There’s a good reason why Jetpack CDN’s DNS lookup was way faster: see . Secondly, CloudFront’s Request Latency was way longer at 0.135 seconds compared to 0.020 seconds. I write about this in the . analysis in “Finding #1” analysis for “Finding #4” The rest of the metrics did not make much difference in absolute time. Yes, CloudFront was “2x faster” TCP handshake (0.025 vs 0.052 seconds) but the absolute difference is only 0.027 seconds. The bigger rocks were in DNS lookup and Request Latency. TL;DR — Jetpack CDN holds files in hot cache for longer than CloudFront. Evidence is Jetpack CDN’s warm cache is nearly the same as its hot cache time. CloudFront degraded about 2x due to slow non-cached DNS lookup and likely non-cached at the point of presence. My initial stance going into this was CloudFront was going to be the outright winner (even after accounting for cost). As it turns out, Jetpack CDN has acquitted itself remarkably well. Based purely on the performance benchmark, I’ve come to the conclusion that Jetpack CDN is better than CloudFront (and yes, Jetpack CDN is free). There two main scenarios for evaluation: 1. Jetpack CDN compresses image better (i.e. smaller) than Cloudfront, 2. Cloudfront and Jetpack CDN deliver the same image. In scenario 1: Jetpack CDN is significantly faster than CloudFront for cold, hot, and warm cache. Even for the hot cache case, Jetpack CDN was 25% faster. . In scenario 2: Jetpack CDN maintains a significant edge over CloudFront in cold and warm cache times (50–100%), while CloudFront has a slight edge in hot cache times (6%) Summary of Jetpack CDN’s superiority over CloudFront: feature works well. It shrunk a 721 KB image to 225 KB. This basically doubles or triples Jetpack CDN’s download speed. This is a big advantage. image_downsizing Jetpack CDN keeps images in hot cache for significantly longer than CloudFront. At one point I even tested Jetpack CDN a full day after the hot cache test, and it was still nearly as quick as hot cache. I know it’s still cached because I’m getting back this header “x-nc: HIT dfw 6” from Jetpack CDN. The warm cache is some fairly impressive stuff from Jetpack CDN. Extraordinarily simple to set up on a WordPress instance. Basically, 1-click install from the Jetpack plugin (which itself has millions of installs). If you’ve ever used CloudFront, you’d know it is very confusing. Take a look at this screenshot, it’s dizzying. Even for a software engineer like me, I had a moment of confusion and minor agony when staring down the options (I thought: Why isn’t it simpler?). There good reasons to use another CDN besides Jetpack CDN despite its great performance, cost, and ease of use. are The main thing is that Jetpack CDN is fairly limited. For one thing, Jetpack CDN can only do images, JS, and CSS…yeah that’s fairly limited. If your website is totally static, you should put HTML on Jetpack CDN too. And also videos and audio if you’ve got them. Here’s a full list of (important) Jetpack CDN limitations. There are other minor limitations but I don’t deem them important. Doesn’t serve video or audio. Doesn’t serve HTML. No cache invalidations — this means it’s much more annoying to update an image. You need to rename the image in order to force CDN to load the new image. Limited to WordPress instances. Need Jetpack installed. So at the time of writing, I would recommend Jetpack CDN if: 1) you’re using WordPress, 2) for images only. You still need to get a CDN like CloudFront to handle all the other stuff like HTML, video, and audio. Things may change in the future that may change my mind, such as Jetpack CDN’s warm cache times. In general though I would recommend getting a premium CDN. See my analysis here on why a . premium CDN is worth it This section is specifically for Jetpack CDN and CloudFront. Here is the article about my . general testing methodology We used a python script that used cURL. We used cURL because of its ability to be flexible and complete complex tasks. You can use curl for things like user authentication, HTTP post, SSL connections, proxy support, and FTP uploads. You can also do simple things with curl, such as download web pages and web images, which we used cURL for in this scenario. Tests are conducted on a Macbook Air 13" 2020 using macOS Catalina v. 10.15.7. This uses an Dual-Core Intel Core i3 processor clocked at 1.1 GHz alongside 8 GB 3733 MHz LPDDR4X RAM and an SSD. Creating python file “ ” test.py Clone this repository. Free speed test! git clone https://github.com/speedtestdemon/speed-tests.git cd speed-tests Install pycurlpip3 install pycurl If you encountered an error, try first sudo apt install libcurl4-openssl-dev libssl-dev Then run again pip3 install pycurl Now that installation is complete, try to run a speed test: python3 " " test.py https://d3va53q3li7xt1.cloudfront.net/wp-content/uploads/2021/05/shoeb-1024x576.png Setup and Testing — Cloudfront Create a distribution in CloudFront Create a behavior with a Path pattern “/wp-content/” 1. Install CDN Enabler plugin on WordPress. 2. Update the Settings of CDN Enabler, input the Domain Name provided by CloudFront from the setup above. 3. Add a new Post in WordPress, upload the image provided above. 4. View Post and right-click on the Image, right-click and Inspect. 5. Copy the link from the Image Source. 6. Download the image using the command, open Terminal, go to the same folder as and paste the command below. test.py 7.a For the different size but same image — due to downsizing capabilities of each CDN python3 “ " test.py https://d3va53q3li7xt1.cloudfront.net/wp-content/uploads/2021/05/shoeb-1024×576.png 7.b For the same size, same image python3 “ " test.py http://d20zaq59cm4c4j.cloudfront.net/wp-content/uploads/2021/05/shoeb-jetpack.png 1 Setup and Testing — Jetpack CDN . Install Jetpack plugin on WordPress site. 1. Go to Settings of Jetpack, enable the Site Accelerator feature under the Performance tab. 2. Add a new Post in WordPress, upload the image above. 3. View Post and right-click on the Image, Right-click and Inspect Copy the link from the Image Source: 1. Go to the same folder as and download the image using cURL with the command below test.py python3 “ test.py https://i0.wp.com/comparecdns.com/wp-content/uploads/2021/05/shoeb.png?resize=1024%2C576&ssl=1 " Related Essays That’s the end of the Appendix. Here are related essays: Is a premium CDN like CloudFront worth it, instead of a free CDN? Cheat Sheet on Curl Performance Metrics: how to benchmark server latency with curl CDN Speed Tests (Testing Methodology) : I use this to automate the collection of metrics about CDNs. Python CDN Speed Test Script