Like the rest of the Rust community, crates.io has been growing rapidly, with download and package counts increasing 2-3x year-on-year. This growth doesn't come without problems, and we have made some changes to download handling on crates.io to ensure we can keep providing crates for a long time to come. crates.io The Problem This growth has brought with it some challenges. The most significant of these is that all download requests currently go through the crates.io API, occasionally causing scaling issues. If the API is down or slow, it affects all download requests too. In fact, the number one cause of waking up our crates.io on-call team is "slow downloads" due to the API having performance issues. Additionally, this setup is also problematic for users outside of North America, where download requests are slow due to the distance to the crates.io API servers. The Solution To address these issues, over the last year we have decided to make some changes: Starting from 2024-03-12, cargo will begin to download crates directly from our static.crates.io CDN servers. Starting from 2024-03-12, cargo CDN This change will be facilitated by modifying the config.json file on the package index. In other words: no changes to cargo or your own system are needed for the changes to take effect. The config.json file is used by cargo to determine the download URLs for crates, and we will update it to point directly to the CDN servers, instead of the crates.io API. config.json cargo config.json cargo Over the past few months, we have made several changes to the crates.io backend to enable this: We announced the deprecation of "non-canonical" downloads, which would be harder to support when downloading directly from the CDN.
We changed how downloads are counted. Previously, downloads were counted directly on the crates.io API servers. Now, we analyze the log files from the CDN servers to count the download requests. We announced the deprecation of "non-canonical" downloads, which would be harder to support when downloading directly from the CDN. announced the deprecation of "non-canonical" downloads We changed how downloads are counted. Previously, downloads were counted directly on the crates.io API servers. Now, we analyze the log files from the CDN servers to count the download requests. The latter change has caused the download numbers of most crates to increase, as some download requests were not counted before. Specifically, crates.io mirrors were often downloading directly from the CDN servers already, and those downloads had previously not been counted. For crates with a lot of downloads these changes will be barely noticeable, but for smaller crates, the download numbers have increased quite a bit over the past few weeks since we enabled this change. Expected Outcomes We expect these changes to significantly improve the reliability and speed of downloads, as the performance of the crates.io API servers will no longer affect the download requests. Over the next few weeks, we will monitor the performance of the system to ensure that the changes have the expected effects. We have noticed that some non-cargo build systems are not using the config.json file of the index to build the download URLs. We will reach out to the maintainers of those build systems to ensure that they are aware of the change and to help them update their systems to use the new download URLs. The old download URLs will continue to work, but these systems will be missing out on the potential performance improvement. config.json We are excited about these changes and believe they will greatly improve the reliability of crates.io. We look forward to hearing your feedback! Tobias Bieniek on behalf of the crates.io team Tobias Bieniek on behalf of the crates.io team the crates.io team the crates.io team Also published here Also published here here Photo by Ruben Mavarez on Unsplash Photo by Ruben Mavarez on Unsplash Ruben Mavarez Unsplash

crates.io: The Changes Being Made to Download Handling

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

All the Details and Changes That Came With Rust 1.82.0

Improperly Stored Session Cookies - What the crates.io Team Is Doing to Fix It

10 Most Sought-After Programming Languages You Should Learn In 2021

The Noonification: How to Move Away From Twitter (12/15/2022)

The Noonification: The Wall of Death (12/26/2022)

6 Best Rust Programming Books Ranked by Reviews

All the Details and Changes That Came With Rust 1.82.0

Improperly Stored Session Cookies - What the crates.io Team Is Doing to Fix It

10 Most Sought-After Programming Languages You Should Learn In 2021

The Noonification: How to Move Away From Twitter (12/15/2022)

The Noonification: The Wall of Death (12/26/2022)

6 Best Rust Programming Books Ranked by Reviews

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps