Persist ~ 70% faster; restore ~ 80% faster, your mileage may vary A few days ago, commented on where I shared how we automate our release process with . can be summarized with “ ” Andrew Stiegmann a blog post of mine CircleCI Andrew’s comment Hey, is there any reason you use CircleCI cache instead of a workspace? I read up on that explains the difference between a cache and a workspace. Their diagram does a great job explaining all that: a CircleCI blog post CircleCI cache vs workspace. Source: CircleCI blog post Our CircleCI workfow contains of five jobs, each needs access to and a bunch of generated files in folders. Our “build job” as outlined in the diagram is where we install all npm packages and generate the files in the folders. We use a monorepo (more about that ), which results in a lot of packages hoisted to the root directory. node_modules dist dist here node_modules Migrate from a cache to a workspace The main change here is in the file. First, the directive needs to be replaced with . Likewise, any occurrence of needs to be replaced with an directive. .circleci/config.yml [save_cache](https://circleci.com/docs/2.0/configuration-reference/#save_cache) [persist_to_workspace](https://circleci.com/docs/2.0/configuration-reference/#persist_to_workspace) [restore_cache](https://circleci.com/docs/2.0/configuration-reference/#restore_cache) [attach_workspace](https://circleci.com/docs/2.0/configuration-reference/#attach_workspace) In our case, that change alone improved persistence of data by about 60% (from 67 seconds writing to the cache to 27 seconds persisting to a workspace), while restoring it in a subsequent job dropped from 35 seconds with a cache to 12 seconds with a workspace (a roughly 65% performance gain). Bonus improvement — prune node_modules So far so good, but while I was at it, I dug a bit deeper. For a while now I’ve had an eye on our directory size… 🙀 Have you ever checked yours? If so, I’m quite sure you’re with me here. If not, go ahead and check yours, then come back here — I’ll wait. node_modules Heaviest objects in the universe — based on estimates Alright, now that all readers understand what I mean, let’s continue. In our case, the size of is 553 MB. Does it have to be though…? Definitely not! There’s no need to have files, documentation assets, tests, temporary files, etc. All these files have to be compressed and decompressed when we share it across jobs on CircleCI, regardless of whether we use a cache or a workspace. node_modules *.md I’m aware of two options, so I implemented both and compared them to make an informed decision on which one to choose. node-prune TJ Holowaychuk built that for one of his products and open-sourced it at . It’s a tiny Go command that can be installed in a single line. In our case, it takes 3 seconds to install and run the command. The output of the command looks as follows: https://github.com/tj/node-prune node-prune doing its job on CircleCI That’s 136 MB of unnecessary files dropped from the directory. That’s also 136 MB less data to be compressed and decompressed when passing data from one CircleCI job to the next. node_modules Now, persisting data to a workspace takes 20s, a roughly 70% improvement compared to the original implementation of using a cache. Restoring that data takes 7 seconds, a total of 80% faster than what we had originally. yarn autoclean I wasn’t satisfied with only one option to prune . Whenever possible, it’s a good idea to have a few extra data points. node_modules I found as documented at . This command cleans up as part of the command. yarn autoclean https://yarnpkg.com/lang/en/docs/cli/autoclean/ node_modules yarn install Running that resulted in slightly slower performance results compared to . took about 20 seconds longer (because it runs ) which is 17 seconds more than what requires to get installed and prune the dependencies. Persisting the repository to a workspace took 23 seconds (about 65% faster than compared to a cache) and restoring the workspace was “only” about 70% faster. node-prune yarn install autoclean node-prune Conclusion This table taken from the PR I opened at work summarizes the results: CircleCI cache vs workspace I completely dropped the cache for now, mainly to keep the PR small and manageable. I’m playing with some thoughts to see whether a cache could help speed up multiple workflow runs. In our case, we save 2.6 minutes for an entire workflow on CircleCI. If you have similar results, or a completely different experience, I’d love to hear from you. Please leave a comment or clap if you found this interesting (those claps are encouraging to write and share more 😀).
