Docker changed the software development game by packaging an application and its dependencies together. It largely eliminated the pain of onboarding developers and deploying applications. But there was an upside to the old way where an oil-covered sysadmin manually built an environment for an application.  As long as nothing was touched, it would run until the server gave out. Today, it's quite common for an application to be built and deployed many, many times a day. If dependencies for your application are unstable, you have to be careful. Say your app depends on something like Joe's gcsfuse library  for "mounting and accessing buckets as local file systems." Cloud Storage This might be pretty crucial to the functioning of your application. It  lives in a vendor repo on the internet and might get installed into your docker file something like this: FROM python:3.9-buster
	
RUN set -e;  
apt-get update -y && apt-get install -y  
tini  
lsb-release;  
gcsFuseRepo=gcsfuse-`lsb_release -c -s`;  
gcsFuseRepo=gcsfuse-`lsb_release -c -s`; \
echo "deb http://packages.eat.at.joes.com/apt $gcsFuseRepo main" | \
tee /etc/apt/sources.list.d/gcsfuse.list; \
curl https://packages.eat.at.joes.com/apt/doc/apt-key.gpg | \
  
apt-key add -;  
apt-get update;  
apt-get install -y gcsfuse  
&& apt-get clean Towards the bottom of the file, you might install your app/language-specific dependencies and then specify the entry point into your container. Docker RUN pip install --no-cache-dir -r requirements.txt 
RUN chmod +x /app/cloud_driver.sh  
CMD ["/app/cloud_driver.sh"] As long as you don't edit the top part of the file, Docker uses the cached layer that lives on your local file system.  And as long as the cache is intact, Docker will never pull from  https://packages.eat.at.joes.com But if the cache is cleared or you run the build process on another machine, that repository must be available. If it's not, YOU ARE DEAD IN THE WATER! In fact, I wrote this up after Google's gcsFuse repository went offline.  In this case, a bunch of folks complained on GitHub. Google eventually provided a workaround. Still, my team was blocked for several days, and someone had to edit the docker file to implement the fix if we were dealing with eat.at.joes.com, who knows what might have happened. Read on if it's crucial that you can always build your containerized application. Intermediate Images to the Rescue My preferred solution is to create two docker images, one to install the unstable dependencies and the other containing your app, which will use the first as a base. So: FROM python:3.9-buster

RUN set -e;  
apt-get update -y && apt-get install -y  
tini  
lsb-release;  
gcsFuseRepo=gcsfuse-`lsb_release -c -s`;  
gcsFuseRepo=gcsfuse-`lsb_release -c -s`; \
echo "deb http://packages.cloud.google.com/apt $gcsFuseRepo main" | \
tee /etc/apt/sources.list.d/gcsfuse.list; \
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | \
  
apt-key add -;  
apt-get update;  
apt-get install -y gcsfuse  
&& apt-get clean Which you then push to a container repository… docker build -t my-intermediary-image:latest .
docker push my-intermediary-image:latest Base your application docker file on the intermediate one: FROM my-intermediary-image:latest

RUN pip install --no-cache-dir -r requirements.txt 
RUN chmod +x /app/cloud_driver.sh  
CMD ["/app/cloud_driver.sh"] My-intermediary-image is effectively immutable, and you'll be able to build your application almost anywhere. You can build and tag intermediate containers whenever there's a new and better version of gcsfuse.  If you don't like it, you can simply build your app with a previous version. What About CI/CD Providers? Yes, CI/CD providers can cache your Docker layers between builds. But  it's often  a premium feature and in all cases, you'll have to make sure the vendor-specific tooling is set up to support it. So go check ! And if it's crucial for you to be able to build your app anywhere, use an intermediate image. your CI/CD setup Other Solutions There are other solutions like hosting the third-party code in an artifact repository like AWS S3, Nexus, Artifactory, etc. If you're already using services, those services might be a good solution for you, though it will complicate your docker file. At an operating system level, you could mirror or proxy the repo using tools like or . apt-mirror apt-cacher-ng These methods require you to come up with a method for updating the third-party code. In many cases, you'll have to implement versioning if you want to fall-back behavior you get out of the box with the intermediate container.

Walkthroughs, tutorials, guides, and tips. This story will teach you how to do something new or how to do something better.

Founder Advice: Choosing the Right Cloud Infrastructure Can Make a Big Difference

Hire me to build something that does good.

Managing Unstable Dependencies in Docker: The Power of Intermediate Images

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

Founder Advice: Choosing the Right Cloud Infrastructure Can Make a Big Difference

104 Stories To Learn About Continuous Integration

139 Stories To Learn About Cicd

15 of the Best Continuous Delivery Tools

5 Best Microservices CI/CD Tools You Need to Check Out

Founder Advice: Choosing the Right Cloud Infrastructure Can Make a Big Difference

104 Stories To Learn About Continuous Integration

139 Stories To Learn About Cicd

15 of the Best Continuous Delivery Tools

5 Best Microservices CI/CD Tools You Need to Check Out

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps