can shrink a 69MB container down to under 2MB, resulting in space savings of 97%, while making it more secure and easier to operate. Constructing can require some tricky modifications, however, so some practice with the tool is useful. TL;DR: Smith apache httpd microcontainers is a builder that I have been working on for the past few months. The shows how to build an ultra-simple microcontainer, but some software can be trickier to “microize”. Apache httpd is a good example of a tricky application because it includes configuration files, has some complex permissions requirements, and loads dynamic libraries. All of these things require some extra steps beyond the basics. Below I detail the process of building a working httpd microcontainer. Smith microcontainer readme First Steps Smith can build directly from yum repositories and rpms using mock, but it can also “microize” existing docker containers. For the purposes of this process we will use the official as our base. The latest tag is built from debian jessie and weighs in at 69MB compressed. There is also an alpine image at a much leaner 28MB. I will use the debian-based latest image, but a similar process could be performed with the alpine image as well. httpd container on Docker Hub The first step is to download and install smith: # before starting ensure you have golang 1.6 or later installedgit clone github.com/oracle/smithcd smithmakesudo make install Now we create a new directory for our build: mkdir smith-httpdcd smith-httpd Next, we download the httpd image so we can use it as a base. (Note that you can also specify the package as the full download path in the smith.yaml, but the image is not cached so it will re-download every time we do a build. Since we are going to be building quite a few times, it is best to download the image manually.) smith download -r \-i httpd.tar.gz https://registry-1.docker.io/library/httpd The next step is to create a base yaml file. I have no idea where the httpd binary is installed in the image so we will just guess at its location. The arg of tells httpd not to daemonize so that docker can pick up logs in the usual manner: -DFOREGROUND cat << EOF > smith.yamltype: ocipackage: httpd.tar.gzpaths:- /usr/bin/httpdcmd:- /usr/bin/httpd- -DFOREGROUNDEOF Now we can try a build: smith -i smith-httpd.tar.gz# …# WARN[0002] chmod failed: chmod: cannot access ‘/usr/bin/httpd’: No such file or directory# WARN[0002] Could not make paths readable: Process chmod -R go+rX /usr/bin/httpd exited with status 1# … It looks like /usr/bin/httpd is not the right location. Fortunately, smith unpacks the layers from our source image into a directory for us, so we can look around. The directory is . Let’s find the right binary: /tmp/smith-unpack-$(id -u) find /tmp/smith-unpack-$(id -u) -name httpd# /tmp/smith-unpack-1000/usr/local/apache2/bin/httpd It looks like the binary has been installed into . In fact if we look around a bit in the unpacked image, it looks like everything has been installed under . Let’s take a bit of a shortcut and just grab the whole directory: /usr/local/apache2/bin/httpd /usr/local/apache2 cat << EOF > smith.yamltype: ocipackage: httpd.tar.gzpaths:- /usr/local/apache2/cmd:- /usr/local/apache2/bin/httpd- -DFOREGROUNDEOF Now we can build again. Since we haven’t changed our source image, we can skip the unpack step by passing in when we build: -f smith -f -i smith-httpd.tar.gz Testing our Image Docker importing an oci image directly via docker load, so we have to push the image to a repository to use it. Docker Hub works just fine. Once you create an account and make a new repository, you can push your image as follows: doesn’t yet support smith upload -r \https://<user>:<pass> -1.docker.io/<user>/<repo> \-i smith-httpd.tar.gz @registry You can optionally append to the repository if you want to tag the image as something other than . Once the upload is complete, you can run the image via or you can follow along by using the images that i have built and uploaded: :<tag> latest docker run -it --rm <user>/<repo> docker run -it --rm vishvananda/smith-httpd:first# …# /usr/local/apache2/bin/httpd: error while loading shared libraries: libuuid.so.1: cannot open shared object file: No such file or directory Hmm, looks like it doesn’t quite work. Fixing our Image Smith is smart enough to download any libraries needed by the binaries selected for the image as long as they are not opened via dlopen at runtime. Unfortunately, apache does dlopen some libraries, including as seen in the error message above. Sometimes you can find libraries that might be opened by running on the binary, but the best method I have found is to try to run the binary and see what fails. Usually the error message will tell you exactly what file is needed. libuuid.so.1 strings In this case, we need to add to our image. This is easily done by appending that filename to paths in our yaml file. After rebuilding and re-pushing the image, remember to use a new tag or use to force it to get the updated version before . If you run the new image, you will see a similar error message complaining about . To save you some time, here is a new yaml with the full list of dynamically loaded libraries which you will need: /lib/x86_64-linux-gnu/libuuid.so.1 docker pull <user>/<repo> docker run librt.so.1 cat << EOF > smith.yamltype: ocipackage: httpd.tar.gzpaths:- /usr/local/apache2/- /lib/x86_64-linux-gnu/libuuid.so.1- /lib/x86_64-linux-gnu/librt.so.1- /lib/x86_64-linux-gnu/libdl.so.2- /lib/x86_64-linux-gnu/libexpat.so.1- /lib/x86_64-linux-gnu/libgcc_s.so.1- /lib/x86_64-linux-gnu/libcrypt.so.1cmd:- /usr/local/apache2/bin/httpd- -DFOREGROUNDEOF After another build/upload/pull we can run it again: docker run -it --rm vishvananda/smith-httpd:second# …# AH00543: httpd: bad user name daemon To understand what is going wrong here, we need to digress a bit. In most installations apache httpd will run as root. It generally needs to run as root so that it can bind a port below 1024. Of course, it is a security risk to have your web server running as root, so it then drops privileges to a user/group that is specified in the config file. In an ideal world we would remove the User and Group commands from the httpd config file and run the binary as an unprivileged user. This can be done, but it requires us to work around the lack of binding low ports in one of the following ways: Change the config to make httpd listen on an unprivileged port pass to (Requires kernel >= 4.11) --sysctl net.ipv4.ip_unprivileged_port_start=0 docker run pass to and also set the proper capability bit on the httpd binary (There was briefly support for ambient capabilities in docker which would obviate the need for setting the capability bit, but it was reverted and has not yet made it back in to docker). --cap-add NET_BIND_SERVICE docker run These options aren’t ideal for everyone, so the creators of the httpd image use the tried-and-true method of starting as root and dropping to another user. In general we do not want microcontainers to run as root, but smith supports it for cases like this. In order to enable the feature we are going to want to set two booleans in our yaml: and . sets up the image to run as root, and installs the required linux bits to make username and group lookups work in the container. Let’s make a new yaml with these settings: root nss root nss cat << EOF > smith.yamltype: ocipackage: httpd.tar.gznss: trueroot: truepaths:- /usr/local/apache2/- /lib/x86_64-linux-gnu/libuuid.so.1- /lib/x86_64-linux-gnu/librt.so.1- /lib/x86_64-linux-gnu/libdl.so.2- /lib/x86_64-linux-gnu/libexpat.so.1- /lib/x86_64-linux-gnu/libgcc_s.so.1- /lib/x86_64-linux-gnu/libcrypt.so.1cmd:- /usr/local/apache2/bin/httpd- -DFOREGROUNDEOF We also want apache to drop to the user that owns the files in the image in order to avoid any permissions issues. The default for all files in microcontainers built with smith is . We are going to have to modify the httpd config to use these names. Additionally, by convention, we put config files for microcontainer processes in , so let’s create an overlay that does that: user:group smith:smith /read # make an overlay read directorymkdir -p rootfs/read# copy the config file out of the imagecp /tmp/smith-unpack-$(id -u)/usr/local/apache2/conf/httpd.conf \rootfs/read/# replace the file in our image with symlink to our filemkdir -p rootfs/usr/local/apache2/conf/ln -s /read/httpd.conf rootfs/usr/local/apache2/conf/httpd.conf Now we can replace the User and Group with the desired settings: sed -i -e ‘s/User daemon/User smith/’ \-e ‘s/Group daemon/Group smith/’ rootfs/read/httpd.conf After a build/upload/pull, we can run the container and expose a port on the host: docker run -it -p 80:80/tcp --rm vishvananda/smith-httpd:third And a quick curl from another shell verifies that httpd is working: curl localhost# <html><body><h1>It works!</h1></body></html> Getting Truly “Micro” Now that the container is working we can work on minimizing it further. There is actually quite a bit of extraneous stuff in the apache2 directory. The only bin file we are using is httpd itself. We do need the modules, but we can pick out just the ones that are enabled in . This is our final build, so we can also switch to pulling the image directly from Docker Hub: httpd.conf cat << EOF > smith.yamlpackage: nss: trueroot: truepaths:- /usr/local/apache2/bin/httpd# dynamic dependencies- /lib/x86_64-linux-gnu/libuuid.so.1- /lib/x86_64-linux-gnu/librt.so.1- /lib/x86_64-linux-gnu/libdl.so.2- /lib/x86_64-linux-gnu/libexpat.so.1- /lib/x86_64-linux-gnu/libgcc_s.so.1- /lib/x86_64-linux-gnu/libcrypt.so.1# modules- /usr/local/apache2/modules/mod_authn_file.so- /usr/local/apache2/modules/mod_authn_core.so- /usr/local/apache2/modules/mod_authz_host.so- /usr/local/apache2/modules/mod_authz_groupfile.so- /usr/local/apache2/modules/mod_authz_user.so- /usr/local/apache2/modules/mod_authz_core.so- /usr/local/apache2/modules/mod_access_compat.so- /usr/local/apache2/modules/mod_auth_basic.so- /usr/local/apache2/modules/mod_reqtimeout.so- /usr/local/apache2/modules/mod_filter.so- /usr/local/apache2/modules/mod_mime.so- /usr/local/apache2/modules/mod_log_config.so- /usr/local/apache2/modules/mod_env.so- /usr/local/apache2/modules/mod_headers.so- /usr/local/apache2/modules/mod_setenvif.so- /usr/local/apache2/modules/mod_version.so- /usr/local/apache2/modules/mod_unixd.so- /usr/local/apache2/modules/mod_status.so- /usr/local/apache2/modules/mod_autoindex.so- /usr/local/apache2/modules/mod_dir.so- /usr/local/apache2/modules/mod_alias.socmd:- /usr/local/apache2/bin/httpd- -DFOREGROUNDEOF https://registry-1.docker.io/library/httpd To follow the microcontainer conventions, we also want to make a few more tweaks to the conf and rootfs: # move the mime.types config file into the read directorycp /tmp/smith-unpack-$(id -u)/usr/local/apache2/conf/mime.types \rootfs/read/ln -s /read/mime.types rootfs/usr/local/apache2/conf/mime.types# create a directory for serving filesmkdir -p rootfs/read/www# copy the index file into our new directorycp /tmp/smith-unpack-$(id -u)/usr/local/apache2/htdocs/index.html \ rootfs/read/www/# change the serve directory in the config# (note we could also have accomplished this with a symlink)sed -i ‘s;/usr/local/apache2/htdocs;/read/www;’ \ rootfs/read/httpd.conf# store the pidfile in /runsed -i ‘/Group smith/a PidFile /run/httpd.pid’ rootfs/read/httpd.conf After a build/upload/pull, we run our microcontainer as securely as possible: docker run -it -p 80:80/tcp --read-only --tmpfs /run \-v httpd-read:/read:ro --rm vishvananda/smith-httpd We have access to the served directory on the host so we can make changes: sudo sed -i ‘s/It works/Hello world/’ \/var/lib/docker/volumes/httpd-read/_data/www/index.htmlcurl localhost# <html><body><h1>Hello world!</h1></body></html> Building From a Yum Repository The yum based build using mock is surprisingly similar. After , you can simply replace the package line in the yaml with . The files will be in different locations than in the docker image, of course, and in this case there is no unpacked image to explore. setting up mock package: httpd To find the files you want to pull out, use after the first build to look around in the mock chroot. You can also use to copy files from the chroot into your local directory. The build/upload/run process is the same, and I encourage you to try to generate a microcontainer without explicit instructions. /usr/bin/mock --shell /usr/bin/mock --copyout Conclusion The first working version of our microcontainer clocked in at a svelte 6MB and the final version is under 2MB. That is less than 3% of the size of the original container! In addition we have removed all extraneous binaries and made the filesystem of the container read-only, improving the operational characteristics and security of the container itself. I hope this inspires you to build some other microcontainers with smith.
