In this article, we’ll do a deep dive into the Buildpacks BuikdKit frontend implementation and see how it actually works under the hood. However, before we do so, we’ll need to breeze through a few concepts.
A container image consists of a config file and layer tarballs. Usually, they are made from a series of instructions like FROM, ADD, and RUN, which are placed in a Dockerfile and built with a docker build. However, a docker build is just one way of ending up with this output.
Buildpacks belong to the family of non-Dockerfile based image-building tools. It takes your application source code and transforms it into an OCI image, usually without any additional input from the user.
To use Buildpacks to package your code, you would run the Pack CLI, specifying your application source code as well as a builder, which is an ordered collection of Buildpacks to be executed.
BuildKit is defined as a toolkit for converting source code into build artifacts in an efficient, expressive, and repeatable manner.
Note that this means that BuildKit is not limited to building OCI images, although it is typically used as such.
In this respect, it can be compared to a docker build albeit with improved performance, storage management, feature functionality, and security. The most significant advantage BuildKit has is the ability to execute build steps in parallel, thanks to its concurrent dependency solver.
BuildKit is also highly configurable, in terms of build definition formats (frontends), output formats (direct to the registry, tarballs, cache warming), and cache imports/exports.
BuildKit builds are based on a low-level, binary intermediate format called LLB (low-level builder). The entire dependency graph of your build, how to execute, and what to cache is defined in LLB.
A frontend is a component that takes a human-readable build definition and converts it to LLB so that BuildKit can execute it. The most well-known frontend is the Dockerfile frontend. (
Since Docker v18.06, BuildKit has been integrated into docker build and can be enabled by setting the environment variable
DOCKER_BUILDKIT=1. There is also a special frontend called gateway (
gateway.v0) that allows any image to be used as a frontend.
Buildkit-pack was authored by Tonis Tiigi, the maintainer of BuildKit. It is the ‘official’ BuildKit frontend for Buildpacks.
Buildkit-pack has been packaged as an image and pushed to the public Docker Hub registry. The last commit on the repository was 3 years ago.
Although not explicitly stated in the README, the ‘high-level’ build definition supported by this frontend is actually a deprecated version of the CloudFoundry manifest. Within the manifest, you can specify the buildpack to use to build your application.
With Docker v18.06+:
DOCKER_BUILDKIT=1 docker build -f manifest.yml .
Where the manifest here contains
# syntax = tonistiigi/pack as the first line, BuildKit detects the syntax keyword and converts it into a source opt.
buildctl build --frontend=gateway.v0 --opt source=tonistiigi/pack --local context=.
You will still require a
manifest.yml. But in this case, you can go without the
# syntax = tonistiigi/pack annotation.
BuildKit then starts a container with the
tonistiigi/pack image and the build opts you pass to the
Looking at the buildkit-pack source code, you’ll find the following directory structure:
cmd hack vendor .travis.yml Dockerfile LICENSE README.md build.go manifest.go vendor.conf
It’s actually a fairly small codebase with the important logic contained within a few files.
This Dockerfile is used to package the frontend as an image. It essentially compiles the Golang code into a single binary, pack. This executable is copied to bin/pack and set as the
ENTRYPOINT of the runtime image.
Contains logic to parse a CloudFoundry manifest and for the following keys:
applications, buildpack, command, env. And for each application, the following keys are parsed:
name, buildpack, command, env.
The implication here is that buildkit-pack essentially limits us to a single buildpack per application. We can’t specify a custom builder, which is a huge drawback, as it means developers cannot simply write application code. They also have to worry about buildpack selection.
Note that is it possible to not specify your buildpack and allow auto-detection to select one of the CloudFoundry system buildpacks.
Contains the bulk of the build definition transformation logic. It parses the build opts invoked with the bin/pack executable along with the manifest. It then uses the BuildKit LLB client to programmatically construct the LLB. The Builder design pattern is used here to assemble the LLB.
There are 3 stages (
build, extract, run), each of them starts a container with a given base image and executes some instructions. The stages are nested one after another to generate the LLB.
The overall call flow is as follows.
docker.io/packs/cflinuxfs2:build) as the root of the
build stagewith the env vars.
build stagewith a
RUNcommand. This command runs the
/packs/builderexecutable and outputs a tarball.
/packs/builderis already contained in the build image and looks to be the CloudFoundry equivalent of the Pack CLI. This is in fact a compatibility layer written for CloudFoundry that implements the Buildpacks lifecycle.
/srcdirectory on the
/tmpdirectory on the
build stageto the
/indirectory on the
extract stagewith a
RUNcommand. This command basically extracts the tarball from the build stage to the
docker.io/packs/cflinuxfs2:run) as the root of the
/outdirectory of the
extract stageto the
SolveRequestto the buildkit daemon
ResolveImageConfigRequestto the buildkit daemon
There are, unfortunately, some issues with this implementation:
Buildkit-pack is no longer being maintained, and as we have seen, it is not a generic implementation for a Buildpacks frontend.
It is rather specific to CloudFoundry and does not support critical features such as providing a builder and using project descriptors. In fact, it was only meant to be a proof of concept to demonstrate the flexibility of BuildKit.
Previously published here.