Cloud-Native Buildpacks, BuildKit, and Container Images: Qu'est-ce Que C'est?

In this article, we’ll do a deep dive into the Buildpacks BuikdKit frontend implementation and see how it actually works under the hood. However, before we do so, we’ll need to breeze through a few concepts. What is a Container Image? A container image of a config file and layer tarballs. Usually, they are made from a series of instructions like FROM, ADD, and RUN, which are placed in a Dockerfile and built with a docker build. However, a docker build is just one way of ending up with this output. consists Cloud-Native Buildpacks Buildpacks belong to the family of non-Dockerfile based image-building tools. It takes your application source code and transforms it into an image, usually without any additional input from the user. OCI To use Buildpacks to package your code, you would run the , specifying your application source code as well as a builder, which is an ordered collection of Buildpacks to be executed. Pack CLI BuildKit BuildKit is defined as a toolkit for converting source code into build artifacts in an efficient, expressive, and repeatable manner. Note that this means that BuildKit is not limited to building OCI images, although it is typically used as such. In this respect, it can be compared to a docker build albeit with improved performance, storage management, feature functionality, and security. The most significant advantage BuildKit has is the ability to execute build steps in parallel, thanks to its concurrent dependency solver. BuildKit is also highly configurable, in terms of build definition formats (frontends), output formats (direct to the registry, tarballs, cache warming), and cache imports/exports. LLB BuildKit builds are based on a low-level, binary intermediate format called LLB (low-level builder). The entire dependency graph of your build, how to execute, and what to cache is defined in LLB. Frontends A frontend is a component that takes a human-readable build definition and converts it to LLB so that BuildKit can execute it. The most well-known frontend is the Dockerfile frontend. ( ). dockerfile.v0 Since Docker v18.06, BuildKit has been integrated into docker build and can be enabled by setting the environment variable . There is also a special frontend called gateway ( ) that allows any image to be used as a frontend. DOCKER_BUILDKIT=1 gateway.v0 How Buildkit-Pack Works Buildkit-pack was authored by Tonis Tiigi, the maintainer of BuildKit. It is the ‘official’ BuildKit frontend for Buildpacks. Buildkit-pack has been packaged as an image and pushed to the public Docker Hub registry. The last commit on the repository was 3 years ago. Source code: https://github.com/tonistiigi/buildkit-pack Image: https://hub.docker.com/r/tonistiigi/pack Although not explicitly stated in the README, the ‘high-level’ build definition supported by this frontend is actually a deprecated version of the CloudFoundry . Within the manifest, you can specify the buildpack to use to build your application. manifest With Docker v18.06+: DOCKER_BUILDKIT=1 docker build -f manifest.yml . Where the manifest here contains as the first line, BuildKit detects the syntax keyword and it into a source opt. # syntax = tonistiigi/pack converts With BuildKit: buildctl build --frontend=gateway.v0 --opt source=tonistiigi/pack --local context=. You will still require a . But in this case, you can go without the annotation. manifest.yml # syntax = tonistiigi/pack BuildKit then a container with the image and the build opts you pass to the command. starts tonistiigi/pack buildctl Deep Diving into the Source Code Looking at the buildkit-pack source code, you’ll find the following directory structure: cmd hack vendor .travis.yml Dockerfile LICENSE README.md build.go manifest.go vendor.conf It’s actually a fairly small codebase with the important logic contained within a few files. Dockerfile This Dockerfile is used to package the frontend as an image. It essentially compiles the Golang code into a single binary, pack. This executable is copied to bin/pack and set as the of the runtime image. ENTRYPOINT manifest.go Contains logic to parse a CloudFoundry manifest and for the following keys: . And for each application, the following keys are parsed: . applications, buildpack, command, env name, buildpack, command, env The implication here is that buildkit-pack essentially limits us to a single buildpack per application. We can’t specify a custom builder, which is a huge drawback, as it means developers cannot simply write application code. They also have to worry about buildpack selection. Note that is it possible to not specify your buildpack and allow auto-detection to select one of the CloudFoundry buildpacks. system build.go Contains the bulk of the build definition transformation logic. It parses the build opts invoked with the bin/pack executable along with the manifest. It then uses the BuildKit LLB client to programmatically construct the LLB. The design pattern is used here to assemble the LLB. Builder There are 3 stages ( ), each of them starts a container with a given base image and executes some instructions. The stages are nested one after another to generate the LLB. build, extract, run The overall call flow is as follows. Constructing the LLB: Parse the build opts. Parse the manifest to obtain the buildpack, start command, and env vars. Resolve the local context (application source code). Resolve the build image (hardcoded as: ) as the root of the docker.io/packs/cflinuxfs2:build build stage. Augment the with the env vars. build stage Augment the with a command. This command runs the executable and outputs a tarball. is already contained in the build image and looks to be the CloudFoundry equivalent of the Pack CLI. This is in fact a written for CloudFoundry that implements the Buildpacks lifecycle. build stage RUN /packs/builder /packs/builder compatibility layer Mount the application source code to the directory on the /src build stage. Mount a cache to the directory on the /tmp build stage. Resolve an alpine image as the root of the extract stage. Mount the to the directory on the build stage /in extract stage. Augment the with a command. This command basically extracts the tarball from the build stage to the directory. extract stage RUN /out Resolve the run image (hardcoded as ) as the root of the docker.io/packs/cflinuxfs2:run run stage. Mount the directory of the to the /out extract stage run stage. Executing the LLB: Converts the LLB to protobuf Sends a to the buildkit daemon SolveRequest Sends a to the buildkit daemon ResolveImageConfigRequest Returns the SolveResponse There are, unfortunately, some issues with this implementation: Only works with local context, and does not support git / http . sources The stack (build image and run image) is pretty much . hardcoded Although the CloudFoundry manifest supports multiple applications (either for a monorepo or multiple process types), only the first application is taken into . consideration The command use to execute the build is not the Cloud Native Buildpacks Pack CLI, but rather, it is a CloudFoundry specific binary. Conclusion Buildkit-pack is no longer being maintained, and as we have seen, it is not a generic implementation for a Buildpacks frontend. It is rather specific to CloudFoundry and does not support critical features such as providing a builder and using project descriptors. In fact, it was only meant to be a to demonstrate the flexibility of BuildKit. proof of concept Previously published . here

Cloud-Native Buildpacks, BuildKit, and Container Images: Qu'est-ce Que C'est?

Too Long; Didn't Read

Company Mentioned

Coin Mentioned

@sssslyz

RELATED STORIES