Iterative Terraform Development with Skaffold and Kubernetes

Kubernetes and Terraform are two well known foundational open source instruments for building infrastructure. They are very different by nature and application scenarios but also have lots similarities and intersections. I’m going to skip philosophical aspects and show practical example of the two technologies can augment each other.

What is Common?

• Multipurpose and span infrastructure, applications and services.
• Provide abstraction and hide complexity of underlying infrastructure.
• Cloud agnostic and enable hybrid cloud development.
• Desired state and allow to focus on the end result, bypassing interim steps programming.

Development Experience

Terraform Configurations vs Kubernetes Manifests

At the lowest level, working with both tools assumes writing DSL manifests in YAML/HCL or JSON, verifying and submiting them for execution with terraform or kubectl cli tools. While this approach is good for ad-hock tasks or as a “quickstart”, it lacks iterative deployment, stacks lifecycle management and 3rd party automation reuse.

Modules and Helm packages

The above limitation is partially addressed by using workspaces and modules with public registry for terraform. For Kubernetes, it’s improved by using helm with numerous charts available. However, development on this level, assumes spending lot’s of manual efford in order to get things done and enable continuous deployment.

Out of the box CI/CD

That’s the most interesting part, where terraform opensource ecosystem is missing a lot comparing to kubernetes and you need to make your hands dirty to get basic CI/CD.

Some examples:

• Managing Secretes is critical since you need to communicate with different cloud platform APIs and you cannot keep sensitive data in the code. While enabled in Kubernetes by design via encrypted secrets, vault integration and service catalog, you spend some valuable time integrating into your CI for terraform.
• Out of the box automated continuous development workflow for dev and prod. Simply saying, terraform is missing “Day 0" tools similar to skaffold, draft, gitkube or even metaparticle, which implement typical development flows and form a basement of you pipelines.

What if we could use Kubernetes tools for Terraform?

Apart from levereging the largest and biggest ecosystem, we could unify tooling and development experience for the both, hence, minimize maintenance efforts.

Prototype

I have created draft pack, which illustrates the above and automates terraform development using skaffold, helm and has built-in Packer, Vault integrations. Essentially, each Terraform change results into an upgrade of helm release and it can be easily integrated with Jenkins X.

Jenkins X Terraform pipeline reference architecture

Quickstart

Prerequesites: installed draft, skaffold, helm, kubernetes

Clone repository and install draft pack:

git clone https://github.com/odzhu/infrapack.gitcd infrapackcd tests/ && make addpack

Run Packer and Terraform build and deploy against sample code using skaffold

make test

Under the hood, it will:

1. Create temp dir and copy sample terraform and packer code the.
2. Execure draft create and enreaching the code with helm charts.
3. Build and deploy.

NAME: infrapackLAST DEPLOYED: Wed May 30 18:58:11 2018NAMESPACE: defaultSTATUS: DEPLOYED

RESOURCES:==> v1/ServiceAccountNAME SECRETS AGEinfrapack-infrapack 1 0s

==> v1/RoleNAME AGEstate-writer-infrapack 0s

==> v1/RoleBindingNAME AGEstate-writer-infrapack 0s

==> v1/JobNAME DESIRED SUCCESSFUL AGEinfrapack-infrapack1 1 0 0s

==> v1/Pod(related)NAME READY STATUS RESTARTS AGEinfrapack-infrapack1-97r8t 0/1 Pending 0 0s

==> v1/SecretNAME TYPE DATA AGEstate-infrapack Opaque 1 0s

==> v1/ConfigMapNAME DATA AGEscripts-infrapack1 3 0s

Deploy complete in 585.116428ms0 tests$ helm listNAME REVISION UPDATED STATUS CHART NAMESPACEinfrapack 1 Wed May 30 18:58:11 2018 DEPLOYED hcl-v0.1.0 default

Switch to temp dir and start skaffold.

cd /tmp/sandbox/ && skaffold dev

The environment is ready for sandboxing, any code change will be automatically applied and you can inspect helm release for the details!