Oleksii Dzhulai

@oleksiidzhulai

Iterative Terraform Development with Skaffold and Kubernetes

Kubernetes and Terraform are two well known foundational open source instruments for building infrastructure. They are very different by nature and application scenarios but also have lots similarities and intersections. I’m going to skip philosophical aspects and show practical example of the two technologies can augment each other.

cisco.com

What is Common?

  • Multipurpose and span infrastructure, applications and services.
  • Provide abstraction and hide complexity of underlying infrastructure.
  • Cloud agnostic and enable hybrid cloud development.
  • Desired state and allow to focus on the end result, bypassing interim steps programming.

Development Experience

Terraform Configurations vs Kubernetes Manifests

At the lowest level, working with both tools assumes writing DSL manifests in YAML/HCL or JSON, verifying and submiting them for execution with terraform or kubectl cli tools. While this approach is good for ad-hock tasks or as a “quickstart”, it lacks iterative deployment, stacks lifecycle management and 3rd party automation reuse.

Modules and Helm packages

The above limitation is partially addressed by using workspaces and modules with public registry for terraform. For Kubernetes, it’s improved by using helm with numerous charts available. However, development on this level, assumes spending lot’s of manual efford in order to get things done and enable continuous deployment.

Out of the box CI/CD

That’s the most interesting part, where terraform opensource ecosystem is missing a lot comparing to kubernetes and you need to make your hands dirty to get basic CI/CD.

Some examples:

  • Managing Secretes is critical since you need to communicate with different cloud platform APIs and you cannot keep sensitive data in the code. While enabled in Kubernetes by design via encrypted secrets, vault integration and service catalog, you spend some valuable time integrating into your CI for terraform.
  • Out of the box automated continuous development workflow for dev and prod. Simply saying, terraform is missing “Day 0" tools similar to skaffold, draft, gitkube or even metaparticle, which implement typical development flows and form a basement of you pipelines.

What if we could use Kubernetes tools for Terraform?

Apart from levereging the largest and biggest ecosystem, we could unify tooling and development experience for the both, hence, minimize maintenance efforts.

Prototype

I have created draft pack, which illustrates the above and automates terraform development using skaffold, helm and has built-in Packer, Vault integrations. Essentially, each Terraform change results into an upgrade of helm release and it can be easily integrated with Jenkins X.

Jenkins X Terraform pipeline reference architecture

Quickstart

Prerequesites: installed draft, skaffold, helm, kubernetes

Clone repository and install draft pack:

git clone https://github.com/odzhu/infrapack.git
cd infrapack
cd tests/ && make addpack

Run Packer and Terraform build and deploy against sample code using skaffold

make test

Under the hood, it will:

  1. Create temp dir and copy sample terraform and packer code the.
  2. Execure draft create and enreaching the code with helm charts.
  3. Build and deploy.
NAME:   infrapack
LAST DEPLOYED: Wed May 30 18:58:11 2018
NAMESPACE: default
STATUS: DEPLOYED
RESOURCES:
==> v1/ServiceAccount
NAME SECRETS AGE
infrapack-infrapack 1 0s
==> v1/Role
NAME AGE
state-writer-infrapack 0s
==> v1/RoleBinding
NAME AGE
state-writer-infrapack 0s
==> v1/Job
NAME DESIRED SUCCESSFUL AGE
infrapack-infrapack1 1 0 0s
==> v1/Pod(related)
NAME READY STATUS RESTARTS AGE
infrapack-infrapack1-97r8t 0/1 Pending 0 0s
==> v1/Secret
NAME TYPE DATA AGE
state-infrapack Opaque 1 0s
==> v1/ConfigMap
NAME DATA AGE
scripts-infrapack1 3 0s
Deploy complete in 585.116428ms
0 tests$ helm list
NAME REVISION UPDATED STATUS CHART NAMESPACE
infrapack 1 Wed May 30 18:58:11 2018 DEPLOYED hcl-v0.1.0 default

Switch to temp dir and start skaffold.

cd /tmp/sandbox/ && skaffold dev

The environment is ready for sandboxing, any code change will be automatically applied and you can inspect helm release for the details!

More by Oleksii Dzhulai

Topics of interest

More Related Stories