Can you run ChefDK and kitchen-docker inside of a Docker container?

I wanted to have my ChefDK setup inside of a Docker container so that all of its dependencies were isolated from any Ruby installs on my host and to allow me to easily run different versions side by side for testing upgrades.

So I started with a basic Dockerfile to install ChefDK…

from ubuntu:16.04

RUN apt-get update

RUN apt-get install -y curlRUN curl -O https://packages.chef.io/files/stable/chefdk/2.4.17/ubuntu/16.04/chefdk_2.4.17-1_amd64.debRUN dpkg -i chefdk_2.4.17-1_amd64.deb

If we build this and then get a bash shell inside it …

docker build -t chefdk .docker run -it chefdk bash

We have Chef and Knife installed!

root@6bb552c50752:/# which chef/usr/bin/chefroot@6bb552c50752:/# which knife/usr/bin/knife

Great, now I’ll need to mount my cookbooks inside of the container so I can run test-kitchen, I’ll mount them at /cookbooks

docker run -it -v /Users/aaronkalair/cookbooks:/cookbooks chefdk

That works but as we didn’t specify a user in the Dockerfile we’re root and so every new file is also owned by root inside the container.

root@d374bab13add:/cookbooks/teabot# touch test.txtroot@d374bab13add:/cookbooks/teabot# ls -lah test.txt-rw-r--r-- 1 root root 0 Dec 23 15:28 test.txt

To avoid any issues here lets make a user inside the container that has the same UID as my user on the host…

On the host I’m UID 501 and this is consistent across the 3 machines I tested it on so it seems that Mac assigns 501 to the first user account created.

Aarons-iMac:chefdk aaronkalair$ iduid=501(aaronkalair) ...

So we can make a user called aaronkalair inside the container with uid 501 by adding this to the end of our Dockerfile

RUN useradd -u 501 aaronkalairRUN mkdir -p /home/aaronkalairRUN chown aaronkalair:aaronkalair /home/aaronkalairUSER aaronkalair

And now let’s test spinning up a test kitchen for my teabot cookbook. First we’ll need to install the gems

root@6c602759d146:/cookbooks/teabot# bundle installbash: bundle: command not found

Hmm we didn’t install Ruby or any of its associated tools, but we shouldn’t need to because Chef comes with an embeded version of them.

root@6c602759d146:/cookbooks/teabot# ls /opt/chefdk/embedded/bin/

... <124 packages including, bundle, gem, ruby etc... >

So lets drop that into our path by adding this to our Dockerfile

ENV PATH="/opt/chefdk/embedded/bin:${PATH}"

And now lets try the bundle install again.

As I use different versions of Gems in different cookbooks lets just install them into folders in this cookbook rather than having a mish mash of different versions installed globally depending on the cookbook I last worked on

bundle install --path vendor --binstubs

Unfortunately this explodes with an error about extconf failing

Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

current directory: /cookbooks/teabot/vendor/ruby/2.4.0/gems/libyajl2-1.2.0/ext/libyajl2/opt/chefdk/embedded/bin/ruby -r ./siteconf20171223-8-1u0q211.rb extconf.rbcreating Makefile/cookbooks/teabot/vendor/ruby/2.4.0/gems/libyajl2-1.2.0/ext/libyajl2extconf.rb:104:in `makemakefiles': unhandled exceptionfrom extconf.rb:138:in `<main>'

extconf failed, exit code 1

Native extensions, Makefiles and extconf sounds like we’re missing some dependencies to install C dependencies, lets grab build-essentials and see what happens and whilst we’re editing the Dockerfile lets alias the bundle command.

RUN apt-get install -y curl build-essential

RUN echo "alias bundle-install='/opt/chefdk/embedded/bin/bundle install --path vendor --binstubs'" >> /home/aaronkalair/.bashrc

There we go that worked

aaronkalair@3ac49ab61513:/cookbooks/teabot$ bundle-install....Bundle complete! 3 Gemfile dependencies, 104 gems now installed.Bundled gems are installed into ./vendor.

Now we need Docker available from within the container so kitchen-docker can make containers. The easiest way to do this appears to be to mount the socket from the host into the container .

So let's change the run command to:

docker run -it -v /Users/aaronkalair/cookbooks:/cookbooks -v /var/run/docker.sock:/var/run/docker.sock chefdk

And install the Docker cli inside the container

RUN apt-get install -y curl build-essential docker.io

This gets us access to the Docker CLI inside the container but it doesn’t appear to be able to talk to the Deamon on the host

aaronkalair@4280bf11224b:/$ docker psGot permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.26/containers/json: dial unix /var/run/docker.sock: connect: permission denied

Looking at its permissions reveals why

aaronkalair@4280bf11224b:/$ ls -lah /var/run/docker.socksrw-rw---- 1 root staff 0 Dec 22 11:32 /var/run/docker.sock

Only the root user or users in the staff group can read and write from the socket, I’m going to fix this by adding my aaronkalair user to the staff group. Be careful what you do here though as anyone with access to the Docker socket has root access to the host.

RUN usermod -a -G staff aaronkalair

And now we can talk to the Docker deamon

aaronkalair@43db476d04f7:/$ docker psCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES43db476d04f7 chefdk "/bin/bash" 2 seconds ago Up 1 second upbeat_panini

So let's try and converge our test kitchen!

aaronkalair@43db476d04f7:/cookbooks/teabot$ ./bin/kitchen converge....Lots of Good Things...Failed to complete #create action: [Cannot assign requested address - connect(2) for [::1]:32776] on default-ubuntu-1604

That’s odd, kitchen list isn’t much more helpful just saying Errno::EADDRNOTAVAIL

Have I just gotten really unlucky and something else has used 32776 for its empheral port? Lets install netstatand have a look.

aaronkalair@43db476d04f7:/cookbooks/teabot$ apt-get install net-tools

E: Could not open lock file /var/lib/dpkg/lock - open (13: Permission denied)E: Unable to lock the administration directory (/var/lib/dpkg/), are you root?

Oops, we’ll need to pop into the container as root to debug this

docker run -it -u root -v /Users/aaronkalair/cookbooks:/cookbooks -v /var/run/docker.sock:/var/run/docker.sock chefdk

And then netstat shows just one connection on port 50332

root@e075956d7a2b:/# netstatActive Internet connections (w/o servers)Proto Recv-Q Send-Q Local Address Foreign Address Statetcp 0 0 e075956d7a2b:50332 keeton.canonical.c:http TIME_WAITActive UNIX domain sockets (w/o servers)Proto RefCnt Flags Type State I-Node Path

Interesting, we are doing some weird Docker inside of Docker thing, maybe its in use on the host?

Aarons-iMac:~ aaronkalair$ netstat | grep 32776

Nope not being used there either.

Can we bind to port 32776 ourselves outside of kitchen?

root@e075956d7a2b:/# nc -l 32776

root@e075956d7a2b:/# nc -l 0.0.0.0 32776

Yep, that all works fine

Can we curl this magical thing that’s hogging port 32776?

root@e075956d7a2b:/# curl -v localhost:32776* Rebuilt URL to: localhost:32776/* Trying 127.0.0.1...* TCP_NODELAY set* connect to 127.0.0.1 port 32776 failed: Connection refused* Trying ::1...* TCP_NODELAY set*** Immediate connect fail for ::1: Cannot assign requested address* Trying ::1...*** TCP_NODELAY set* Immediate connect fail for ::1: Cannot assign requested address* Failed to connect to localhost port 32776: Connection refused* Closing connection 0curl: (7) Failed to connect to localhost port 32776: Connection refused

That’s interesting, ::1: gives us the same error as from test-kitchen, is this some weird IPv6 thing?

A quick Google confirms that IPv6 only works in Docker if you specify a special flag — https://docs.docker.com/engine/userguide/networking/default_network/ipv6/

By default, the Docker daemon configures the container network for IPv4 only. You can enable IPv4/IPv6 dualstack support by running the Docker daemon with the --ipv6flag.

Ok well I don’t actually need IPv6 so how do I stop it trying to use it? Well localhost is defined in /etc/hosts as

root@e075956d7a2b:/# cat /etc/hosts127.0.0.1 localhost::1 localhost ip6-localhost ip6-loopbackfe00::0 ip6-localnetff00::0 ip6-mcastprefixff02::1 ip6-allnodesff02::2 ip6-allrouters172.17.0.2 e075956d7a2b

So if I just nuke those IPv6 lines maybe it will be fine?

(Sidenote: Apparently editing /etc/gai.confcan alter the IPv6 before IPv4 behaviour — https://askubuntu.com/a/38468 )

root@e075956d7a2b:/# vi /etc/hostsroot@e075956d7a2b:/# cat /etc/hosts127.0.0.1 localhost172.17.0.2 e075956d7a2b

And lets try to converge again (with debug logging incase its helpful)

root@e075956d7a2b:/cookbooks/teabot# ./bin/kitchen converge -l debug...D [SSH] opening connection to kitchen@localhost<{:user_known_hosts_file=>"/dev/null", :port=>32776, :compression=>false, :compression_level=>0, :keepalive=>true, :keepalive_interval=>60, :timeout=>15, :keys_only=>true, :keys=>["/cookbooks/teabot/.kitchen/docker_id_rsa"], :auth_methods=>["publickey"], :verify_host_key=>false}>$$$$$$ [SSH] connection failed, terminating (#<Errno::ECONNREFUSED: Connection refused - connect(2) for 127.0.0.1:32776>)

Alright then that confirms the issue above was from trying to use IPv6 in a Docker environment not configured for it.

Now we have a new issue, the SSH connection to port 32776 is being refused.

Is anything actually listening on that port locally? Back to our friend netstat

root@e075956d7a2b:/cookbooks/teabot# netstat --listeningActive Internet connections (only servers)Proto Recv-Q Send-Q Local Address Foreign Address StateActive UNIX domain sockets (only servers)Proto RefCnt Flags Type State I-Node Path

Nope.

Has test-kitchen actually managed to make a container running an SSH deamon?

root@e075956d7a2b:/cookbooks/teabot# docker psCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES9ca4e56ab0b4 2fe546b254ce "/usr/sbin/sshd -D..." 56 minutes ago Up About an hour 0.0.0.0:32776->22/tcp defaultubuntu1604-nologin-43db476d04f7-hds4zcup

Certainly looks like it, if we exec into that container is sshdactually running?

root@e075956d7a2b:/cookbooks/teabot# docker exec -it 9ca4e56ab0b4 bashroot@9ca4e56ab0b4:/# ps -efUID PID PPID C STIME TTY TIME CMDroot 1 0 0 16:17 ? 00:00:00 /usr/sbin/sshd -D -o UseDNS=no -o UsePAM=no -o PasswordAuthentication=yesroot 6 0 0 17:15 pts/0 00:00:00 bashroot 16 6 0 17:15 pts/0 00:00:00 ps -ef

Yep!

And looking at the output of docker ps above we see where 32776 came from, its the port that 22 in the container is mapped to on the host.

But we’re not on the host machine we’re inside a container on Dockers own network.

Ok so here’s where we are:

• kitchen-docker has made a container that it can run chef inside but it assumes we’re running on the host machine and there will be sshd listening at the port Docker has mapped 22 onto, on localhost
• We’re inside a Docker container on the network Docker has made, not the host, and therefore can’t use the port mapping 22 -> 32776 and no matter what port it's on its not on localhost for us

Fortunately all of these problems should be solvable, we’re on the same network as the container running sshd so we should be able to talk to IP via its IP address.

root@e075956d7a2b:/cookbooks/teabot# docker inspect 9ca4e56ab0b4 | grep -i ipaddress"SecondaryIPAddresses": null,"IPAddress": "172.17.0.3","IPAddress": "172.17.0.3",

A quick sanity check to see if we can talk to it

In the container created by kitchen-docker

root@9ca4e56ab0b4:/# nc -l -p 5000PING^C

In the container running chefdk

root@e075956d7a2b:/cookbooks/teabot# telnet 172.17.0.3 5000Trying 172.17.0.3...Connected to 172.17.0.3.Escape character is '^]'.PING

Excellent so that works, now to modify kitchen-docker

The port and hostname are set here — https://github.com/test-kitchen/kitchen-docker/blob/master/lib/kitchen/driver/docker.rb#L124

So let's just quickly hack the IP address and port 22 into there to check it works

root@e075956d7a2b:/cookbooks/teabot# vi vendor/ruby/2.4.0/gems/kitchen-docker-2.6.0/lib/kitchen/driver/docker.rb

<some quick edits>

And lets see if that works:

root@e075956d7a2b:/cookbooks/teabot# ./bin/kitchen converge -l debug...

D [SSH] [email protected]<{:user_known_hosts_file=>"/dev/null", :port=>"22", :compression=>false, :compression_level=>0, :keepalive=>true, :keepalive_interval=>60, :timeout=>15, :keys_only=>true, :keys=>["/cookbooks/teabot/.kitchen/docker_id_rsa"], :auth_methods=>["publickey"], :verify_host_key=>false}> (sudo -E sh -c '

[SSH] Established...

There we go it works!

I wrote this patch for kitchen-docker to do the above without hardcoding in the IP address — https://github.com/test-kitchen/kitchen-docker/pull/283/files

(Sidenote: A really easy way to get the IP address of a Docker container given its ID is)

docker inspect --format '{{ .NetworkSettings.IPAddress }}' <container id>

So there we go you can run ChefDK and kitchen-docker inside of a Docker container (with some small changes to kitchen-docker). To finish this off I installed vim inside the container, and mounted my vim, gitand chefconfig from the host into the container so I don’t have to leave the container to switch between editing and testing.

(Although the cookbooks are mounted from the host so you could just edit them on the host and then switch to the container to test if you wanted)

You can see the final Dockerfile and run commands on my Github repo here — https://github.com/AaronKalair/docker_apps/tree/master/chefdk

Follow me on Twitter @AaronKalair