I wanted to have my setup inside of a container so that all of its dependencies were isolated from any Ruby installs on my host and to allow me to easily run different versions side by side for testing upgrades. ChefDK Docker So I started with a basic Dockerfile to install ChefDK… from ubuntu:16.04 RUN apt-get update RUN apt-get install -y curlRUN curl -O RUN dpkg -i chefdk_2.4.17-1_amd64.deb https://packages.chef.io/files/stable/chefdk/2.4.17/ubuntu/16.04/chefdk_2.4.17-1_amd64.deb If we build this and then get a bash shell inside it … docker build -t chefdk .docker run -it chefdk bash We have Chef and Knife installed! root@6bb552c50752:/# which chef/usr/bin/chefroot@6bb552c50752:/# which knife/usr/bin/knife Great, now I’ll need to mount my cookbooks inside of the container so I can run test-kitchen, I’ll mount them at /cookbooks docker run -it -v /Users/aaronkalair/cookbooks:/cookbooks chefdk That works but as we didn’t specify a user in the Dockerfile we’re root and so every new file is also owned by root inside the container. root@d374bab13add:/cookbooks/teabot# touch test.txtroot@d374bab13add:/cookbooks/teabot# ls -lah test.txt-rw-r--r-- 1 root root 0 Dec 23 15:28 test.txt To avoid any issues here lets make a user inside the container that has the same UID as my user on the host… On the host I’m UID 501 and this is consistent across the 3 machines I tested it on so it seems that Mac assigns 501 to the first user account created. Aarons-iMac:chefdk aaronkalair$ iduid=501(aaronkalair) ... So we can make a user called inside the container with by adding this to the end of our Dockerfile aaronkalair uid 501 RUN useradd -u 501 aaronkalairRUN mkdir -p /home/aaronkalairRUN chown aaronkalair:aaronkalair /home/aaronkalairUSER aaronkalair And now let’s test spinning up a test kitchen for my teabot cookbook. First we’ll need to install the gems root@6c602759d146:/cookbooks/teabot# bundle installbash: bundle: command not found Hmm we didn’t install Ruby or any of its associated tools, but we shouldn’t need to because Chef comes with an embeded version of them. root@6c602759d146:/cookbooks/teabot# ls /opt/chefdk/embedded/bin/ ... <124 packages including, bundle, gem, ruby etc... > So lets drop that into our path by adding this to our Dockerfile ENV PATH="/opt/chefdk/embedded/bin:${PATH}" And now lets try the again. bundle install As I use different versions of Gems in different cookbooks lets just install them into folders in this cookbook rather than having a mish mash of different versions installed globally depending on the cookbook I last worked on bundle install --path vendor --binstubs Unfortunately this explodes with an error about extconf failing Gem::Ext::BuildError: ERROR: Failed to build gem native extension. current directory: /cookbooks/teabot/vendor/ruby/2.4.0/gems/libyajl2-1.2.0/ext/libyajl2/opt/chefdk/embedded/bin/ruby -r ./siteconf20171223-8-1u0q211.rb extconf.rbcreating Makefile/cookbooks/teabot/vendor/ruby/2.4.0/gems/libyajl2-1.2.0/ext/libyajl2extconf.rb:104:in `makemakefiles': unhandled exceptionfrom extconf.rb:138:in `<main>' extconf failed, exit code 1 Native extensions, Makefiles and extconf sounds like we’re missing some dependencies to install C dependencies, lets grab and see what happens and whilst we’re editing the Dockerfile lets alias the bundle command. build-essentials RUN apt-get install -y curl build-essential RUN echo "alias bundle-install='/opt/chefdk/embedded/bin/bundle install --path vendor --binstubs'" >> /home/aaronkalair/.bashrc There we go that worked aaronkalair@3ac49ab61513:/cookbooks/teabot$ bundle-install....Bundle complete! 3 Gemfile dependencies, 104 gems now installed.Bundled gems are installed into ./vendor. Now we need Docker available from within the container so kitchen-docker can make containers. The easiest way to do this appears to be to mount the socket from the host into the container . So let's change the run command to: docker run -it -v /Users/aaronkalair/cookbooks:/cookbooks -v chefdk /var/run/docker.sock:/var/run/docker.sock And install the Docker cli inside the container RUN apt-get install -y curl build-essential docker.io This gets us access to the Docker CLI inside the container but it doesn’t appear to be able to talk to the Deamon on the host aaronkalair@4280bf11224b:/$ docker psGot permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get : dial unix /var/run/docker.sock: connect: permission denied http://%2Fvar%2Frun%2Fdocker.sock/v1.26/containers/json Looking at its permissions reveals why aaronkalair@4280bf11224b:/$ ls -lah /var/run/docker.socksrw-rw---- 1 root staff 0 Dec 22 11:32 /var/run/docker.sock Only the user or users in the group can read and write from the socket, I’m going to fix this by adding my user to the staff group. Be careful what you do here though as anyone with access to the Docker socket has root access to the host. root staff aaronkalair RUN usermod -a -G staff aaronkalair And now we can talk to the Docker deamon aaronkalair@43db476d04f7:/$ docker psCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES43db476d04f7 chefdk "/bin/bash" 2 seconds ago Up 1 second upbeat_panini So let's try and converge our test kitchen! aaronkalair@43db476d04f7:/cookbooks/teabot$ ./bin/kitchen converge....Lots of Good Things... Failed to complete #create action: [Cannot assign requested address - connect(2) for [::1]:32776] on default-ubuntu-1604 That’s odd, isn’t much more helpful just saying kitchen list Errno::EADDRNOTAVAIL Have I just gotten really unlucky and something else has used 32776 for its empheral port? Lets install and have a look. netstat aaronkalair@43db476d04f7:/cookbooks/teabot$ apt-get install net-tools E: Could not open lock file /var/lib/dpkg/lock - open (13: Permission denied)E: Unable to lock the administration directory (/var/lib/dpkg/), are you root? Oops, we’ll need to pop into the container as root to debug this docker run -it -v /Users/aaronkalair/cookbooks:/cookbooks -v /var/run/docker.sock:/var/run/docker.sock chefdk -u root And then shows just one connection on port netstat 50332 root@e075956d7a2b:/# netstatActive Internet connections (w/o servers)Proto Recv-Q Send-Q Local Address Foreign Address Statetcp 0 0 e075956d7a2b:50332 keeton.canonical.c:http TIME_WAITActive UNIX domain sockets (w/o servers)Proto RefCnt Flags Type State I-Node Path Interesting, we are doing some weird Docker inside of Docker thing, maybe its in use on the host? Aarons-iMac:~ aaronkalair$ netstat | grep 32776 Nope not being used there either. Can we bind to port ourselves outside of kitchen? 32776 root@e075956d7a2b:/# nc -l 32776 root@e075956d7a2b:/# nc -l 0.0.0.0 32776 Yep, that all works fine Can we this magical thing that’s hogging port ? curl 32776 root@e075956d7a2b:/# curl -v localhost:32776* Rebuilt URL to: localhost:32776/* Trying 127.0.0.1...* TCP_NODELAY set* connect to 127.0.0.1 port 32776 failed: Connection refused* Trying ::1...* TCP_NODELAY set*** Immediate connect fail for ::1: Cannot assign requested address* Trying ::1...*** TCP_NODELAY set* Immediate connect fail for ::1: Cannot assign requested address* Failed to connect to localhost port 32776: Connection refused* Closing connection 0curl: (7) Failed to connect to localhost port 32776: Connection refused That’s interesting, gives us the same error as from test-kitchen, is this some weird IPv6 thing? ::1: A quick Google confirms that IPv6 only works in Docker if you specify a special flag — https://docs.docker.com/engine/userguide/networking/default_network/ipv6/ By default, the Docker daemon configures the container network for IPv4 only. You can enable IPv4/IPv6 dualstack support by running the Docker daemon with the flag. --ipv6 Ok well I don’t actually need IPv6 so how do I stop it trying to use it? Well is defined in as localhost /etc/hosts root@e075956d7a2b:/# cat /etc/hosts127.0.0.1 localhost::1 localhost ip6-localhost ip6-loopbackfe00::0 ip6-localnetff00::0 ip6-mcastprefixff02::1 ip6-allnodesff02::2 ip6-allrouters172.17.0.2 e075956d7a2b So if I just nuke those IPv6 lines maybe it will be fine? (Sidenote: Apparently editing can alter the IPv6 before IPv4 behaviour — ) /etc/gai.conf https://askubuntu.com/a/38468 root@e075956d7a2b:/# vi /etc/hostsroot@e075956d7a2b:/# cat /etc/hosts127.0.0.1 localhost172.17.0.2 e075956d7a2b And lets try to converge again (with debug logging incase its helpful) root@e075956d7a2b:/cookbooks/teabot# ./bin/kitchen converge -l debug...D [SSH] opening connection to kitchen@localhost<{:user_known_hosts_file=>"/dev/null", :port=>32776, :compression=>false, :compression_level=>0, :keepalive=>true, :keepalive_interval=>60, :timeout=>15, :keys_only=>true, :keys=>["/cookbooks/teabot/.kitchen/docker_id_rsa"], :auth_methods=>["publickey"], :verify_host_key=>false}>$$$$$$ [SSH] connection failed, terminating (#<Errno::ECONNREFUSED: Connection refused - connect(2) for 127.0.0.1:32776>) Alright then that confirms the issue above was from trying to use IPv6 in a Docker environment not configured for it. Now we have a new issue, the SSH connection to port is being refused. 32776 Is anything actually listening on that port locally? Back to our friend netstat root@e075956d7a2b:/cookbooks/teabot# netstat --listeningActive Internet connections (only servers)Proto Recv-Q Send-Q Local Address Foreign Address StateActive UNIX domain sockets (only servers)Proto RefCnt Flags Type State I-Node Path Nope. Has test-kitchen actually managed to make a container running an SSH deamon? root@e075956d7a2b:/cookbooks/teabot# docker psCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES9ca4e56ab0b4 2fe546b254ce "/usr/sbin/sshd -D..." 56 minutes ago Up About an hour 0.0.0.0:32776->22/tcp defaultubuntu1604-nologin-43db476d04f7-hds4zcup Certainly looks like it, if we exec into that container is actually running? sshd root@e075956d7a2b:/cookbooks/teabot# docker exec -it 9ca4e56ab0b4 bashroot@9ca4e56ab0b4:/# ps -efUID PID PPID C STIME TTY TIME CMDroot 1 0 0 16:17 ? 00:00:00 /usr/sbin/sshd -D -o UseDNS=no -o UsePAM=no -o PasswordAuthentication=yesroot 6 0 0 17:15 pts/0 00:00:00 bashroot 16 6 0 17:15 pts/0 00:00:00 ps -ef Yep! And looking at the output of above we see where came from, its the port that 22 in the container is mapped to on the host. docker ps 32776 But we’re not on the host machine we’re inside a container on Dockers own network. Ok so here’s where we are: kitchen-docker has made a container that it can run inside but it assumes we’re running on the host machine and there will be listening at the port Docker has mapped onto, on chef sshd 22 localhost We’re inside a Docker container on the network Docker has made, not the host, and therefore can’t use the port mapping and no matter what port it's on its not on localhost for us 22 -> 32776 Fortunately all of these problems should be solvable, we’re on the same network as the container running so we should be able to talk to IP via its IP address. sshd root@e075956d7a2b:/cookbooks/teabot# docker inspect 9ca4e56ab0b4 | grep -i ipaddress"SecondaryIPAddresses": null,"IPAddress": "172.17.0.3","IPAddress": "172.17.0.3", A quick sanity check to see if we can talk to it In the container created by kitchen-docker root@9ca4e56ab0b4:/# nc -l -p 5000PING^C In the container running chefdk root@e075956d7a2b:/cookbooks/teabot# telnet 172.17.0.3 5000Trying 172.17.0.3...Connected to 172.17.0.3.Escape character is '^]'.PING Excellent so that works, now to modify kitchen-docker The port and hostname are set here — https://github.com/test-kitchen/kitchen-docker/blob/master/lib/kitchen/driver/docker.rb#L124 So let's just quickly hack the IP address and port 22 into there to check it works root@e075956d7a2b:/cookbooks/teabot# vi vendor/ruby/2.4.0/gems/kitchen-docker-2.6.0/lib/kitchen/driver/docker.rb <some quick edits> And lets see if that works: root@e075956d7a2b:/cookbooks/teabot# ./bin/kitchen converge -l debug... D [SSH] kitchen@172.17.0.3<{:user_known_hosts_file=>"/dev/null", :port=>"22", :compression=>false, :compression_level=>0, :keepalive=>true, :keepalive_interval=>60, :timeout=>15, :keys_only=>true, :keys=>["/cookbooks/teabot/.kitchen/docker_id_rsa"], :auth_methods=>["publickey"], :verify_host_key=>false}> (sudo -E sh -c ' [SSH] Established... There we go it works! I wrote this patch for kitchen-docker to do the above without hardcoding in the IP address — https://github.com/test-kitchen/kitchen-docker/pull/283/files (Sidenote: A really easy way to get the IP address of a Docker container given its ID is) docker inspect --format '{{ .NetworkSettings.IPAddress }}' <container id> So there we go you can run ChefDK and kitchen-docker inside of a Docker container (with some small changes to kitchen-docker). To finish this off I installed inside the container, and mounted my , and config from the host into the container so I don’t have to leave the container to switch between editing and testing. vim vim git chef (Although the cookbooks are mounted from the host so you could just edit them on the host and then switch to the container to test if you wanted) You can see the final Dockerfile and run commands on my Github repo here — https://github.com/AaronKalair/docker_apps/tree/master/chefdk Follow me on Twitter @AaronKalair
