A Step-By-Step Guide to Test Chef Using Test Kitchen with Docker
- Reference of this article: https://fuhton.com/Testing-Chef-with-Docker/, as I encountered many issues when setting up and testing, I decided to write my own manual.
- The workstation used is an air-gapped rhel server, so the procedures become complicated, it would be much easier if your workstation has internet access.
- A simple method has been added at the end of this article.
- For more resources like this, please check https://medium.com/@yangpeng_tech
Prerequisite
A rhel server with internal yum repo configured, the following packages are also installed on this server.
- Chef Workstation 1.0.11
- Docker CE 18.09.6
- Gems: kitchen-docker 2.9.0
$ chef --version
Chef Workstation version: 1.0.11
Chef Infra Client version: 15.2.20
Chef InSpec version: 4.10.4
Test Kitchen version: 2.2.5
Foodcritic version: 16.1.1
Cookstyle version: 0.72.0
$ docker version
Client:
Version: 18.09.6
API version: 1.39
Go version: go1.10.8
Git commit: 481bc77156
Built: Sat May 4 02:34:58 2019
OS/Arch: linux/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 18.09.6
API version: 1.39 (minimum version 1.12)
Go version: go1.10.8
Git commit: 481bc77
Built: Sat May 4 02:02:43 2019
OS/Arch: linux/amd64
Experimental: false
# chef gem list kitchen-docker
*** LOCAL GEMS ***
kitchen-docker (2.9.0)
Setup and Test
1. Prepare cookbook
$ chef generate cookbook cookbook-test
$ cd cookbook-test
$ ls
CHANGELOG.md chefignore kitchen.yml LICENSE metadata.rb Policyfile.rb README.md recipes spec test
$ mkdir -p files/default
$ mkdir -p test/recipes
$ cat > files/default/log.sh << EOF
#!/bin/bash
echo "this is the log.sh"
EOF
$ cat > recipes/default.rb << EOF
directory "/etc/testbook" do
owner "root"
group "root"
mode "0755"
action :create
end
cookbook_file "/etc/testbook/log.sh" do
source "log.sh"
group "root"
owner "root"
mode "0644"
end
EOF
$ cat > test/recipes/default_test.rb << EOF
describe directory("/etc/testbook") do
it { should exist }
end
describe file("/etc/testbook/log.sh") do
it { should exist }
its('content') { should match /echo "this is the log.sh"/ }
end
EOF
2. Prepare Policyfile.rb
In earlier version of Chef Workstion, the Policyfile.rb will not be generated, instead, Berksfile will be generated. In that case, generate the policy and modify it with: chef generate policyfile. Then we run chef install to generate Policyfile.lock.json.
$ vi Policyfile.rb (comment the following line as we cannot connect to the internet and don't use external cookbooks for now)
default_source :supermarket
$ chef install
3. Prepare files required for container
- CentOS-Base.repo: copied from workstation to container, used to run yum install later
- install-chef-client.sh: self-made script, configure artifactory repo and install chef from there
- other files: the keys and certs to make sure CentOS-Base.repo works in container
$ cd cookbook-test
$ mkdir file4container
$ cp /etc/yum.repos.d/redhat.repo CentOS-Base.repo
$ vi install-chef-client.sh <--yum install chef via internal rpm artifactory
$ cp /etc/pki/entitlement/* . <--keys and certs used in CentOS-Base.repo
$ cp /etc/rhsm/ca/katello-server-ca.pem .
4. Pull internal CentOS docker image
# docker pull dtr.lxc.example.com/centos/centos7
# docker images | grep -e centos7 -e TAG
REPOSITORY TAG IMAGE ID CREATED SIZE
dtr.lxc.example.com/centos/centos7 latest 5182e96772bf 12 months ago 200MB
5. Create Dockerfile
As our workstation is air gapped, we use our own Dockerfile instead. We need to change highlighted part below.
$ cat > Dockerfile << EOF
FROM dtr.lxc.example/centos/centos7:latest
COPY file4container/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo
COPY file4container/*.pem /etc/pki/entitlement/
COPY file4container/katello-server-ca.pem /etc/rhsm/ca/katello-server-ca.pem
RUN yum clean all
EOF
6. Edit kitchen.yml
The file name was .kitchen.yml in older version of Chef Workstation.
The run_list described in this file will be ignored, so we don’t write it, kitchen instead uses the run list in Policyfile.rb.
---
driver:
name: docker
use_sudo: false
provisioner:
name: chef_zero
always_update_cookbooks: true
client_rb:
chef_license: accept
verifier:
name: inspec
platforms:
- name: centos-7
driver_config:
platform: centos
dockerfile: ./Dockerfile
suites:
- name: default
verifier:
inspec_tests:
- test/recipes
attributes:
7. docker build
We have to use docker command to help us to build a base image for our test.
Use -t option to make a tag for it.
# docker build . -t kitchen-image-base
...
Successfully built 206dbbd70d5c
Successfully tagged kitchen-image-base:latest
8. Run container and execute commands
The reason I need to execute command manually in container is the /etc/hosts gets reset each time when the container is run, and I need to update it to make sure it can solve the hostname specified in CentOS-Base.repo. If there is way to update the /etc/hosts in Dockerfile and maintain it unchanged, the procedure will be much easier. Pass the image ID in previous step to docker run command.
# docker run -it 206dbbd70d5c bash
# cat >> /etc/hosts << EOF
xx.xx.xx.xx example.local
EOF
# yum install -y sudo openssh-server openssh-clients which curl htop # ssh-keygen -A
# ls /etc/ssh/
# mkdir -p /var/run/sshd
# username=kitchen
# password=kitchen
# useradd -d /home/$username -m -s /bin/bash $username
# echo "$username:$password" | chpasswd
# echo "$username ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers
# mkdir -p /home/$username/.ssh
# chown -R $username /home/$username/.ssh
# chmod 0700 /home/$username/.ssh
# touch /home/$username/.ssh/authorized_keys
# chown $username /home/$username/.ssh/authorized_keys
# chmod 0600 /home/$username/.ssh/authorized_keys
!! Do not exit for now as we haven't saved the status yet.
9. Commit current container as new image centos/kitchen-centos:version1
We will be using the new image in our updated Dockerfile later.
# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
f0335b209531 206dbbd70d5c "bash" 7 minutes ago Up 7 minutes condescending_margulis
# docker commit f0335b209531 centos/kitchen-centos:version1
# docker images centos/kitchen-centos
REPOSITORY TAG IMAGE ID CREATED SIZE
centos/kitchen-centos version1 7af2ccd7e480 5 seconds ago 344MB
10. Update Dockerfile to below
FROM centos/kitchen-centos:version1
COPY file4container/install-chef-client.sh /etc/tmp/install-chef-client.sh
RUN bash /etc/tmp/install-chef-client.sh
RUN echo '<%= IO.read(@public_key).strip %>' >> /home/<%= @username %>/.ssh/authorized_keys
RUN ln -sf /usr/share/zoneinfo/Asia/Tokyo /etc/localtime
11. kitchen create
This will create a .kitchen directory in your cookbook. It contains the definition of our VM, the connection keys to SSH in, and our logs.
# kitchen create
...
Successfully built dcb2414cbb86
6e50294dea47b6e7c9b6172f0c9d454b93bda80cf476f428e7de84dea16dce53
0.0.0.0:32774
Waiting for SSH service on localhost:32774, retrying in 3 seconds
[SSH] Established
Finished creating <default-centos-7> (1m19.93s).
-----> Kitchen is finished. (1m30.09s)
# kitchen list
Instance Driver Provisioner Verifier Transport Last Action Last Error
default-centos-7 Docker ChefZero Inspec Ssh Created <None>
12. kitchen converge
To run the recipes in the VM (container), run kitchen converge.
# kitchen converge
Chef Infra Client finished, 2/2 resources updated in 04 seconds
Downloading files from <default-centos-7>
Finished converging <default-centos-7> (0m26.84s).
-----> Kitchen is finished. (0m35.72s)
# kitchen list
Instance Driver Provisioner Verifier Transport Last Action Last Error
default-centos-7 Docker ChefZero Inspec Ssh Converged <None>
13. kitchen verify
To test the recipes, we can run kitchen verify to check. kitchen verify will also call kitchen create and kitchen converge, if those two have been run, it will only run the tests.
# kitchen verify
Target: ssh://kitchen@localhost:32774
Directory /etc/testbook
✔ should exist
File /etc/testbook/log.sh
✔ should exist
✔ content should match /echo "this is the log.sh"/
Test Summary: 3 successful, 0 failures, 0 skipped
Finished verifying <default-centos-7> (0m2.99s).
-----> Kitchen is finished. (0m12.25s)
# kitchen list
Instance Driver Provisioner Verifier Transport Last Action Last Error
default-centos-7 Docker ChefZero Inspec Ssh Verified <None>
We can also ssh to the container and check the file manually:
# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
6e50294dea47 dcb2414cbb86 "/usr/sbin/sshd -D -…" 16 minutes ago Up 16 minutes 0.0.0.0:32774->22/tcp hopeful_lamarr
f0335b209531 206dbbd70d5c "bash" 43 minutes ago Up 43 minutes condescending_margulis
# docker exec -it hopeful_lamarr bash
# ls -lh /etc/testbook/log.sh
-rw-r--r-- 1 root root 39 Aug 13 05:14 /etc/testbook/log.sh
14. kitchen destroy
To destroy the container we created, run kitchen destroy. It will stop the container. The images we created with docker build and docker commit will not be deleted. When we run kitchen create again, it will use the image created by last kitchen create. So we have to manually delete the image if we want a clean one for next kitchen create.
# kitchen destroy
Finished destroying <default-centos-7> (0m2.50s).
-----> Kitchen is finished. (0m11.19s)
# kitchen list
Instance Driver Provisioner Verifier Transport Last Action Last Error
default-centos-7 Docker ChefZero Inspec Ssh <Not Created> <None>
# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
<none> <none> 9a730be48b97 19 minutes ago 585MB
Simple Method
In the above processes, we had to use docker build and docker commit to help us save the current container as new image as we were not able to change the /etc/hosts in container, a workaround has been found in which we can use a single Dockerfile.
- Prepare cookbook — same as above
- Prepare policyfile.rb— same as above
- Prepare files required for container— same as above
- Pull internal CentOS docker image — same as above
- Create Docker file
FROM dtr.lxc.example/centos/centos7:latest
COPY file4container/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo
COPY file4container/<multipleline>.pem /etc/pki/entitlement/<multipleline>.pem
COPY file4container/katello-server-ca.pem /etc/rhsm/ca/katello-server-ca.pem
RUN yum clean all
RUN echo xx.xx.xx.xx exmaple.local >> /etc/hosts; yum install -y sudo openssh-server openssh-clients which curl htop
RUN ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key
RUN ssh-keygen -t dsa -f /etc/ssh/ssh_host_dsa_key
RUN mkdir -p /var/run/sshd
RUN useradd -d /home/<%= @username %> -m -s /bin/bash <%= @username %>
RUN echo <%= "#{@username}:#{@password}" %> | chpasswd
RUN echo '<%= @username %> ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers
RUN mkdir -p /home/<%= @username %>/.ssh
RUN chown -R <%= @username %> /home/<%= @username %>/.ssh
RUN chmod 0700 /home/<%= @username %>/.ssh
RUN touch /home/<%= @username %>/.ssh/authorized_keys
RUN chown <%= @username %> /home/<%= @username %>/.ssh/authorized_keys
RUN chmod 0600 /home/<%= @username %>/.ssh/authorized_keys
COPY file4container/install-chef-client.sh /etc/tmp/install-chef-client.sh
RUN bash /etc/tmp/install-chef-client.sh
RUN echo '<%= IO.read(@public_key).strip %>' >> /home/<%= @username %>/.ssh/authorized_keys
RUN ln -sf /usr/share/zoneinfo/Asia/Tokyo /etc/localtime
6. kitchen create
7. kitchen converge
8. kitchen verify
9. kitchen destroy