I heard Chef many times but never took it seriously. Recently I’m learning AWS OpsWorks - the hosted services to use Chef cookbooks and solutions for configration management and automation, and fully stuck by Chef this time. I did some research on Internet and was confused by many terminologies used by Chef, such as kitchen, cookbook, recipe and knife etc, which makes me a little bit distracted. For most of blogs and articles on Internet, they are helpful to review Chef instead of learning it at the first time, I was anxious with learning the new stuff and facing a steep learning curve. After investing the whole two days on learning Ruby – another OOP and explanation language like Python, installing and playing with Chef on my virtual lab environment, finally I was able to understand the core concepts of Chef and move forward with AWS OpsWorks. Learning unknowns is always a painful and sweet journey !
Let’s review the core concepts regarding to Chef before going to the lab and configuration:
Chef, Puppet, Ansible and SaltStack present different paths to achieve a common goal of managing vast numbers of computers efficiently, with minimal input from developers and sysadmins. Chef is one of the most popular confiugration management tools, and it uses Ruby as its reference language to define patterns in recipes and cookbooks to configure, deploy and manage nodes across the network. Chef provides several management mode:
1) Client-Server. The main management mode which contains three components: A Chef Client is an agent that runs locally on every node managed by Chef, it performs the following steps to bring the node into the expected state: registering and authenticating the node with Chef Server, synchronizing cookbooks and configuring the node. The Chef Server acts as a hub for configuration data to store the cookbooks and policies that are applied to nodes, and metadata that describes each registered node that is being managed by Chef Client. Chef Development Kit on Chef Workstation is a package that contains everything needed to start using Chef, such as Chef Client, Chef and Knife CLI and test tools, to build and test cookbooks and interact with Chef Server.
2) Chef-Solo. Doesn’t require the Chef Server and doesn’t support the following functionality by the Client-Server mode either: centralized distribution of cookbooks, a centralized API Gateway that interacts with and integrates infrasture components, and authentication or authorization. All the managed network nodes are only installed with the Chef Client software and use the local mode of Chef-Client to run the recipes.
3) Hosted Chef - the cloud-based SaaS. For example, AWS OpsWorks is a configuration management service that helps you configure and operate applications in a hybrid cloud environment by using Puppet or Chef. AWS OpsWorks for Chef Automate lets you create AWS-managed Chef servers that include Chef Automate premium features, and use the Chef DK and other Chef tooling to manage them. AWS OpsWorks Stacks provides a simple and flexible way to create and manage stacks and applications, and run s the Chef recipes using Chef-Solo to handle tasks such as installing packages on instances, deploying apps, and running scripts. It supports both AWS EC2 instances and on-premises servers that are installed with the OpsWorks Stack Agent and have connection to AWS public endpoints.
Cookbooks are the configuration units allowing us to configure and perform specific tasks within Chef on the remote nodes and bring a node into a specific state. Cookbooks are created on the workstation and then uploaded to a Chef Server. From there, recipes and policies described within the cookbook can be assigned to nodes as part of the node's "run-list". A run-list is a sequential list of recipes and roles that are run on a node by chef-client in order to bring the node into compliance with the policy you set for it.
A recipe is the main workhorse of the cookbook. A cookbook can contain more than one recipe, or depend on outside recipes. Recipes are used to declare the state of different resources. A recipe is a list related resources that tell Chef how the system should look if it implements the recipe. When Chef runs the recipe, it checks each resource for compliance to the declared state. If the system matches, it moves on to the next resource; otherwise, it attempts to move the resource into the given state.
The chef-client will run a recipe only when asked. When the chef-client runs the same recipe more than once, the results will be the same system state each time. When a recipe is run against a system, but nothing has changed on either the system or in the recipe, the chef-client won’t change anything.
For the Client-Server management mode, 4 VMs have been created by me to practice the soltuion. Below are the configurations:
Chef Server | Chef Workstation | Node1 | Node2 |
// Upgrade the system – Ubuntu 16.04 LTS sudo apt-get update sudo apt-get upgrade sudo apt-get dist-upgrade
// Change the interace name to eth0 sudo gedit /etc/default/grub GRUB_CMDLINE_LINUX_DEFAULT="quiet splash" GRUB_CMDLINE_LINUX="net.ifnames=0 biosdevname=0" sudo update-grub sudo grub-mkconfig -o /boot/grub/grub.cfg
// Install openssh & git sudo apt-get install openssh-server sudo apt-get install openssh-client sudo apt-get install sysv-rc-conf sudo sysv-rc-conf ssh on sudo sysv-rc-conf --list | grep ssh netstat -ta | grep ssh
sudo apt-get install git-core | |||
// Configue interface, IP addr, hostname and DNS | |||
sudo gedit /etc/network/interfaces auto eth0 iface eth0 inet static address 10.0.0.51 netmask 255.255.255.0 gateway 10.0.0.2 dns-nameserver 10.0.1.1 | sudo gedit /etc/network/interfaces auto eth0 iface eth0 inet static address 10.0.0.52 netmask 255.255.255.0 gateway 10.0.0.2 dns-nameserver 10.0.1.1 | sudo gedit /etc/network/interfaces auto eth0 iface eth0 inet static address 10.0.0.61 netmask 255.255.255.0 gateway 10.0.0.2 dns-nameserver 10.0.1.1 | sudo gedit /etc/network/interfaces auto eth0 iface eth0 inet static address 10.0.0.62 netmask 255.255.255.0 gateway 10.0.0.2 dns-nameserver 10.0.1.1 |
// Configure hostname | |||
sudo gedit /etc/hostname server | sudo gedit /etc/hostname workstation | sudo gedit /etc/hostname node1 | sudo gedit /etc/hostname node2 |
// Configure the hosts file | |||
sudo gedt /etc/hosts 127.0.0.1 localhost 127.0.1.1 server.xgf.com server | sudo gedt /etc/hosts 127.0.0.1 localhost 127.0.1.1 workstation.xgf.com workstation | sudo gedt /etc/hosts 127.0.0.1 localhost 127.0.1.1 node1.xgf.com node1 | sudo gedt /etc/hosts 127.0.0.1 localhost 127.0.1.1 node2.xgf.com node2 |
10.0.0.51 server.xgf.com server 10.0.0.52 workstation.xgf.com workstation 10.0.0.61 node1.xgf.com node1 10.0.0.62 node2.xgf.com node2 | |||
// Install the Chef Server
sudo dpkg -i chef-server-core_12.17.33-1_amd64.deb
sudo chef-server-ctl reconfigure
// Create a Chef User and Organization
mkdir ~/.chef
// Create an user sudo chef-server-ctl user-create gset guofeng xue [email protected] 'ipcc2014' --filename ~/.chef/gset.pem
// Create an organization and add the user to the admins sudo chef-server-ctl org-create california "California, USA" --association_user gset --filename ~/.chef/california.pem
gset@server:~/.chef$ ls -l total 8 -rw-r--r-- 1 root root 1674 Aug 20 17:42 california.pem -rw-r--r-- 1 root root 1674 Aug 20 17:40 gset.pem | // Install ChefDK on Chef Workstation
wget https://packages.chef.io/files/stable/chefdk/3.1.0/ubuntu/16.04/chefdk_3.1.0-1_amd64.deb
sudo dpkg -i chefdk_3.1.0-1_amd64.deb
| ||
// Install the GUI - Chef Manage
wget https://packages.chef.io/files/stable/chef-manage/2.5.16/ubuntu/16.04/chef-manage_2.5.16-1_amd64.deb
sudo dpkg -i chef-manage_2.5.16-1_amd64.deb
sudo chef-manage-ctl reconfigure
// Visit the GUI
| |||
Chef Workstation | Node1 | Node2 |
// Generate the chef-repo and move into it gset@workstation:~$ chef generate app chef-repo gset@workstation:~$ cd chef-repo
// Copy the RSA private keys generated when setting up the Chef Server to the workstation gset@workstation:~/chef-repo$ mkdir .chef gset@workstation:~/chef-repo$ scp [email protected]:~/.chef/*.pem ~/chef-repo/.chef/
gset@workstation:~/chef-repo/.chef$ ls -l total 16 -rw-r--r-- 1 gset gset 1674 Aug 20 17:47 california.pem -rw-r--r-- 1 gset gset 1674 Aug 20 17:47 gset.pem -rw-rw-r-- 1 gset gset 496 Aug 20 18:02 knife.rb drwxrwxr-x 2 gset gset 4096 Aug 20 18:05 trusted_certs | ||
// Add the version control with Github gset@workstation:~/chef-repo$ git config --global user.name xueguofeng gset@workstation:~/chef-repo$ git config --global user.email [email protected] gset@workstation:~/chef-repo$ echo ".chef" > .gitignore gset@workstation:~/chef-repo$ git add . gset@workstation:~/chef-repo$ git commit -m "initial commit" gset@workstation:~/chef-repo$ git status On branch master nothing to commit, working directory clean | ||
// Generate knife.rb – the knife configuration file gset@workstation:~/chef-repo$ cd .chef gset@workstation:~/chef-repo/.chef$ gedit knife.rb current_dir = File.dirname(__FILE__) log_level :info log_location STDOUT node_name 'gset' client_key "gset.pem" validation_client_name 'california-validator' validation_key "california-validator.pem" chef_server_url 'https://server.xgf.com/organizations/california' cache_type 'BasicFile' cache_options( :path => "#{ENV['HOME']}/.chef/checksums" ) cookbook_path ["#{current_dir}/../cookbooks"]
// Copy the needed SSL certificates from the Chef Server gset@workstation:~/chef-repo/.chef$ cd .. gset@workstation:~/chef-repo$ knife ssl fetch gset@workstation:~/chef-repo$ knife client list | ||
// Bootstrap a node - install the Chef Client and validate the node gset@workstation:~/chef-repo/.chef$ knife bootstrap 10.0.0.61 -x gset -P ipcc2014 --sudo --node-name node1 gset@workstation:~/chef-repo/.chef$ knife bootstrap 10.0.0.62 -x gset -P ipcc2014 --sudo --node-name node2 gset@workstation:~/chef-repo$ knife client list gset@workstation:~/chef-repo$ knife node list
gset@workstation:~/chef-repo$ knife client list california-validator node1 node2
gset@workstation:~/chef-repo$ knife node list node1 node2 | ||
// Create a cookbook – “nginx” gset@workstation:~/chef-repo/cookbooks$ chef generate cookbook nginx
gset@workstation:~/chef-repo/cookbooks/nginx$ ls -l total 40 -rw-rw-r-- 1 gset gset 77 Aug 20 18:29 Berksfile -rw-rw-r-- 1 gset gset 148 Aug 20 18:29 CHANGELOG.md -rw-rw-r-- 1 gset gset 1090 Aug 20 18:29 chefignore drwxrwxr-x 3 gset gset 4096 Aug 20 18:34 files -rw-rw-r-- 1 gset gset 70 Aug 20 18:29 LICENSE -rw-rw-r-- 1 gset gset 748 Aug 20 18:29 metadata.rb -rw-rw-r-- 1 gset gset 53 Aug 20 18:29 README.md drwxrwxr-x 2 gset gset 4096 Aug 20 21:37 recipes drwxrwxr-x 3 gset gset 4096 Aug 20 18:29 spec drwxrwxr-x 3 gset gset 4096 Aug 20 18:29 test
// Create the default recipe – “default.rb”, which will install and run Nginx, and transfer the html file into Nginx’s root gset@workstation:~/chef-repo/cookbooks/nginx/recipes$ gedit default.rb package 'nginx' do action :install end service 'nginx' do action [ :enable, :start ] end cookbook_file "/var/www/html/index.html" do source "index.html" mode "0644" end
// Make the index.html file gset@workstation:~/chef-repo/cookbooks/nginx/files/default$ gedit index.html <html> <head> <title>Hello World</title> </head> <body> <h1>Welcome to Chef !</h1> <p> --- Guofeng Xue</p> </body> </html>
// Upload the cookbook – “nginx” to the Chef Server gset@workstation:~/chef-repo$ knife cookbook upload nginx gset@workstation:~/chef-repo$ knife cookbook upload –a
gset@workstation:~/chef-repo/cookbooks/nginx$ knife cookbook upload -a Uploading chef-repo [0.1.0] Uploading learn_chef_httpd [0.2.0] Uploading nginx [0.1.0] Uploading xgf [0.1.0] Uploaded all cookbooks.
// Modify the run-list of nodes gset@workstation:~/chef-repo$ export EDITOR=vi gset@workstation:~/chef-repo$ knife node edit node1 { "name": "node1", "chef_environment": "_default", "normal": { "tags": [ ] }, "run_list": [ “recipe[nginx]” ] } gset@workstation:~/chef-repo$ knife node edit node2 { "name": "node2", "chef_environment": "_default", "normal": { "tags": [ ] }, "run_list": [ “recipe[nginx]”,” recipe[xgf]” ] }
gset@workstation:~/chef-repo$ knife node show node1 Node Name: node1 Environment: _default FQDN: node1.xgf.com IP: 10.0.0.61 Run List: recipe[nginx], recipe[xgf] Roles: Recipes: nginx, nginx::default, xgf, xgf::default Platform: ubuntu 16.04 Tags:
gset@workstation:~/chef-repo$ knife node show node2 Node Name: node2 Environment: _default FQDN: node2.xgf.com IP: 10.0.0.62 Run List: recipe[nginx] Roles: Recipes: nginx, nginx::default Platform: ubuntu 16.04 Tags:
// SSH into the nodes and run the Chef Client gset@workstation:~/chef-repo$ ssh [email protected] gset@node1:~$ sudo chef-client gset@workstation:~/chef-repo$ ssh [email protected] gset@node2:~$ sudo chef-client | chef-client –v
chef-solo -v | chef-client -v
chef-solo –v |
gset@workstation:~/chef-repo$ chef -v Chef Development Kit Version: 3.1.0 chef-client version: 14.2.0 delivery version: master (6862f27aba89109a9630f0b6c6798efec56b4efe) berks version: 7.0.4 kitchen version: 1.22.0 inspec version: 2.1.72 gset@workstation:~/chef-repo$ chef-client -v Chef: 14.2.0 gset@workstation:~/chef-repo$ chef-solo -v Chef: 14.2.0 | sudo chef-client | sudo chef-client |
With the Client-Server management mode, we can also use the local mode of Chef Client on the managed nodes:
// Create a local recipe gset@node2:~$ cat hello.rb directory "/home/gset/xgf111" do action :create end
// Run the recipe locally gset@node2:~$ sudo chef-client --local-mode hello.rb |
For the Chef-Solo management mode, below are the configurations:
// Install the Chef Client with the Chef install script curl -L https://omnitruck.chef.io/install.sh | sudo bash
// Create the recipe gset@ubuntu16:~/chef-solo/cookbooks/helloworld/recipes$ cat default.rb directory "/home/gset/xgf3" do action :create end file "/home/gset/xgf3/abc.txt" do content "Hello World!" end
// Build the configure file gset@ubuntu16:~$ cat solo.rb file_cache_path "/home/gset/chef-solo" cookbook_path ["/home/gset/chef-solo/cookbooks"]
// Define the run_list object gset@ubuntu16:~$ cat node.json { "run_list": [ "recipe[helloworld]" ] }
// Run the recipe with Chef-Sole gset@ubuntu16:~$ chef-solo -c ~/solo.rb -j ~/node.json |
An Overview of Chef
https://docs.chef.io/chef_overview.html
Install a Chef Server Workstation on Ubuntu 18.04
How To Create Simple Chef Cookbooks to Manage Infrastructure on Ubuntu
Nodes - Install via URL
https://docs.chef.io/install_omnibus.html
Executing Cookbooks with Chef-Solo
https://aggarwalarpit.wordpress.com/2015/12/01/executing-cookbooks-with-chef-solo/comment-page-1/