Confessions of an Agile Coach: June 2019

Playbooks are the primary way to produce "infrastructure as code" when using Ansible. Infrastructure as code is the aesthetic of having all (or as much as you know how to) infrastructure setup, configuration changes, monitoring, rollback, and tear down represented as automated code/scripts. Code for the demos are in a GitHub branch here.

Playbooks

The goal of a "play" is, like a sports team (let's say American football) to map players to their role on the field, and to an activity. In the context of IT, routers, servers, clouds,... are players. Roles are routing, firewall, blog, CMS, .... And activities could be: Update, migrate, re-route, copy file, ....

The IT professional writes playbooks to handle activities they care about. Playbooks are checked into source control and become the system of record.

Playbooks are:

are human readable
combines the act of writing notes to document configuration with the act of declaring configuration, and with act of confirming/testing configuration.
describe a policy you want your remote system to enforce
a set of steps in a general IT process
manage configuration and deployment to remote machines
(advanced) sequence multi tier rollouts involving rolling updates, delegate actions to other hosts, interact with monitors and load balancers

Although one could get by using ansible to "execute" scripts to multiple hosts via Ansible modules, Playbooks are a structured way to organize interaction between equipment, their roles, and activities. Playbooks are defined in a .yml file and start with "---". For each play in a playbook, you get to choose what to target in your infrastructure and what remote user executes the tasks.
The metaphors connect like this: playbook -> {play -> {tasks -> modules}*}*
Where a playbook may have zero, one, or many plays. And a play may have zero, one, or many tasks. A task requires a call to a module in order to affect some change.

A single play is in verify-apache.yml playbook:

Two plays are in this playbook:

The above examples are just one way to declare remote_user after hosts. The remote_user can be declared in many different contexts. Here is a short document that covers YAML syntax.

Modules

A task is a call to an ansible module. Here the connections to the metaphors: task -> module
In Yaml, tasks are started with "-name." Modules are started on the next line.

The Yaml after the module name are key=value arguments, space delimited. "yum:", "template:", "service:" are references to modules which are packages of features you can use with Ansible. Ansible will pass whatever is after the module, into the module as attributes and value pairs. In Play 1, "yum:" module will get the following list as arguments {name=httpd, state=latest}. The Yum module documentation explains what it does with arguments such as "name" and "state." When Yum receives "state: latest" it will check that the latest httpd is installed and if not, upgrade it.

https://docs.ansible.com/ansible/latest/user_guide/modules_intro.html

Modules are a way of re-using code in Ansible. They can also directly invoked from the ansible command line via "-m":

$ ansible webservers -m service -a "name=httpd state=started"

$ ansible webservers -m ping

$ ansible webservers -m command -a "/sbin/reboot -t now"

The modules used above are: service, ping, and command. Here is a reboot activity done in a playbook:
---
- hosts: webservers

- name: reboot the servers

action: command /sbin/reboot -t now

Rather than on the basic "action" module, we can use a more specific module, "command:"

---
- hosts: webservers

- name: reboot the servers

command: /sbin/reboot -t now

Here is the "restart httpd activity" done in a playbook:

---
- hosts: webservers

- name: restart webserver

service:

state: restarted

Notice how playbooks connect the activities to a name so the playbook writer is organizing their scripts by giving them names. "restart httpd" becomes "restart webserver." Having good names is important to supporting maintainability and human readability.

Here are example playbooks that do "real worlwork." However, I found the use of roles and how the files are organized confusing for getting started. And there is the real possibility that there is a better, less confusing way to organize an enterprises inventory, roles, and files.

Playbooks are executed by the ansible-playbook command.

Write a playbook

Let's build a "compliance" playbook that establishes the state "no errors should be in system logs." The biggest complication I had in learning Ansible was writing YAML and understanding its implicit syntax between Lists and Dictionary. These problems go away with experience and since writing YAML is foundational, it's important to get this working "for you" rather than "against you." So to get some experience, I found slowly "growing" (iterating on) the simplest playbook into something useful, helped me learn how to work with YAML. Jumping straight to the example playbooks listed above didn't give me the experience with YAML that I needed.

Here is the simplest possible playbook:
---
...
Run it:

Let's connect to a host by adding it as a list item.
---
- hosts: blogs
...

From a YAML syntax perspective, "- " means list. "hosts: blog" is a key (hosts) and value (blog).
Run it:

Since I didn't add the inventories, it couldn't resolve what "blogs" meant. So I added the inventory file (-i ) just like I did in the Enterprise Infrastructure as Code with Ansible article.

Now it is resolving the definition of "blogs" to a hostname. But it says the host is unreachable because I need to pass along the username via (-u ). (You can also embed that in the inventory or even playback. I chose to use the command line to pass the username so I can put my files into public source control without exposing my username.)

Success! Our first end-to-end playbook execution. You can get the code on GitHub at this branch.

In YAML, "-hosts ..." is how a list is declared (dash with a space). Then "hosts : blogs" is a key and value pair. So Ansible loaded the YAML and accesses the list and looks for the key named "hosts." It then tried to resolve the value "blogs." Since there isn't a hostname literally named blog, it couldn't ignored that host until an inventory was referenced which defined "blogs."

Let's start checking logs

As you develop a playbook, keep the Ansible Module documentation handy. Since we'll need to execute a shell script on our remote servers, let's do something easy to "kick the tires" of the shell module. (For tips for deciding between command, shell, and script module see the module notes here and here. In this case, we could start with the Command module but since we'll eventually want to use pipes we need to use the Shell module. ) Make the changes needed to have simplest_playbook.yml as the following:

---

- hosts: blogs

tasks:

- shell: echo "hello" > hello.txt

- shell: grep "hello" hello.txt

...

And test the file for errors:

It's happy with the YAML.

Syntax Sidebar:

"- " declares a sequence. "hosts: blogs" is a key and value pair.

"tasks: " maps the tasks to whatever follows, a sequence of more key values: "- " shell: ...:

Ansible requires the key following "tasks: " to be either "name: " or a module.

Note about YAML style:
In YAML the following two playbooks are equivalent. Normally, I advise people to use the most succinct style but I had a lot of confusion separating the sequence marks "- " from the mappings. Otherwise, at least to me, "- hosts: blogs" and "tasks:" don't seem to both be mappings. My brain keeps seeing the "- " and that keeps interfering with my understanding. If you agree with me, great. If you don't then reformat it to how you like. This article about YAML describes how to work with sequences and mappings in a general sense. It also was less confusing than the others.

This:

---

- hosts: blogs

tasks:

- shell: echo "hello" > hello.txt

- shell: grep "hello" hello.txt

...

Verses this:

---

hosts: blogs

tasks:

shell: echo "hello" > hello.txt

shell: grep "hello" hello.txt

...

Are equivalent.

Let's execute this against the remote host by telling ansible-playbook about our inventory and user:

Notice that it reports "OK." Notice the TASK [shell] which echoes back what module executed. More information can be sent to the console by describing what is happing with the call to the modules using the "name :" attribute before the call to the module:

---

hosts: blogs

tasks:

shell: echo "hello" > hello.txt

shell: grep "hello" hello.txt

...

Notice the output about TASK contains what was mentioned by the "name: "mapping.

The play can be more DRY (Do not Repeat Yourself) by declaring a variable.

---

hosts: blogs

vars:

message: hello

tasks:

shell: echo {{message}} > hello.txt

shell: grep {{message}} hello.txt

...

Check server logs

Now we can write a new playbook that checks that our system logs are error free. First, I developed the following by working and testing in a terminal window:

find /var/log -name "*.log" -type f -exec grep -i "error" {} + | grep -v "error_log" | wc -l | grep "0"

If Shell detects a command that returns a none zero code, it will signal there was an error.
(See Ansible and error codes if you want the details.) The above is designed to return a 0 if there aren't any errors, or a non-zero if there are errors found in the log files.

---
-
hosts: blogs

tasks:
-
shell: find /var/log -name "*.log" -type f -exec grep -i "error" {} + | grep -v "error_log" | wc -l | grep "0"

And run it:

If your logs aren't clean, then the PLAY RECAP will tell you the results, which in this case is "failed." Ansible dumps a json file containing what was sent to Shell, what was sent to stderror, and so on. After cleaning the logs up (or you can cheat and chang the script to look for something more "unique" than error), run the command again.

When running the playbook, which runs top to bottom, hosts with failed tasks are taken out of the rotation for the entire playbook. If things fail, simply correct the playbook file and rerun.
Adding a "name: " mapping would make TASK output more sensible. "TASK [shell] ******" isn't a very useful message:

---
-
hosts: blogs

tasks:
-
name: Scanning Logs for error.
shell: find /var/log -name "*.log" -type f -exec grep -i "error" {} + | grep -v "error_log" | wc -l | grep "0"
...

Notice the yellow box below contains more meaningful output.

File Organization

A very basic organization is a directory that's checked into source control and containing:

directory of staging inventory
directory of production inventory
playbooks

This is the organization used in the example code used in this tutorial. Building on that idea, most IT departments will need to add:

directory of roles
directory of group_vars
directory of host_vars

And eventually, an organization with a mature use of Ansible will be creating their own modules for code reuse and will need directories for these files:

library <- code="" li="" lives="" module="" the="" where="">
module_utils <- common="" for="" li="" libraries="" live="" modules="" those="" where="">
filter_plugins

Modules "should be idempotent" meaning running the module many times should have the same affect as running it once. They should be designed to check that the final state has already been achieved and if so, exit without performing any actions.

If you're curious to dig deeper into learning more about the more advanced levels of organization, more details here.

Next Level Ansible

Once you've got some experience making a few playbooks, these topics that will take your Ansible work to the next level. Each other the below are an article by themselves.