Debugging Ansible Playbooks
It's a fact: errors do occur. Whether it’s a bug in a playbook or a config file on your control machine with the wrong configuration value, eventually something’s going to go wrong. In this article, I’ll review some techniques I recently learned myself that you can use to help track down those errors.
SSH Issues
When Ansible fails to make a successful SSH connection with the host it’s helpful to see exactly what arguments Ansible is passing to the underlying SSH client so that you can reproduce the problem manually on the command line.
If you invoke ansible-playbook with the -vvv argument, you can see the exact SSH commands that Ansible invokes.
TASK: [copy TLS key] *********************************************************
<127.0.0.1> ESTABLISH CONNECTION FOR USER: vagrant
<127.0.0.1> ESTABLISH CONNECTION FOR USER: vagrant
<127.0.0.1> EXEC ['ssh', '-C', '-tt', '-q', '-o', 'ControlMaster=auto', '-o',
'ControlPersist=60s', '-o', 'ControlPath=/users/sampleuser/.ansible/cp/ansible-ssh-%h-%p-%r',
'-o', 'Port=2222', '-o', u'IdentityFile="/users/sampleuser/.vagrant.d/insecure_private_key"',
'-o', 'KbdInteractive Authentication=no', '-o',? PreferredAuthentications=gssapi-with-mic,gssapi-keyex,
hostbased,publickey', '-o', 'PasswordAuthentication=no', '-o', 'User=vagrant',
'-o', 'ConnectTimeout=10', u'127.0.0.1', u'/bin/sh -c \'sudo -k && sudo -H -S -p
[sudo via ansible, key=ypkyixkznvqmrbmlhezlnlujtdhrcoam] password: " -u root bin/sh -c \'"\'"\'echo SUDO-SUCCESS-ypkyixkznvqmrbmlhezlnlujtdhrcoam; rc=0;
[ -r "/etc/nginx/ssl/nginx.key" ] || rc=2; [ -f "/etc/nginx/ssl/nginx.key" ] ||
rc=1; [ -d "/etc/nginx/ssl/nginx.key" ] && echo 3 && exit 0; (/usr/bin/md5sum /etc/nginx/ssl/nginx.key 2>/dev/null) ||
(/sbin/md5sum -q /etc/nginx/ssl/nginx.key 2>/dev/null) || (/usr/bin/digest -a md5 /etc/nginx/ssl/nginx.key 2>/dev/null) ||
(/sbin/md5 -q /etc/nginx/ssl/nginx.key 2>/dev/null) || (/usr/bin/md5 -n /etc/nginx/ssl/nginx.key 2>/dev/null) ||
(/bin/md5 -q /etc/nginx/ssl/nginx.key 2>/dev/null) || (/usr/bin/csum -h MD5 /etc/nginx/ssl/nginx.key 2>/dev/null) ||
(/bin/csum -h MD5 /etc/nginx/ssl/nginx.key 2>/dev/null) || (echo "${rc} /etc/nginx/ssl/nginx.key")\'"\'"\'\'']
In order to see an error message that the SSH client is throwing you might need to use -vvvv when debugging a connection issue.
For example, if the host doesn’t have SSH running, you’ll see the following error:
testserver | FAILED => SSH encountered an unknown error. The output was
OpenSSH_6.2p2, OSSLShim 0.9.8r 8 Dec 2011
debug1: Reading configuration data /etc/ssh_config
debug1: /etc/ssh_config line 20: Applying options for *
debug1: /etc/ssh_config line 102: Applying options for *
debug1: auto-mux: Trying existing master
debug1: Control socket "/users/sampleuser/.ansible/cp/ansible-ssh-127.0.0.1- 2222-vagrant" does not exist
debug2: ssh_connect: needpriv 0
debug1: Connecting to 127.0.0.1 [127.0.0.1] port 2222.
debug2: fd 3 setting O_NONBLOCK
debug1: connect to address 127.0.0.1 port 2222: Connection refused
ssh: connect to host 127.0.0.1 port 2222: Connection refused
If host key verification is enabled and the host key in ~/.ssh/known_hosts doesn’t match the host key of the server, then using -vvvv will output an error like this:
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ ? ? WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!? ? @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is c3:99:c2:8f:18:ef:68:fe:ca:86:a9:f5:95:9e:a7:23.
Please contact your system administrator.
Add correct host key in /users/sampleuser/.ssh/known_hosts to get rid of this message.
Offending RSA key in /users/sampleuser/.ssh/known_hosts:1
RSA host key for [127.0.0.1]:2222 has changed and you have requested strict checking.
Host key verification failed.
If that’s the case, you should delete the offending entry from your ~/.ssh/known_hosts file.
The Debug Module
It’s Ansible’s version of a print statement. The Ansible debug package can identify any issues while executing the playbook. The issues could be a grammatical mistake, a conceptual mistake, or a missed argument in a playbook. The Ansible debug package shows information as the playbook is being executed in the Ansible terminal. You can use it to print out either the value of a variable or an arbitrary string.
- debug: var=myvariable
- debug: msg="The value of myvariable is {{ var }}"
Debug Module Parameters
Msg Parameter: The “msg” parameter stores the value of the string in itself. If we want to display any message which we want to display in the ansible terminal, we will utilize this parameter. When we did not have any message in the playbook, a generic greeting like “Hello World!” will be displayed.
Var Parameter: The var” parameter has been set either through Ansible facts or the playbook that accepts string value as the input value. The var parameter executes in a jinja2 environment which is why the value stored in var will also include implicit double interpolation. So, if you want to display information that is stored in var that we will use the double interpolation, then we do not need to employ the Jinja2 terminator. Whenever we showcase a parameter in an expression, we will utilize double interpolation.
Verbosity Parameter:? The verbosity parameter has a default value which is 0. This parameter determines whether the debugger is running or not. Like, if the parameter contains a value of 2, then the debugger will only be performed when the playbook is being executed with -v or above.
The Assert Module
The assert module will fail with an error if a specified condition is not met. Assert doesn’t instruct the system to make a decision so it isn’t a conditional statement. It simply tells the system that the statement being written is true.
The assert module can be used in combination with any other module. You can integrate it with other modules so that you receive a success message as soon as the condition proves to be true.
For example, to fail the playbook if there’s no eth1 interface:
- name: assert that eth1 interface exists
??assert:
????that: ansible_eth1 is defined
When debugging a playbook, it can be helpful to insert assertions so that a failure happens as soon as some assumption you’ve made has been violated.
If you want to check on the status of some file on the host’s file system, then it’s useful to call the stat module first and make some assertion based on the return value of that module:
- name: stat /opt/fo
??stat: path=/opt/foo
??register: st
- name: assert that /opt/foo is a directory
??assert:
????that: st.stat.isdir
The stat module collects information about the state of a file path. It returns a dictionary that contains a stat field with the values shown below.
attributes – Returns the attributes of the specified file.
executable – Returns true if the invoking user executed permissions on the target path.
exists – Returns true if the specified path exists.
gr_name – Returns the name of the group of the file owner.
islbk – Returns true if the specified file is a block device
ischr – Returns true if the specified file is a character file.
isreg – Returns true if the specified file is a regular file
isdir – Returns true if the specified file is a directory.
islnk – Returns true if the target file is a link
领英推荐
mode – Returns the file permission in octal notation
Checking Your Playbook Before Execution
The ansible-playbook command supports several flags that allow you to sanity-check your playbook before you execute it.
The --syntax-check flag will check that your playbook’s syntax is valid, but it will not execute it.
$ ansible-playbook --syntax-check playbook.yml
The --list-hosts flag will output the hosts that the playbook will run against, but it will not execute the playbook.
$ ansible-playbook --list-hosts playbook.yml
The --list-tasks flag will output the tasks that the play‐ book will run against. It will not execute the playbook.
$ ansible-playbook --list-tasks playbook.yml
The -C or --check flags will run Ansible in check mode (known as dry- run), which tells you whether each task in the playbook would modify the host, but does not make any actual changes to the server.
$ ansible-playbook -C playbook.yml
$ ansible-playbook --check playbook.yml
By using this flag you should consider that later parts of a playbook might only succeed if earlier parts of the playbook were actually executed.
The -D or --diff flags will output differences for any files that will be changed on the remote machine. It’s a helpful option to use in conjunction with --check to show how Ansible would change the file if it were run normally.
$ ansible-playbook -D --check playbook.yml
$ ansible-playbook --diff --check playbook.yml
It will show the changes in .diff format, like this:
TASK: [set the gunicorn config file] *****************************************
--- before: /home/vagrant/example/project/gunicorn.conf.py
+++ after: /users/sampleuser/dev/ansiblebook/ch06/playbooks/templates/gunicorn.conf.py.j2
@@ -1,7 +1,7 @@
from __future__ import unicode_literals
import multiprocessing
bind = "127.0.0.1:8000"
workers = multiprocessing.cpu_count() * 2 + 1
-loglevel = "error"
+loglevel = "warning"
proc_name = "example"
Limiting Which Tasks to Run
Sometimes you don’t want Ansible to run every single task in your playbook, particularly when you’re first writing and debugging the playbook. Ansible provides several command-line options that let you control which tasks to run.
The --step flag will have Ansible prompt you before running each task, like this:
Perform task: install packages (y/n/c):
You can choose to execute the task (y), skip it (n), or tell Ansible to continue running the rest of the playbook without prompting you (c).
$ ansible-playbook --step playbook.yml
The --start-at-task flag tells Ansible to start running the playbook at the specified task, instead of at the beginning. This can be very handy if one of your tasks fails because there was a bug in one of your tasks, and you want to re-run your playbook starting at the task you just fixed.
$ ansible-playbook --start-at-task="install packages" playbook.yml
Tags
Ansible allows you to add one or more tags to a task or a play. For example, here’s a play that’s tagged with foo and a task that’s tagged with bar and quux:
- hosts: sample_servers
tags:
- foo
tasks:
- name: install editors
apt: name={{ item }}
with_items:
- vim
- emacs
- nano
- name: run arbitrary command
command: /opt/myprog
tags:
- bar
- quux
Use the -t or --tags flag to tell Ansible to only run plays and tasks that have certain tags. Use the --skip-tags flag to tell Ansible to skip plays and tasks that have certain tags.
$ ansible-playbook -t foo,bar playbook.yml
$ ansible-playbook --tags=foo,bar playbook.yml
$ ansible-playbook --skip-tags=bar,quux playbook.yml
Conclusion
Debugging playbooks can be a challenge, but Ansible provides tools to make the process easier. You’ll find the Ansible debugger is an invaluable tool for troubleshooting issues and understanding the behavior of your automation.
(c) Ansible: Up and Running