On September 24, 2014 someone posted on the oss-sec mailing list about a bash vulnerability that likely affects several decades of bash versions (something like 1.14 – 4.3!). The vulnerability — aptly named “Shellshock” — can lead to remote code execution on un-patched hosts, for example web servers parsing HTTP environment variables via CGI GET requests, sshd configurations using ForceCommand, DHCP clients, etc.

Anyways, I’ll leave the infosec community to expound on attack vectors. The point of this post is really to illustrate that you should be using an infrastructure orchestration tool like Ansible to manage your servers.

Painless patching with Ansible

Patching your systems is painlessly easy if you manage your server infrastructure with something like Ansible. Using a one-off command you can easily update all “web” servers, for example:

$ ansible web -m apt -a "name=bash state=latest update_cache=yes" -K -s

That’s great, but what if you have both Ubuntu and CentOS hosts in the “web” group? CentOS doesn’t use apt for package management, so this has effectively only updated hosts running Debian-family GNU/Linux distros.

Playbooks: the power of Ansible

When you have more than a handful of servers, the combinations of DNS names, IP addresses, roles, and distros becomes overwhelming. With Ansible you define your inventory of hosts, allocate them into groups, and then write “playbooks” to mold your servers into functional roles, ie web, database, compute, proxy, etc servers; the personal relationship between sysadmin and server is gone.

Here’s a simple playbook I wrote which takes into account the different OS families in our infrastructure and updates the bash package on each host.

shellshock.yml:

---
# To update hosts for "Shellshock" bash vulnerability
# See: https://en.wikipedia.org/wiki/Shellshock_(software_bug)

- hosts: all
  sudo: yes
  tasks:
    - name: Update on Debian-based distros
      apt: name=bash state=latest update_cache=yes
      when: ansible_os_family == "Debian"

    - name: Update on RedHat-based distros
      yum: name=bash state=latest
      when: ansible_os_family == "RedHat"

# vim: set sw=2 ts=2:

And then run the playbook with:

$ ansible-playbook shellshock.yml -K -s

In our case we patched twenty-five CentOS 6.x, Debian 6, Debian 7, Ubuntu 12.04, and Ubuntu 14.04 hosts living locally, in Amazon EC2, and in Linode. With one command. In less than five minutes!

Stay vigilant!

Vendors started pushing patched versions of bash on September 26th, two days after the initial disclosure. Two days after those patched versions were released there were new variations of this bug discovered, and new packages issued (and we patched our systems again!).

As of now, five days after initial disclosure, there exist five CVE identifiers for this bug! So keep an eye on social media (#shellshock?), Hacker News, and sites monitoring this bug, because more new vectors may emerge!