overview

I’ve been thinking about dynamic inventories and Ansible, especially when using AWS. One major difference between static IT infrastructures and dynamic AWS infrastructures is that IP addresses may change, instances may be terminated, and things just flat out change. Your resources are a moving target. Finding those instances/servers/VMs is the first step in managing your inventory.

Ansible has a great tool for this, “ec2.py”. It basically pulls information on instances in AWS at runtime (as -i ./ec2.py, for instance). Not too shabby a solution.

There are a couple things I don’t like about this:

• Credentials need to be passed in, either through a Boto profile or the command line environment variables (environment variables are nicer IMHO).
• After ec2.py has already pulled the inventory, the Ansible playbook may have added/removed instances. It could be out of date or just missing newly created instances.
• It’s kind of… slow. When the cache times out, it takes a while to rescan.
• If instances have been provisioned by Ansible, they may not exist when the ec2.py script runs.

These issues bothered me a bit. Would there be a better way around this? Behold, the ec2_remote_facts module.

the stuffs - ec2_remote_facts

ec2_remote_facts, as the name implies, retreives your AWS EC2 inventory from within Ansible itself. Good stuff. Very similar to the functionality that is ec2.py. Using add_host, systems found from ec2_remote_facts can be added to Ansible groups. This is a win.

example

An example can be found here.

- name: Gather EC2 facts.
  ec2_remote_facts:
    region: "{{ region }}"
  register: ec2_facts

- name: Debug
  debug:
    msg: "{{ ec2_facts }}"
{% endraw %}

The output of an instance will look something like:

PLAY ***************************************************************************

TASK [setup] *******************************************************************
ok: [localhost]

TASK [aws.ec2_remote_facts : Gather EC2 facts.] ********************************
ok: [localhost]

TASK [aws.ec2_remote_facts : Debug] ********************************************
ok: [localhost] => {
    "msg": {
        "changed": false,
        "instances": [
            {
                "ami_launch_index": "0",
                "architecture": "x86_64",
                "client_token": "WQrLv1455228748711",
                "ebs_optimized": false,
                "groups": [],
                "hypervisor": "xen",
                "id": "i-2144fee6",
                "image_id": "ami-ff4baf9f",
                "instance_profile": null,
                "interfaces": [],
                "kernel": null,
                "key_name": "b_dev_demo",
                "launch_time": "2016-02-11T22:12:29.000Z",
                "monitoring_state": "disabled",
                "persistent": false,
                "placement": {
                    "tenancy": "default",
                    "zone": "us-west-2a"
                },
                "private_dns_name": "",
                "private_ip_address": null,
                "public_dns_name": "",
                "ramdisk": null,
                "region": "us-west-2",
                "requester_id": null,
                "root_device_type": "ebs",
                "source_destination_check": null,
                "spot_instance_request_id": null,
                "state": "terminated",
                "tags": {
                    "Name": ""
                },
                "virtualization_type": "hvm",
                "vpc_id": null
            },
            {
                "ami_launch_index": "0",
                "architecture": "x86_64",
                "client_token": "DXLfN1455231776555",
                "ebs_optimized": false,
                "groups": [
                    {
                        "id": "sg-1fbf6278",
                        "name": "ssh_inbound"
                    }
                ],
                "hypervisor": "xen",
                "id": "i-5f2f9598",
                "image_id": "ami-f0091d91",
                "instance_profile": null,
                "interfaces": [
                    {
                        "id": "eni-d7600caf",
                        "mac_address": "02:98:7d:c2:fe:99"
                    }
                ],
                "kernel": null,
                "key_name": "b_dev_demo",
                "launch_time": "2016-02-11T23:02:57.000Z",
                "monitoring_state": "disabled",
                "persistent": false,
                "placement": {
                    "tenancy": "default",
                    "zone": "us-west-2a"
                },
                "private_dns_name": "ip-10-148-0-173.us-west-2.compute.internal",
                "private_ip_address": "10.148.0.173",
                "public_dns_name": "",
                "ramdisk": null,
                "region": "us-west-2",
                "requester_id": null,
                "root_device_type": "ebs",
                "source_destination_check": "true",
                "spot_instance_request_id": null,
                "state": "running",
                "tags": {
                    "Name": "linux_instance_1",
                    "custom_key": "custom_value",
                    "platform": "testing"
                },
                "virtualization_type": "hvm",
                "vpc_id": "vpc-eabed38f"
            }
        ]
    }
}

Cool. Note that it includes what is a null instance - that is a terminated instance. But what if the data needs to be filtered? Enter Jinja2 filters. For instance, filter all running instances.

Filters get us a lot:

- name: Get only running instance IP addresses.
  debug:
    msg: "Instance: {{ item.0 }} has IP address: {{ item.1 }}"
  with_together:
    - "{{ ec2_facts.instances|selectattr('state', 'equalto', 'running')|map(attribute='tags.Name')|list }}"
    - "{{ ec2_facts.instances|selectattr('state', 'equalto', 'running')|map(attribute='private_ip_address')|list }}"

This task will produce:

TASK [aws.ec2_remote_facts : Get only running instance IP addresses.] **********
ok: [localhost] => (item=[u'linux_instance_1', u'10.148.0.173']) => {
    "item": [
        "linux_instance_1",
        "10.148.0.173"
    ],
    "msg": "Instance: linux_instance_1 has IP address: 10.148.0.173"
}

Filtered. Any Jinja2 filter can be created and applied. The filters should also produce lists that are 1-to-1 parallel. There’s probably a few ways to take this on.

The next step that we might want to do is chain this to the Ansible module add_host. This will allow us to define a playbook to configure our newly created system:

- name: Add instances to running Ansible group in memory (not persistent between playbook runs).
  add_host:
    groups: "{{ item.0 }}"
    hostname: "{{ item.1 }}"
  with_together:
    - "{{ ec2_facts.instances|selectattr('state', 'equalto', 'running')|map(attribute='tags.Name')|list }}"
    - "{{ ec2_facts.instances|selectattr('state', 'equalto', 'running')|map(attribute='private_ip_address')|list }}"

Sweet.

summary

So there’s a dynamic inventory, using a built in Ansible module, that can be called whenever, producing a very up-to-date inventory. Only one file handles the credentials and it is encrypted using ansible-vault. The vault-password-file is kept outside of the repository, and if concerned over security, the password can be passed in at runtime, not saved to disk. It’s a very handy tool. I tend to use this over the ec2.py script.

More info can be found in my ansible_snippets example. Still updating documentation, bear with me. :)

-b