Retrieve AMI IDs from the EC2 API

[zl4bv]

This is a feature request intending the replace some or all of the functionality provided by the amis.json file. This is also a potential solution to #117.

Rather than creating a static mapping between a specific OS/image flavour, an EC2 region/AZ, and an AMI ID, I propose using the describe-images API endpoint to look up the AMI ID on-the-fly.

• You can search for AMIs by name using wildcards e.g. 'ubuntu/images/hvm-ssd/ubuntu-trusty-14.04-amd64-server-*' => latest Ubuntu 14.04 image
• AMI names often include the date which is why the wildcard is useful
• This ensures you get the latest version of an image (no more updating a list of mappings every month!)
• You can also filter by region, etc.

Either you could change amis.json to map a short name/alias to the filter parameters that get passed to describe-images e.g.

{
  "ubuntu-14.04": [
    {"name": "ubuntu/images/hvm-ssd/ubuntu-trusty-14.04-amd64-server-*"}
  ],
  "windows2012r2-base": [
    {"name": "Windows_Server-2012-R2_RTM-English-64Bit-Base-*"}
  ]
}

and/or you could leave it up to the user to provide the filter(s) in their .kitchen.yml file e.g.

platforms:
  - name: windows2012r2
    driver:
      image:
        - name: Windows_Server-2012-R2_RTM-English-64Bit-Base-*

Thoughts @fnichol @tyler-ball ?

[sqorbrody]

👍 I currently am just manually changing the ami's from the following list, (https://cloud-images.ubuntu.com/trusty/current/) This would be really nice

[litjoco]

👍 Agree this capability would be useful, and helpful for scenarios requiring private and frequently changing amis

[proffalken]

+1 for this, it would make the lookup part of #199 redundant (although the "default to HVM+EBS would still stand)

[freimer]

I'm not sure if this is related or not, but...
We are using custom AMI images in AWS, and it looks like the driver in kitchen-ec2 relies on specific names for images to map the image version (https://github.com/test-kitchen/kitchen-ec2/blob/master/lib/kitchen/driver/aws/standard_platform/windows.rb). Our kitchen create is bombing out with “Failed to complete #create action: [undefined method `windows_version_parts' for nil:NilClass]” because our image names don’t follow the “standard” that kitchen expects (we have our own standard). Any way we can use a tag for that instead of requiring specific AMI names for images?

Same thing with RHEL, by the way, and I expect with all platforms.

[zl4bv]

@freimer as described here, the image_search feature allows you to search for an AMI using any of the filters offered by the EC2 API. For example to search for an image with a specific tag name and value, you could do something like the following:

platforms:
  - name: custom-platform-name
    image_search:
      "tag:foo": bar

Be aware that image_search is a separate approach to setting the image than the platform.name approach. With the latter, there is indeed a specific naming convention in order to lookup the image for a given platform.

[cheeseplus]

Based on the merges above and original request, closing this as resolved but feel free to re-open if I'm incorrect here.

[jrnt30]

@zl4bv There also is a requirement for the Windows provider to work properly with the image-search that it follow an explicit naming convention currently. If it does not include windows in the AMI ID then there is no way for the Windows platform to actually register itself it seems.

I made a somewhat hacky work around when debugging this https://github.com/test-kitchen/kitchen-ec2/compare/master...jrnt30:windows-driver-association?expand=1 but I'm not sure I particularly like it. Even here, the driver.windows_os? relies on the string then starting with win

There is a PR out there from @freimer to just explicitly state how a lot of the functionality is driven off the AMI name which I think could help alleviate some of the issues.