The Guardian uses Amazon Web Services (AWS) for the majority of the services behind the website and content management systems. We’re constantly striving to improve our security practices and so we’ve decided to adopt a policy of encrypting everything that we can by default. AWS services make this easy when using S3 buckets, Dynamo tables, RDS instances or EBS volumes. One notable exception is that encrypting the root volume of EC2 instances is non trivial.
Switching to encrypt our root volumes meant changing how we create and distribute the Amazon Machine Images (AMIs) that we create. Our previous approach was to bake all of our AMIs in a single AWS account (using our in house baking tool AMIgo) and then grant permission for all of our other accounts to use them. However, encrypted AMIs can only be used in the account where they were created (as that is where the encryption key is) so we now need to create individual encrypted AMIs in each account that wants to use it.
We chose to create a lambda function (called Image Copier) in each account that fired whenever a new AMI was created by AMIgo. The lambda is simple: it makes an encrypted copy of the AMI in the local AWS account and then add tags to the new AMI to indicate the AMI from which it was copied.
This simple description omits a number of details that are needed to make this work.
First of all, in order to copy an AMI the other AWS account must be given permission to both launch the AMI and also access the EBS snapshot associated with the AMI. We use Hashicorp’s Packer to bake AMIs and we needed to upgrade the version we were using to a more recent version that has support for snapshot permissions (see this PR).
Secondly, tags associated with an AMI are not visible from other accounts and so they are not copied with the AMI. We need the tags to keep track of the identity of each image. To keep them we included the set of tags on the SNS topic that was used to trigger the Lambda. In addition to the existing tags we also appended two new tags: Encrypted and CopiedFromAMI. These indicate that the AMI has an encrypted root volume and the ID of the original AMI it was copied from.
Thirdly, we don’t want to make an encrypted copy of every image that AMIgo bakes in every account that contains the Image Copier lambda. To solve this, each recipe in AMIgo has a list of target accounts and these are also sent in the message on the topic. When an Image Copier lambda receives a message it looks to see if it is in an account that is listed in the message. If so it will create an encrypted copy, otherwise it will ignore the message.
Finally, it’s worth saying that we used AWS CloudFormation StackSets to deploy this Lambda function to all of our accounts so that we didn’t need to do it by hand and can easily update it in the future.
You can find the source code for AMIgo and Image Copier in our GitHub repository.