Automated Tagging for Cost Optimization | Automate Savings

• Posted August 3rd, 2020

Resource tagging is an important step in understanding your AWS environment. By having a good tagging strategy you can get very granular in relation to your environments, applications and one of the more important aspects, cost optimization. Having a tagging strategy is important but it is only as good as how it is implemented. If you are relying on manual, human intervention then there is always the chance that something either goes without being tagged, or even worst it could be tagged incorrectly. With automated deployments a lot of this is negated, however you are still at the mercy of DevOps engineering teams to correctly configure the pipeline and writing the correct tags. We take a different approach where we look at our AWS infrastructure as the source of truth which also helps in keeping environment sprawl to a minimum. Ultimately we want to ensure that a development machine lives in development environment and a machine living in a development environment does not become a production machine as that will open us up to an entire new set of problems in relation to security and compliance. Tagging becomes essential for us for AWS Cost Optimization as we want to know what systems to target for lights out scripts or what should be targeted or considered for Savings Plans or Reserved Instances. If you want more information on how tagging can help with AWS Cost Optimization then please refer are tagging strategy in there.

Automating Asset Tagging

Adding Tags to our VPC's

We will focus on setting our tags based on the tags that we have outlined for AWS Cost Optimization. We will now setup automation based on certain events in order to automatically put these tags in place. Depending on your Segregation Model you will want to create these tags either on a VPC, Subnet or Security Group. We will go with the concept that Production, Development and QA are based on separate VPC’s so we will put our important tags directly on the VPC’s.

1. Log into your AWS console and navigate to the VPC under services
2. Select the VPC that you want to specify the Environment and Department
3. Select the Tags header in the pain below and add in 2 new tags:

    

   

    

Setting up the Lambda Function

We will create a basic Lambda function that we will subscribe to any “EC2 Instance State-change” event via notification that takes place in our account. Since this is dealing with a specific region, you will want to replicate this function and process to each region where you spin up EC2 instances.

1. In the AWS console navigate to Lambda Functions
2. Press the button in the upper right corner “Create Function”
3. At the lambda function screen select “Author from scratch”
4. In “Function name” specify the name and for “Runtime” specify Node.js 12.x and allow the function to create the “Execution role” for us.
5. Under Basic Settings change the timeout value from 3 seconds to 30seconds
6. in the Function Code box paste the following code and save the function.

                    
const AWS = require('aws-sdk');
exports.handler = async (event) => {
    var newTags = [];
    if (event.detail.userIdentity.type === 'Root') {
        newTags.push({ Key: 'Owner', Value: 'Root' });
    } else {
        newTags.push({ Key: 'Owner', Value: event.detail.userIdentity.arn.split('/').pop() });
    };
    newTags.push({ Key: 'Created', Value: event.detail.eventTime });

    /** handling our logic specific to ec2 and run instances. We can add in other logic here */
    if (event.source === 'aws.ec2' && event.detail.eventName === 'RunInstances') {
        /** calling the update function for each instance */
        var instances = await Promise.all(event.detail.responseElements.instancesSet.items.map(x => getInstanceInfo({ InstanceIds: [x.instanceId] })));

        /** calling the update function for each instance */
        await Promise.all(instances.map(x => ec2Update(x, newTags)));

    } else if (event.source === '') {
        // place logic for other service here!
    }
    return "tags have been set!";
};

var ec2Update = (instance, tags) => new Promise(async (resolve) => {
    /** creating a new list of myTags since they are unique to this instance */
    var myTags = JSON.parse(JSON.stringify(tags));

    /** setting tag keys that we want to ignore and setting all to lowercase */
    var ignoreKeysLowerCase = ['name'];

    /** Getting the tags from the VPC to pass onto the instance */
    var vpc = await getVpc({ VpcIds: [instance.VpcId] });

    /** Getting Ignoring keys that we do not want to passed to the instance */
    vpc.Tags.map(x => { if (!ignoreKeysLowerCase.includes(x.Key.toLowerCase())) { myTags.push(x) } });

    /** Getting Ignoring keys that we do not want to passed to the instance */
    await setTags(instance.InstanceId, myTags);
    resolve();
});

var getInstanceInfo = (params) => new Promise((resolve) => {
    console.log(params)
    var aws = new AWS.EC2({ region: process.env.AWS_REGION }).describeInstances(params);
    aws.on('success', r => {
        resolve(r.data.Reservations[0].Instances[0]);
    }).on('error', e => {
        console.log('error in describeInstances', e.message);
    }).send();
});

var getVpc = (params) => new Promise((resolve) => {
    var aws = new AWS.EC2({ region: process.env.AWS_REGION }).describeVpcs(params);
    aws.on('success', r => {
        resolve(r.data.Vpcs[0])
    }).on('error', e => {
        console.log('error in describeVpcs', e.message);
    }).send();
});

var setTags = (instanceId, tags) => new Promise((resolve) => {
    var params = { Resources: [instanceId], Tags: tags };
    console.log('setting the following tags', params);
    var aws = new AWS.EC2({ region: process.env.AWS_REGION }).createTags(params);
    aws.on('success', r => {
        resolve(r.data);
    }).on('error', e => {
        console.log('error in setTags', e.message);
    }).send();
});
                    
                

 

7. At the top of the function page, navigate to the Permissions tab and select the Rolename hyperlink in the Execution Role window.

    

   

    

8. From here select Attach polices button and on the next window select Create policy
9. At the Create policy window paste the following policy and choose Review policy:

 

                        
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "ec2:CreateTags",
            "Resource": [
                "arn:aws:ec2:*:*:instance/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeInstances",
                "ec2:DescribeVpcs",
                "ec2:DescribeSubnets"
            ],
            "Resource": "*"
        }
    ]
}
                        
                    

 

10. Now name your policy and save it

     

    

     

11. Navigate back to your role and attach the policy that you have just created

Setting up our CloudWatch Event

1. Navigate back to your Lambda Function
2. In the Designer select the Add trigger button
3. Follow the settings in theTrigger configuration image below:

    

   

    

Validate that your event pattern looks like this:

                    
{
  "source": [
    "aws.ec2"
  ],
  "detail-type": [
    "AWS API Call via CloudTrail"
  ],
  "detail": {
    "eventSource": [
      "ec2.amazonaws.com"
    ],
    "eventName": [
      "RunInstances"
    ]
  }
}
                    
                

Now whenever launching a new EC2 Instance your instance will automatically be tagged with the tags we have specified to be inherited from the VPC. Having these tags in place will help out with further automation that we can put in place for Lights Out Scripts.

Conclusion

This is a basic example of what can be achieved with automated tagging. CloudWatch Events coupled with Lambda Functions gives endless possibilities to extending this to include any other type of resource, such as ECS, RDS & DynamoDB.