ometimes metadata is arguably more important than the data. Especially when it comes to complex environments, we need effective metadata to navigate the data. An effective metadata when combined with the data, can turn data into information that can be exploited for many useful purposes.
A context has to be set here: the data I'm discussing here are the AWS Cloud resources itself. An AWS Cloud can certainly become a complex web over the time, where it's easy to get tangled. It becomes cumbersome to navigate through the data. Take a simple example, you need to find all the resources belonging to a particular service(s). Although you may have an architectural diagram, which may build a conceptual model for you, the reality in cloud could be much worse. The ephemeral nature of cloud resources combined with auto-scalability is enough to take any serious cloud maintainer to the brink of insanity.
Fortunately, the AWS provides a construct called "Tags" which can help us to combine meta and data. Mind it though, as usual the AWS coverage is not adequate, which means there would be times when you need to rollup your sleeves and write your own solutions. But today, we would only discuss this critical piece of infrastructure cement which is often neglected. Any serious cloud developer and maintainer would never ignore it though. Like many good things in life, tag usage needs a proper strategy and a disciplined approach. More importantly the strategy and the process needs to be aligned to your organisations needs.
In cloud context, the AWS Control plane APIs provide the data and tags provide the metadata.Tags, in its simplest form are the metadata stored against any AWS cloud resource. Nothing but key and value strings. Lot of us, get introduced to tags in UI itself, and most of us start with tags like Name and descriptions.
As we will see tag requires a good amount of strategy, planning and discipline to be successful. It also requires cloud engineers to put extra efforts on their part. So naturally question arises why should we do it? What is the benefits of generating metadata with tags?
Working with a handful resources is straightforward. When you move towards multiple accounts and/or larger number of resources, when multiple teams start working on the same cloud, it becomes difficult to manage cloud, from cost, security and architecture perspective. You would find yourselves desperately in need of answers of questions like: "who created this resource?", "For what service you need this resource?", and more importantly "who is paying for this resources?" and plethora of such questions.
An effective Cloud tagging strategy is one which enables us to answer these questions, efficiently and effectively with minimal efforts. Tagging when combined with proper tools, enables you to trace and audit resources. And that is invaluable. To illustrate lets first start out with the few representative use cases:
You get the idea! There are myriad of ways this metadata can be used, based on your business or technical requirements. All this metadata is stored in simple string key/values. If planned and executed properly, this metadata megastore can be the powerhouse of all the endeavours you plan in cloud.
Remember, it is quite difficult to retroactively apply and fix the tags, hence you need to plan the tags and their supported values even before you actually start tagging.
The tags have some limitations though. Eg. Not all resources support tagging. The tag key can have maximum of 128 characters, and values can have maximum 256 Unicode characters. At the moment we have at max 50 user created tags. Also know that tags are case sensitive, which means CostCenter and costcenter are different tags.
Many tags are predefined in AWS or created automatically by AWS services. Many AWS defined tags are in lowercase and are separated by hyphens.
Many such tags also have prefixes to identify the source service. e.g. aws:ec2spot:fleet-request-id identifies the Amazon EC2 Spot Instance Request that launched the instance
Before we start looking into how to approach tagging, you need to decide what is going to be your authoritative data source for cloud governance. Authoritative data source is to be used for providing the data to the all governance activities. Eg. if you're a small/medium startup with complete footprint in the cloud, either tags itself, or any other cloud based (homegrown) governance tool may be sufficient
However for large enterprises with on-premise and cloud footprint, Configuration Management Data base (or CMDB) is an authoritative data source. CMDB is used in mostly on-premises large enterprise environments to maintain hardware and software information. However, in AWS cloud hardware information is maintained by AWS while the software instances to represent the compute, network or storage constructs are provisioned by the customers. In AWS cloud, configuration items (CIs) are these entities, also known as AWS Cloud resources.
In case of CMDB or any external such tool, your tagging strategy should be to keep minimal data in tags and mostly push the information from cloud to CMDB. For CMDB based environments the data can flow in both directions. Eg. service owner and cost center information can come from CMDB, while IP address, instance type etc information can be fetched from cloud to CMDB. In such cases, you also need to ensure that the AWS Tag naming convention indicates that externally sourced information is coming from the CMDB.
AWS documentation suggests one such example:
anycompany:cmdb:cost-center – the Cost Center code associated with the owning application, sourced from the CMDB
This makes it clear that the tags are provided for convenience, and that the authoritative source of the data is the CMDB.
Note that for cloud native workloads CMDB is more like an anti-pattern. For such cloud native environments, keeping the information in the cloud itself ensures that cloud dynamism is maintained.
As a general naming strategy - have the tag names in lowercase, words separated by hyphens and prefixed with abbreviated organisation or your company name. An advantage with prefixing the name of organisation is that it identifies that tag doesn't belong to any third party or AWS, and instead belongs to your organization/company.
As a bonus you save on a tag for naming organisation separately. e.g.
yourcompany:environment to identify whether the environment is development, test, or production
Many AWS Resources have a predefined Name tag which needs to have a value assigned. Typically this is used in management console. While naming AWS Resources you should adapt a cattle naming approach instead of naming them a pets. It simply means instead of having an elaborate naming scheme which identifies the application name/tier, organization etc in the name itself, use ephemeral attributes for naming.
As an illustration, in case of EC2 instance, you may use instance id as Name tag value and use the same value as .yourcompany.com as FQDN while registering the DNS name for these servers.
Each tag should have a defined scope (e.g. organization, team, initiative), and state (mandatory/optional).
AWS control plane provides extensive information about cloud resources. Eg. Information such as availability zone, regions, instance type, ARN etc shouldn't be generally kept in tags.
It is a better to keep only static information in the tags. Keeping dynamic information or information which is frequently updated is asking for operational overhead. In case of later, you will have to find all such tags and keep them in sync as the information changes.
The tag values can be 256 Unicode characters long. Thus it is possible to store multiple values for a single tag. These values can be encoded as JSON values or comma separated values. E.g. an automation specific tag, organization:shutdown, which identifies at what time resources can be shutdown may have values such as {"time":"22:30", "offdays":"Sat,Sun"}. In this case, it simply means resources can be shutdown everyday at 10:30 PM, and must be shut off on weekend.
Many AWS Resources are related and these relationships may change over the period of time. e.g. an EC2 instance may have multiple EBS Volumes. Each EBS volume may have multiple EBS snapshots. These EBS snapshots are AWS Resources on their own. In addition, the EBS Volumes may be detached and attached to different EC2 instances. In such cases, EBS volume tags may be copied to the EBS snapshots.
The tags can be used by both humans as well as code. Thus it is important that you identify the tags which are machine readable and optimize them accordingly. E.g. it is important that you save the time, number or date information in consistent and agreed format. The tag should carry the semantics, eg. it would make more sense to have a human readable one line description rather than keeping ARNs or IDs for description.
Incomplete or invalid tag values are useless. The tags should be ideally enforced via code only. The code can be reviewed, tested and validated for agreed values/formats. Treat this code as your first class citizen with all the proper testing constructs such as unit tests.
Tag space is constrained and even though max number of tags is 50, if not used judiciously they may be over before you realize. You should periodically review all your tags, find any deprecated information/tags and get rid of them. It also ensures that the meta information is accurate and the decisions based on them are accurate too.
When working in collaboration with multiple teams, it is imperative that everyone who works on the cloud (irrespective of via code, console or CLIs) is agreed on a standardised approach. This tagging policy should be accessible to everyone in these teams. This document should probably identify all the items we discussed above.
In addition, it should also maintain the tags themselves with the expected value. This part of the document is important to identify redundant and deprecated tags.
Such tag information can be maintained as (in table):
As I said earlier, it's important to avoid manual entries in tags. Tagging is very important aspect of AWS cloud management and hence AWS and third parties have tooling ready for this. There are many tools which may help your organisation to automate the tagging process. This section lists some tools that you may find useful.
If you're a fan of AWS console, Tag Editor is for you. Tag editor allows you to search for resources using a variety of search criteria and add, modify, or delete tags in bulk. The AWS Resource Tagging API allows you to perform these same functions programmatically.
Using these APIs you may write your own tools to tag, untag, and locate resources using tags. This API has a good coverage for many AWS services.
This recently introduced solution can be used to apply and manage tags on existing as well new resources. Looking at the implementation guide suggest that it takes care of complete tag lifecycle, which is a very useful.
Yor helps tagging from the IaC itself and has support for Terraform, CloudFormation and Serverless.
Terratag is yet another tool working on IaC. Terratag will apply tags or labels to any AWS, GCP and Azure resources
I hope this would help you to understand the importance of tagging, managing its lifecycle. The tooling above can be used to tag resources effectively from the coding phase to runtime. Together these help you to build meta megastore which serves as powerhouse to your cloud journey.
Get the latest updates, news, and exclusive offers delivered to your inbox.
Stay up to date with our informative blog posts.
