How to find all regions an AWS Service is available

riting management tools have always been fun and a challenge. Fun because it’s challenging. You always learn a thing or two. For example, we learned yet one more weird fact about AWS API inconsistency, maybe you already know- The S3 Bucket location API always returns null for all the buckets in US East (N. Virginia) region and returns actual AWS API region names for all other locations. Not an interesting bit of information unless you’re dealing with AWS cloud at the API level.

AWS continuously rolls out new innovative AWS services and enhancements to the existing services. Keeping track of AWS services, and the AWS resources created is of tremendous importance for security, cost and general cloud hygiene. Hence we started CloudYali to be the single only source of truth, which has all the AWS resource (AWS assets) information for all AWS services being deployed from all the AWS regions. Missing even a single resource from any remote region or AWS service is simply out of question. This is easier said than done, but that’s where the challenge comes.

Discovering and keeping inventory of AWS assets require the use of AWS APIs. This requires very detailed knowledge of AWS services as well as in which regions these services are available. Unfortunately, we soon discovered there was no single place where we could look and answer questions like "In what AWS regions AWS service X is available", so as to discover AWS assets belonging to that service in the region.

Update: We recently launched our AWS Services Availability Tool with a simple and intuitive UI to get information about all AWS services and regions. Using this tool you can directly find out services a region hosts or in which regions a service is available.

AWS Regions

The very first step is to ensure that we discover all AWS regions. AWS originally "enabled" all the regions by default, but as the global footprint started expanding not every AWS customer needed these new regions. Hence all new regions (post-March 20, 2019) are disabled by default.

These new regions can be enabled. These manually enabled regions can be disabled as well. But you cannot disable any region which is enable by default. Enabling regions can take some time, as AWS takes time to populate global resources such as AWS IAM resources into these new regions.

Understand that any programmatic or console access to the active resources in the disabled region is forbidden, but the active resources inside these regions continue to accrue charges. Making a region disabled does not stop or delete any AWS resources inside the region.

Any AWS inventory software which does not keep resource history and discover resources by directly calling the AWS APIs is likely to fail to track the (still) active resources in the disabled region. CloudYali in contrast keeps a track of the complete resource history timeline along with the resource configuration. Hence only CloudYali continues to track these active resources still in these disabled regions, provided CloudYali was implemented before disabling the region.

To discover “all” regions use DescribeRegions API with AllRegions true. The AllRegions parameter control whether to return a list of all supported regions or only enabled regions. Using API is always preferred as this dynamically gives the latest information.

// query for all regions
req, resp := svc.DescribeRegionsRequest(&ec2.DescribeRegionsInput{
	AllRegions: aws.Bool(true),
})
err = req.Send()
if err != nil{
	// log error 
	return
}
for _, r := range resp.Regions {
// ...
// ...
}

AWS Services

The AWS regions host the AWS service endpoints. The global services are also hosted regionally but have a global namespace. Identity and access management (IAM), S3, CloudFront are some examples of global services.

AWS is continuously rolling out new regions around the globe. The US East (Northern Virginia) is of special importance as most new AWS services start rolling in this region first. The newer regions typically get these services rolled over a period of a few months or years. It means there is always a disparity between the regions when it comes to the AWS services. Some regions have all the AWS services available while other regions have fewer services.

Many Cloud applications need to be architectured and developed for multiple AWS regions. While developing these solutions as an architect it is important to know which services are available in the target region. If any services are not available in the target regions, alternate solutions should be deployed. This requires access to AWS service availability information.

One way to access this information is to consult AWS documentation. Unfortunately, there is no straightway to find out the regions the service is available. For example, the global infrastructure page lists the AWS services region-wise but if you’re looking for the regions AWS Redshift is supported, you may have to go through all regions. Again if multiple versions of the service are available, say as, in the case of AWS Inspector, it does not show whether it supports only v1 or v2 or both.

While developing Cloudyali's inventory service, we needed information about available services in a specific region. Extracting this information manually from documentation would be cumbersome, error-prone and require high maintenance. We were looking for a programmatic way and in the process discovered the following different ways.

AWS Services Regional table

One way is to fetch this information from https://api.regional-table.region-services.aws.a2z.com/index.json. This JSON then can be parsed and consumed. You may write your parser or can use the parser available at https://github.com/burib/aws-region-table-parser.

Or simply use curl with jq:

curl -s https://api.regional-table.region-services.aws.a2z.com/index.json 
| jq -r '.prices[].attributes|select(."aws:serviceName" == "Amazon Inspector") 
| ."aws:region"'

AWS SDK metadata

The AWS SDK (V1.6.0+) provides endpoint metadata. This metadata maintains this information in code. The SDK's metadata is a fixed point in time and will be of the date when new regions are added or new services are enabled until the SDK is updated in the application. The V2 SDK internalized this metadata information instead, and thus not available for consumption. For these reasons, we continued with our quest to find a programmatic way to get the latest information on AWS regions and AWS services.

Ultimately found this information in an unexpected source, Systems Manager Parameters Store (SSM). Parameter Store, a capability of AWS Systems Manager, provides secure, hierarchical storage for configuration data management and secrets management. Looking back now, it completely made sense that AWS kept this crucial configuration information in SSM.

Using SSM

The SSM allows querying the parameter store for path /aws/service/global-infrastructure/services/<service name>/regions to get regions where a service is available. The beauty of this is that it is even possible to get this information for different versions of the service.
For example, to get the information about AWS Inspector's v1 and v2, we simply execute the below commands.

aws ssm get-parameters-by-path 
--path /aws/service/global-infrastructure/services/inspector/regions 
--output json 
| jq '.Parameters[].Name'

aws ssm get-parameters-by-path 
--path /aws/service/global-infrastructure/services/inspector2/regions 
--output json 
| jq '.Parameters[].Name'

We now programmatically fetch AWS services availability information at the start of the inventory collection and ensure we call only available AWS service APIs in any enabled region to discover and inventory the AWS resource types or AWS assets as they are widely called.

const ParamPath := "/aws/service/global-infrastructure/regions/%s/services"
func fetchEnabledServicesInRegion(region *string, sess *session.Session, cfg *aws.Config) 
(service map[string]bool, err error) {
	service = make(map[string]bool)
	svc := ssm.New(sess, cfg)
	var NextToken *string
next:
	req, resp := svc.GetParametersByPathRequest(&ssm.GetParametersByPathInput{
		Path: aws.String(fmt.Sprintf(ParamPath, *region)),
		NextToken: NextToken,
	})

	err = req.Send()
	if err != nil {
		return
	}

	NextToken = resp.NextToken
	if len(resp.Parameters) > 0 {
		// fetch the service name, process 
		for _, p := range resp.Parameters {
			if p.Value != nil {
				srv := formatServiceName(*p.Value)
				service[srv] = true
			}
		}
	}

  // AWS API call sometimes behave erratically, returning empty pages with NextToken as non null
	if NextToken != nil {
		goto next
	}
	return
}

func formatServiceName(s string) (name string) {
	name = strings.TrimSpace(s)

	// Replace all Non-letter/number values with space
  // AWS services are not named consistently
	runes := []rune(name)
	for i := 0; i < len(runes); i++ {
		if r := runes[i]; !(unicode.IsNumber(r) || unicode.IsLetter(r)) {
			runes[i] = ' '
		}
	}
	name = string(runes)
	// Title case name so its readable as a symbol.
	name = strings.ToLower(name)
	// Strip out spaces.
	name = strings.Replace(name, " ", "", -1)
	return
}

You can clearly see in this sample output the disparity of AWS services in different regions.
‍

In region: us-east-1 found: 306 services
In region: us-west-2 found: 288 services
In region: eu-west-1 found: 275 services
In region: ap-southeast-2 found: 261 services
In region: ap-northeast-1 found: 261 services
In region: eu-central-1 found: 258 services
In region: ap-southeast-1 found: 246 services
In region: us-east-2 found: 242 services
In region: eu-west-2 found: 234 services
In region: ap-northeast-2 found: 224 services
In region: ap-south-1 found: 215 services
In region: ca-central-1 found: 211 services
In region: eu-north-1 found: 195 services
In region: us-west-1 found: 194 services
In region: sa-east-1 found: 189 services
In region: eu-west-3 found: 188 services
In region: ap-northeast-3 found: 127 services

‍