A Complete Inventory of Your AWS Footprint in 5 Minutes

As companies continue to migrate their workloads from on-premise solutions to public Cloud Service Providers (CSPs) like AWS, it becomes increasingly difficult to understand the complete picture of what might be running within an organization’s cloud. Even simple applications commonly have multiple environments such as Development, Staging, and Production. For more complex workloads, there are additional environments, microservices, queues, notification services (e.g. SNS, SES), data stores, and more.

Even when all this resides under a single AWS account, it is hard to conceptualize let alone manage sprawling cloud resources. Take two seemingly simple questions: “How many EC2 instances am I currently running across all my AWS accounts?” and, “Have I correctly tagged all of my production environment EC2 instances with environment: production?”. These are very difficult to answer using the AWS console or CLI.

This difficulty is compounded by common best practices such as spreading resources across regions and separating workloads across several accounts within an AWS Organization.

Performing an inventory using the AWS console requires keeping a manual count while changing regions and accounts to ensure you didn’t miss anything. Unfortunately, using the AWS CLI is not much easier. For each region and account you wish to query, you need to multiple commands, calling the correct APIs for each asset type (i.e. aws ec2 describe-instances). Then you must manually aggregate the results and double-check you’re not missing any data.

Rather than give yourself a migraine dealing with all of this cross-region, cross-account manual data fetching – let’s see what this would look like using CloudGraph. CloudGraph is the free and open-source GraphQL API for AWS that vastly simplifies the process of answering asset inventory, compliance, and billing questions.

So how do you get started?

First, you install CloudGraph using the single npm install install command:

npm install -g @cloudgraph/cli

Next, you scan all of your AWS accounts at once by configuring CloudGraph using the credentials you already have stored in your local ~/.aws/credentials file (see the CloudGraph AWS provider README if you’d like more information on that). You can also scan accounts using IAM roles if you prefer. Initialize CloudGraph for our AWS accounts by running the following command:

cg init aws

If you have a local AWS credentials file, you should see a list of profiles to select. In my case, I will choose the profiles default and master so I can scan the AWS accounts running my Development and Production environments.

Next, you can pick the regions you want CloudGraph to scan. In my case this is us-east-1, us-east-2, and us-west-1. If you’re unsure what regions you have resources in, you can always select them all. CloudGraph will automatically scan all supported resources. If necessary, this can be modified by passing the -r flag to the init command (see the CloudGraph README for more information on passing flags).

Now you are ready to scan your AWS accounts! Run the following command to launch a local instance of Dgraph, the graph database that CloudGraph uses to locally store your data.

cg launch

And finally, run the following command to scan all of your configured AWS accounts, regions, and services:

cg scan aws

Feel free to grab a quick cup of coffee while this runs.

After the scan is complete, CloudGraph will open the GraphQL query tool you selected in the cg init step. You can use this query tool to ask pretty much any question about your AWS resources.

Let’s answer the first question I posed at the start of this post: “How many EC2 instances are currently running across all my AWS accounts?”. This is simple to answer with CloudGraph.

As you can see, I have 46 EC2 instances running across the two environments I chose to scan. Note that CloudGraph also provides the total of all scanned resources in the report printed at the end of each scan.

Maybe I want to take this a step further and view metadata about these EC2 instances such as their instanceType. I can easily run a GraphQL query against my EC2 instances and request only the specific metadata I care about.

Now let's answer the second question I posed: “Have I correctly tagged all of my production environment EC2 instances with environment: production?”. I could actually do this in a couple of different ways.

First, I could directly query the EC2 instances in Production and see their tags, like so:

query {
  queryawsEc2(filter: {accountId: {eq: "632*****77"}}) {
      id
      tags {
        key
        value
      }
  }
}

This output clearly shows me all my EC2 instances in account “632…” and their tags.

A second and more direct way of asking this question would be to use queryawsTag and filter on the key and value I would like to target:

query {
  queryawsTag(filter: {key: {eq: "environment"} and: {value: {eq: "production"}}}) {
    ec2Instance(filter: {accountId: {eq: "632*****77"}}) {
      id
      arn
    }
  }
}

Now I’m only getting back EC2 instances that are tagged with “environment: production”.

This would also work for other taggable AWS resources such as VPCs, IAM users, S3 buckets, or EKS clusters. CloudGraph makes it easy to inventory your assets no matter how many different regions or accounts they reside in. The sky's the limit for what you can query!

If you have any questions or suggestions on how we can improve CloudGraph, please drop us a line on slack or in the CloudGraph GitHub issues.

Tyler Dunkel Director of Engineering at AutoCloud