A platform-agnostic way of accessing credentials in Python. Even though AWS enables fine-grained access control via IAM roles, sometimes in our scripts we need to use credentials to external resources, not related to AWS, such as , or of any kind. API keys, database credentials passwords There are a myriad of ways of handling such sensitive data. In this article, I’ll show you an incredibly simple and effective way to manage that using AWS and Python. Popular ways of managing credentials Depending on your execution platform ) or version control hosting platform ( ), you may use a different method to manage confidential access data. Here is a list of the most common ways to handle credentials that I’ve heard about so far: (Kubernetes, on-prem servers, distributed cloud clusters Github, Bitbucket, Gitlab, Gitea, SVN, … environment variables, ingesting credentials as part of CI/CD deployment, leveraging specific plugins from a development tool ex. , ingesting environment variables in in Pycharm, Serverless Credentials Plugin Run/Debug configuration storing credentials within the workflow orchestration solution ex. or , Airflow Connections Prefect Secrets storing credentials as Kubernetes or Docker secrets objects leveraging tools such as . HashiCorp Vault All of the above solutions are perfectly feasible, but in this article, I want to demonstrate an by leveraging . This method will be secure ( ) and will work the same way regardless of whether you run your Python script locally, in AWS Lambda, or on a standalone server provided that your execution platform is authorized to access AWS Secrets Manager. alternative solution AWS Secrets Manager encrypted using AWS KMS Our Use Case We will perform the following steps: on the platform so that we can get stock market data from this API, create an API key Alpha Vantage inside of AWS Secrets Manager, store the API key within our script by using just two lines of Python code retrieve this API key use the key to get the most recent Apple stock market data build function and test the same functionality there. AWS Lambda Implementation — PoC showing this method Create the API Key If you want to follow along, go to and get your API key. https://www.alphavantage.co/ AWS Secrets Manager First, make sure that you configured with an that has access to interact with the AWS Secrets Manager. Then, you can using the following simple command in your terminal: AWS CLI IAM user store the secret To see whether it worked, you can list all secrets that you have in your account using: If your credentials should change later ( ), updating the credentials is as simple as the following command: ex. if you changed your password Retrieve the credentials using awswrangler AWS Secrets Manager allows storing credentials in a . This means that a single secret could hold your entire , i.e., your user name, password, hostname, port, database name, etc. JSON string database connection string The package offers a method that this data into a Python dictionary. When combined with **kwargs, you could unpack all the credentials from a dictionary straight into your Python function performing authentication. awswrangler deserializes My looks as follows ( ): requirements.txt using Python 3.8 Then, to retrieve the secret stored using AWS CLI, you just need these 2 lines: Using the retrieved credentials to get stock market data There is a handy Python package called that allows to easily retrieve data from various sources and store it as a Pandas dataframe. In the example below, we’re retrieving Apple stock market data ( ) for the last two days. pandas_datareader intraday Note that we are passing the from AWS Secrets Manager to authenticate with the Alpha Vantage data source. API key Here is a dataframe that I got: AWS Lambda function using the credentials Since we were able to access the credentials on a local machine, the next step is to do the same in AWS Lambda to demonstrate that this method is platform agnostic and works in any environment that can run Python. I’m using the new alternative way of packaging AWS Lambda function with a . Side note: Docker container image I use the following Dockerfile as a basis for my AWS Lambda function: The script inside of directory looks as follows: lambda.py src To build and package the code to a Docker container, we use the commands: Finally, we build an ECR repository and push the image to ECR: Replace 123456789 with your AWS account ID. Also, adjust your AWS region, accordingly — I’m using . Note: eu-central-1 We are now ready to build and test our AWS Lambda function in the AWS management console. Getting insights about our Lambda function If you are running multiple Lambda function workloads, it’s beneficial to consider using an observability platform that will help you keep an overview of all your serverless components. In the example below, I’m using Dashbird to obtain additional information about the Lambda function executed above, such as: the execution duration per specific function invocation, the cold starts, memory utilization, number of invocations and percentage of errors across them, …and much more. Using Dashbird to debug and gain insights about the AWS Lambda function invocations — image by the author You can see in the image above that the first function execution had a . The second one used . cold start 100% of memory Those insights helped me to optimize the resources by increasing the allocated memory. In the subsequent invocation, my function and didn’t max out the total memory capacity. ran faster The benefits of AWS Secrets Manager Hopefully, you could see how easy it is to store and retrieve your sensitive data using this AWS service. Here is a list of benefits that this method gives you: — credentials are encrypted using AWS KMS Security — if you decide to store all your credentials there, you get a single place where all credentials are located. When authorized, you can also view and update the secrets in the AWS management console. One central place for all credentials — we can access the secrets from a local machine, from serverless functions or containers, or even from on-prem servers, provided that those compute platforms or processes are authorized to access AWS Secrets Manager resources. Accessibility & platform-independent credentials store — AWS provides SDKs for various languages so that you can access the same credentials using Java, Node.js, JavaScript, Go, C++, .NET, and more. Portability across programming languages — when enabling CloudTrail traces, you can keep track of who accessed specific credentials and when — this provides you with an audit trail about the usage of your resources. AWS CloudTrail integration — we can easily grant permissions only for specific credentials to particular users, making it much easier to have an overview of who has access to what. Granularity in access control The potential downsides with AWS Secrets Manager I have a policy to always provide pros and cons to any technology without sugar-coating anything. Those are the risks or downsides I see so far using this service to manage credentials on an enterprise-scale: If you store and you don’t give access according to the least-privilege principle, i.e., some super-user has access to all credentials, then when credentials of such super-user get compromised, you risk exposing all secrets. This is , but I still wanted to include it for the sake of completeness. all credentials in a single place only true if you use the service inappropriately — since you pay for each secret per month, you must be aware that the price can sum up if you use the service to store a large number of credentials. Costs — it’s still hard to convince some IT managers that cloud services can be more secure than on-prem resources when configured correctly. Having said that, many IT managers still don’t trust any cloud vendor enough to literally confide in them with their secrets. Trust Your execution platform must have itself. This means that you need to either configure an IAM role, or you need to store this single secret some other way. It’s not really a downside or risk, but simply, you need to be aware that access to the Secrets Manager needs to be additionally managed in some way, too. access to the Secrets Manager Wrapping up In this article, we looked at the AWS Secrets Manager as a way of managing credentials in Python scripts. We could see how easy it is to put, update, or list secrets using . We then looked at how we can access those credentials in Python using just two lines of code thanks to the package . AWS CLI awswrangler Additionally, we deployed the script to to demonstrate that this method is platform-agnostic. As a bonus section, we looked at how we can add observability to our Lambda functions using Dashbird. Finally, we discussed the of AWS Secrets Manager as a way of managing credentials on an enterprise-scale. AWS Lambda pros and cons Previously published at https://dashbird.io/blog/aws-secrets-manager-python/

Too Long; Didn't Read

Developers: Build Less. Deliver More. [Learn how]	

AWS Secrets Manager: How to Manage Credentials in Python

Too Long; Didn't Read

Taavi Rehemägi

About Author

TOPICS

THIS ARTICLE WAS FEATURED IN...

AWS Secrets Manager: How to Manage Credentials in Python

Too Long; Didn't Read

Taavi Rehemägi

About Author

TOPICS

THIS ARTICLE WAS FEATURED IN...

RELATED STORIES