Streamlining Secrets Management with AWS

At Nawy, we are constantly evolving our infrastructure to improve security, automation, and developer autonomy. As a DevOps engineer, I’ve been part of the journey from our old configuration management setup to a more dynamic and developer-friendly process. In this post, I’ll walk you through how we used to manage application secrets and environment variables, and how our new approach leverages AWS Secrets Manager and custom packages to give developers more control and reduce DevOps overhead.

Previous Setup: Static Configuration with Terragrunt

In our previous setup, we deployed our applications using Amazon ECS and AWS Lambda. The infrastructure was managed using Terragrunt, a tool that helps with managing Terraform configurations.

Environment Variables and Secrets Management

In the current setup, environment variables and secrets were passed to ECS containers and Lambdas via the Terragrunt YAML files. These files referenced a Secrets Manager ID that stored the sensitive data for each service.

ECS Containers: Environment variables and secrets were passed at the time of container provisioning through the ECS task definition. Any change in these secrets required restarting the ECS container to pick up the updated values.
Lambdas: Similarly, secrets for AWS Lambdas were managed through the same Terragrunt process, with the added problem of having those secrets being loaded as plain text environment variables for the lambdas , clearly visible in the lambda’s console view . Changes to these secrets required reapplying the Terragrunt configuration, which would re-provision the Lambda with the new values.

Here’s a visual of how this process worked for ecs using terragrunt yaml file configuration :

Terragrunt yaml ECS task definition file

and for lambda:

Terragrunt yaml lambda file

DevOps Intervention

In both cases, adding new environment variables or updating secrets required intervention from the DevOps team. Developers had to raise a Pull Request (PR), and once merged, the relevant ECS task definition or Lambda function would be updated with the new values.

Example: Updating ECS Secrets

Step 1: Developer creates a PR to add a new secret or update an existing one.
Step 2: DevOps team reviews the PR and merges it.
Step 3: Terragrunt configuration is applied to update the ECS task definition or Lambda.

The New Setup: Dynamic Secrets Management with a Custom Typescript Package

To enhance security, streamline the process, and reduce cross-team involvement, we have shifted to a new setup that dynamically loads secrets using a custom package. This package is written in TypeScript and is used in both our Lambda functions and ECS containers.

Key Improvements

No More Static Environment Variables: The only environment variable passed to the Lambdas and ECS containers is the Secrets Manager ID.
Dynamic Secret Loading: The custom package communicates directly with AWS Secrets Manager to load secrets dynamically during application startup using that ID as an environment variable.

For Lambdas, we’ve added the AWS Parameters and Secrets Lambda Extension layer, which loads a local http server inside the Lambda to store the secrets and parameters ad an in-memory cache using the secret manager id passed to the lambda as in environment variable in the terragrunt file. This server loads the secrets values everytime the lamdba runs, ensuring the Lambda has access to the latest secrets at runtime without the need for redeployment.
For ECS containers, secrets are fetched at the initial boot, but they can also be updated in real-time by calling the package’s method if needed.

Example: Lambda with AWS Parameters and Secrets Lambda Extension

Custom Typescript Package built and pushed to AWS CodeArtifact for use by Lambda:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55


import * as http from 'http';
export * from './'
const awsSessionToken = process.env.AWS_SESSION_TOKEN;

export const retrieveExtensionValue = (url: string): Promise<any> => {
    return new Promise((resolve, reject) => {
        const options: http.RequestOptions = {
            hostname: 'localhost',
            port: 2773,
            path: url,
            method: 'GET',
            headers: {
                'X-Aws-Parameters-Secrets-Token': awsSessionToken || ''
            }
        };

        const req = http.request(options, (res) => {
            let data = '';

            res.on('data', (chunk) => {
                data += chunk;
            });

            res.on('end', () => {
                try {
                    resolve(JSON.parse(data));
                } catch (error) {
                    reject(error);
                }
            });
        });

        req.on('error', (e) => {
            reject(e);
        });

        req.end();
    });
};

export const Secret = async (secretId: string , key: string) => {
    const secretsUrl = `/secretsmanager/get?secretId=${secretId}&versionStage=AWSCURRENT`;
    let value : any ;
    try {
        const response = await retrieveExtensionValue(secretsUrl);
        const secretString = JSON.parse(response.SecretString);
        value = secretString[key]
        console.log('Secrets initialized successfully');
        return value;

    } catch (error) {
        console.error(`Error retrieving secrets during initialization: ${error}`);
        throw new Error(`Error initializing secrets`);
    }
}

Here’s how the new setup works for lambdas:

New Secrets Manager layer setup in terragrunt

For AWS Lambdas, we’ve added the AWS Parameters and Secrets Lambda Extension. This extension runs a local server inside the Lambda, which the custom package connects to in order to retrieve secrets. The server caches the secrets for a certain duration of the Lambda’s runtime and fetches new values if updated in Secrets Manager when the cache is invalidated . The Lambda no longer requires static environment variables to be passed through Terragrunt. Instead, secrets are dynamically fetched during execution using the local server managed by the extension.

this would be the environments.ts file inside the lambda’s source code , it imports the custom package and has an method that loads secrets as needed by the developer

 1
 2
 3
 4
 5
 6
 7
 8
 9
10



import {Secret} from  '@nawy-utils/lambda-secrets-manager';
export  class  Environments {

    public  static <PROPERTY>: string;

    public  static  async  initializeSecrets() {
        Environments.PROPERTY = await  Secret(process.env.<LAMDBDA_ENV_VAR_FOR_SECRETS_MANAGER_ID> as  string, <KEY> )
    }
}

This is the index.ts file

1
2
3
4
5
6


import {Handler} from  'aws-lambda';
import {Environments} from  './environments';

export  const  handler: Handler = async (event: any): Promise<Response> => {
    await  Environments.initializeSecrets();
}

Each Lambda execution uses the most up-to-date values from Secrets Manager, without needing redeployment through terragrunt .

Example: ECS Container with On-Demand Secret Fetching

Custom Typescript Package built and pushed to AWS CodeArtifact for use by our Applications:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


import { SecretsManagerClient, GetSecretValueCommand } from '@aws-sdk/client-secrets-manager';

const secretsManagerClient = new SecretsManagerClient({ region: process.env.AWS_DEFAULT_REGION });

export async function Secret(secretId: string, secretKey: string): Promise<string | undefined> {
    try {
        const command = new GetSecretValueCommand({ SecretId: secretId });
        const secretResponse = await secretsManagerClient.send(command);

        if (secretResponse.SecretString) {
            const parsedSecret = JSON.parse(secretResponse.SecretString);
            return parsedSecret[secretKey]; // Return the value of the secret key if found
        } else {
            // SecretString is missing, explicitly return undefined
            return undefined;
        }
    } catch (error) {
        console.error(`Error fetching secret for ${secretId}:`, error);
        // Return undefined in case of error
        return undefined;
    }
}

For ECS containers, the initial secrets are fetched during startup. However, if there is a critical need to update a secret (e.g., database credentials), the application can call the package’s function to get the current value from AWS Secrets Manager.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23



import {Variable} from  '@decorators/env-variable.decorator';
import {Secret} from  '@nawy-utils/nest-secrets-manager';
import {config} from  'dotenv';

config();

export  class  Environments {
    @Variable('NODE_ENV') public  static  NODE_ENV: string;
    @Variable('SERVER_PORT', '3000') public  static  SERVER_PORT: string;

    public  static  DB_HOST_MASTER: string;
    public  static  async  initializeSecrets() {
        Environments.DB_HOST_MASTER = await  Secret(process.env.<ECS_SECRET_ID_ENV_VAR>, <KEY>) ?? '';
    // Load other secrets similarly...
    }

// run-time secret fetching
    public  static  async  fetchSecret(key: string): Promise<string> {
        return  await  Secret(process.env.<ECS_SECRET_ID_ENV_VAR>, key) ?? '';
    }

}

Benefits of the New Setup

Reduced Cross-team Intervention: Developers no longer need to submit PRs for every secret update. They can control secrets through AWS Secrets Manager and load them directly in the code or at the most do a container restart / lambda new run.
Improved Security: Secrets are never stored as static environment variables. They are fetched at runtime using the custom package, reducing the attack surface.
Flexibility and Agility: With the ability to fetch secrets on demand, our applications are more resilient to changes in the infrastructure, allowing for smoother operations.

Conclusion

The new configuration management setup at Nawy significantly reduces the manual steps involved in managing secrets and environment variables, while empowering developers to take full control of their application configuration. By adopting this dynamic approach, we have enhanced our security posture and made the deployment process more efficient.

If you’re facing similar challenges in your own infrastructure, consider this approach to reduce overhead and give developers the flexibility they need to move fast without compromising on security.

Stay tuned for more technical insights from our team!