Managing Kubernetes Secrets with AWS Secrets Manager
In the last part of this series, I introduced GoDaddy’s Kubernetes External Secrets Manager. In this installment, we will leverage it to configure secrets backed by Amazon Web Services‘ Secrets Manager.
GoDaddy extensively relies on Amazon Web Services‘ EKS for running their Kubernetes infrastructure. The engineering team at GoDaddy realized that there is no integration between EKS and other managed services like Amazon Secrets Manager and AWS Systems Manager. To bridge the gap between the two, they built a Kubernetes custom controller and a custom resource definition called External Secrets, which can manage and rotate the keys originating in AWS Secrets Manager for any Kubernetes deployment.
Let’s see Kubernetes External Secrets in action.
Prerequisites:
1. Active AWS Subscription
2. AWS CLI
3. Kubernetes Cluster
We will start by creating an AWS Identity Access Management (IAM) Policy and an AWS IAM User with just enough permissions to read the secrets from AWS Secrets Store. In the next step, we will deploy the Kubernetes External Secrets Manager Helm Chart associated with the IAM User credentials. After that, we create a secret in the AWS Secrets Store and pointing it to the Kubernetes External Secret created in the local cluster. Finally, we update the cloud-based secret and verify if it’s refreshed in the Kubernetes cluster.
Step 1: Create an AWS IAM Policy and IAM User to Access Secrets Store
Let’s start by defining the IAM Policy needed to access the secrets. Create a JSON file with the below content and save it in extsecpol.json file.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"secretsmanager:GetRandomPassword",
"secretsmanager:GetResourcePolicy",
"secretsmanager:GetSecretValue",
"secretsmanager:DescribeSecret",
"secretsmanager:ListSecretVersionIds",
"secretsmanager:ListSecrets"
],
"Resource": "*"
}
]
}
Define the environment variables and proceed with the creation of IAM Policy and Role.
POLICYNAME="ExternalSecurityPolicy" IAMUSERNAME="SecretReader"
aws iam create-policy \ --policy-name $POLICYNAME \ --policy-document file://extsecpol.json
Retrieve and store the Amazon Resource Name (ARN) for the policy in an environment variable.
export POLICYARN=$(aws iam list-policies --query 'Policies[?PolicyName==`ExternalSecurityPolicy`].{ARN:Arn}' --output text)
Make sure you replace ExternalSecurityPolicy with the correct policy name.
Create the IAM User based on the above policy.
aws iam create-user --user-name $IAMUSERNAME aws iam attach-user-policy --user-name $IAMUSERNAME --policy-arn $POLICYARN aws iam create-access-key --user-name $IAMUSERNAME
From the output of the last command, retrieve the Access Key and Secret Key associated with the user.
Step 2: Install Kubernetes External Secrets Helm Chart
Before we deploy the Kubernetes External Secrets custom controller, we need to register the AWS AccessKey and SecretAccessKey as Kubernetes secrets in the same namespace where the custom controller runs.
Let’s start by creating the namespace.
kubectl create namespace external-secrets
Encode AccessKey and SecretAccessKey into base64 and turn them into a Kubernetes secret.
echo -n "AKIATWRVOGK5V3ZITMWO" | base64 echo -n "6nGIROhIkZGFFvvhkTFWefX5ONFCZyvZZdTIDllR" | base64
Create a file named aws-secrets.yaml with the base64-encoded values and apply it with kubectl command.
apiVersion: v1 kind: Secret metadata: name: aws-credentials type: Opaque data: id: QUtJQVRXUlZPR0s1VjNaSVRNV08= key: Nm5HSVJPaElrWkdGRnZ2aGtURldlZlg1T05GQ1p5dlpaZFRJRGxsUg==
kubectl apply -f aws-secrets.yaml -n external-secrets
Let’s associate the secret with the Helm Chart by defining the below values.yaml file:
env:
AWS_REGION: ap-south-1
AWS_DEFAULT_REGION: ap-south-1
envVarsFromSecret:
AWS_ACCESS_KEY_ID:
secretKeyRef: "aws-credentials"
key: "id"
AWS_SECRET_ACCESS_KEY:
secretKeyRef: "aws-credentials"
key: "key"
Replace the AWS_REGION with the region where you created the secret.
Now, we are ready to install the Kubernetes External Secrets Helm Chart.
helm repo add external-secrets https://external-secrets.github.io/kubernetes-external-secrets/ helm repo update
helm install \ --generate-name \ --namespace external-secrets \ external-secrets/kubernetes-external-secrets \ --values values.yaml
Verify the deployment and ensure there are no errors.
kubectl get pods -n external-secrets
Step 3: Create a Secret in AWS Secrets Manager
With the Kubernetes cluster connected to the AWS Secrets Manager through the External Secrets controller, we are ready to create secrets in the cloud and propagate them to the local cluster.
aws secretsmanager create-secret \
--region ap-south-1 \
--name edgesecrets/dbcred \
--secret-string '{"username":"admin","password":"password@1234"}'
In the next step, we will create an External Secret pointed to the secret created in the AWS Secrets Manager.
Step 4: Create an External Secret Resource in Kubernetes
Let’s create an External Secret custom resource called dbcred associated with the cloud-based secret created in the previous step in the default namespace.
apiVersion: "kubernetes-client.io/v1"
kind: ExternalSecret
metadata:
name: dbcred
spec:
backendType: secretsManager
region: ap-south-1
dataFrom:
- edgesecrets/dbcred
kubectl create -f dbcred-external.yaml
Verify that the External Secret and the Kubernetes secret are created successfully.
kubectl get es
kubectl get secrets
Let’s retrieve the values of the secret by decoding the base64 values.
kubectl get secret dbcred -o yaml
echo " YWRtaW4=" | base64 -d
echo "cGFzc3dvcmRAMTIzNA==" | base64 -d
Access the AWS Management Console to retrieve the secret values.
As you can see, the same values showed in the AWS Console are available in the Kubernetes cluster.
Step 5: Rotating/Refreshing the Secrets
Let’s now modify the secret stored on AWS Secrets Store to see if it automatically gets propagated to the Kubernetes cluster.
Run the below command to change the password to a new value:
aws secretsmanager update-secret \
--region ap-south-1 \
--secret-id edgesecrets/dbcred \
--secret-string '{"username":"admin","password":"password@0000"}'
Retrieve the password value from the Kubernetes cluster.
kubectl get secret dbcred -o yaml
echo "cGFzc3dvcmRAMDAwMA==" | base64 -d
As we can see, the updated password instantly becomes available in the Kubernetes cluster.
Step 6: Cleaning up Resources
Delete Kubernetes resources:
kubectl delete -f dbcred-external.yaml helm ls -n external-secrets helm delete RELEASE_NAME kubectl delete -f aws-secrets.yaml -n external-secrets
Delete AWS IAM Policy:
IAMUSERNAME=SecretReader
export POLICYARN=$(aws iam list-policies --query 'Policies[?PolicyName==`ExternalSecurityPolicy`].{ARN:Arn}' --output text)
aws iam detach-user-policy --user-name $IAMUSERNAME --policy-arn $POLICYARN
Delete AWS IAM User:
export ACCESSKEYID=$(aws iam list-access-keys --user-name SecretReader --query 'AccessKeyMetadata[0].AccessKeyId' --output text) aws iam delete-access-key --user-name $IAMUSERNAME --access-key-id $ACCESSKEYID aws iam delete-user --user-name $IAMUSERNAME aws iam delete-policy --policy-arn $POLICYARN
Delete AWS Secret Manager secret:
aws secretsmanager delete-secret --secret-id edgesecrets/dbcred
