Kubernetes Secrets Management

in Visualization , Monitoring and Observability

Kubernetes Secrets Management .png

Kubernetes is now the standard for container orchestration. With organizations slowly adopting container first development structure, a large part of existing workloads are still running on virtual machines, either in the public cloud or private data centres.


    Introduction

    Kubernetes is now the standard for container orchestration. With organizations slowly adopting container first development structure, a large part of existing workloads are still running on virtual machines, either in the public cloud or private data centres. Many companies are now faced with migrating from their previous methods to Kubernetes.

    Migrating to Kubernetes touches the entire devops process, including monitoring, logging, CI/CD, and most importantly, security. Security can be handled both at the cluster level and also at the application level. 

    In this post, we will try to gain more insight into how we can manage application secrets effectively in Kubernetes. 

    In Kubernetes, sensitive information such as API integration tokens, OAuth tokens, and database passwords are managed by a secret object. These secrets are made accessible to pods as a mounted volume. 

    If you are running different Kubernetes clusters for different environments (which is highly recommended) you might want to store all your environment specific secrets in a single place. Then, make sure you have a secret management tool which can smartly identify the environment the pod is deployed in and fetch secrets accordingly. We will learn more about doing this later in this post. 

    ‍ 

    Getting Started with Kubernetes Secrets

    Creating Secrets Using Kubectl and Mounting Secrets as Volume Mounts

    Let’s create a secret for a token to be used by an application for authenticating with a 3rd party service. 

    We can either create a secret from a literal value or a file. In this case we have put our secret information in a file named access.txt. 

    $cat access.txt
    APP_AUTH_TOKEN=WEj4VmNF755uc9vZdz98zvPXB6DkHp
    
    $ kubectl create secret generic auth-token --from-file=./access.txt
    secret "auth-token" created
    
    $kubectl get secrets
    NAME                  TYPE                                  DATA      AGE
    auth-token            Opaque                                1         14s
    default-token-k7vmv   kubernetes.io/service-account-token   3         17m
    
    $ kubectl describe secret auth-token
    Name:         auth-token
    Namespace:    default
    Labels:       <none>
    Annotations:  <none>
    
    Type:  Opaque
    
    Data
    ====
    access.txt:  46 bytes

    Let’s now use this secret in your pod as a mounted volume.

    First, we create a demo pod and apply the manifest.

    $ cat pod.yaml 
    ---
    apiVersion: v1
    kind: Pod
    metadata:
      name: demo-pod
    spec:
      containers:
      - name: demo-pod
        image: ubuntu
        command: ["/bin/bash", "-ec", "while :; do echo '.'; sleep 5 ; done"] 
        volumeMounts:
        - name: myvolume
          mountPath: "/tmp"
          readOnly: true
      volumes:
      - name: myvolume
        secret:
          secretName: auth-token
    
    $ kubectl apply -f pod.yaml
    pod "demo-pod" created
    
    $ kubectl get pods
    NAME       READY     STATUS    RESTARTS   AGE
    demo-pod   1/1       Running   0          1m

    Now, if we exec inside this pod, we should be able to find our secret mounted to /tmp directory. 

    $ kubectl exec -it demo-pod /bin/bash
    root@demo-pod:/# ls /tmp
    access.txt
    root@demo-pod:/# cat /tmp/access.txt 
    APP_AUTH_TOKEN=WEj4VmNF755uc9vZdz98zvPXB6DkHp
    root@demo-pod:/#

    We can use this method to safely mount configuration files which contain sensitive data. Then, they can be read by the application from the mounted directory. However, sometimes we also want sensitive data to be available as env var for our application. Let’s try to do that in the next section. 

     ‍

    Making Secrets Available as Pod Environment Variables

    In order to make secrets available as env var, we will create a secret object manifest and apply it. We will first encode our data before putting it in the secret file as plain text. 

    $ echo WEj4VmNF755uc9vZdz98zvPXB6DkHp | base64
    V0VqNFZtTkY3NTV1Yzl2WmR6OTh6dlBYQjZEa0hwCg==
    
    $ cat auth_secret.yaml 
    apiVersion: v1
    kind: Secret
    metadata:
      name: mysecret
    type: Opaque
    data:
      auth_token: V0VqNFZtTkY3NTV1Yzl2WmR6OTh6dlBYQjZEa0hwCg==
    
    $ kubectl apply -f auth_secret.yaml
    secret "mysecret" created
    
    $ kubectl describe secret mysecret
    Name:         mysecret
    Namespace:    default
    Labels:       <none>
    Annotations:  
    Type:         Opaque
    
    Data
    ====
    auth_token:  31 bytes

    In order to inject the secret as an env var we will have to access the secret using the key at which it is stored and map it to an env var for the pod. In the following example we do this with another demo pod, demo-pod-2. 

    $ cat pod-2.yaml 
    apiVersion: v1
    kind: Pod
    metadata:
      name: demo-pod-2
    spec:
      containers:
      - name: demo-pod-2
        image: ubuntu
        command: ["/bin/bash", "-ec", "while :; do echo '.'; sleep 5 ; done"] 
        env:
          - name: APP_AUTH_TOKEN
            valueFrom:
              secretKeyRef:
                name: mysecret
                key: auth_token
    
    $ kubectl apply -f pod-2.yaml 
    pod "demo-pod-2" created

    Now, if we exec into demo-pod-2 and check the value for APP_AUTH_TOKEN env var, we should see the decoded value of our secret.  

    $ kubectl exec -it demo-pod-2 /bin/bash
    root@demo-pod-2:/# echo $APP_AUTH_TOKEN
    WEj4VmNF755uc9vZdz98zvPXB6DkHp
    root@demo-pod-2:/#

    Further Reading

    Recently Kubernetes came out with a new feature for encrypting secrets at rest. I highly recommend reading it. The feature is quite new, but that shouldn’t stop us from trying it out.

    Advanced Management of Kubernetes Secrets

    So far, we have learned about how Kubernetes secrets work and how we can consume them in a pod. However, when we are running multiple Kubernetes clusters (development/staging/production) we need a centralized secret store and a secure mechanism to populate our pod environment with the required secrets. 

    Today we will explore two solutions for this:

    1. Using Vault with Kubernetes based auth
    2. Integrating Chamber in your Dockerfiles to populate secrets from AWS Parameter Store (for Kubernetes clusters on AWS)

    ‍ 

    Using Vault with Kubernetes Based Auth

    The workflow for Vault based authentication can be summarized as follows:

    1. A pod gets deployed in a particular namespace and is associated with a particular service account.
    2. The namespace and the associated service account is tied to a Kubernetes authentication role in the Vault backend.
    3. The pod uses the token from the service account to authenticate with Vault and get the VAULT_TOKEN.
    4. Using the VAULT_TOKEN and the VAULT_ADDRESS we can retrieve secrets from Vault in a secure manner. 

    Furthermore, in order to inject those secrets as environment variables into the pod we have a few options, which we will talk about later on. 

     ‍

    What Is Vault

    Vault is a lightweight tool for effectively storing and managing secrets. It has great support for Kubernetes authentication, which is totally encrypted and secure. Vault is usually backed by Consul as the storage engine. Therefore it is highly reliable and resilient to node failures.  

    ‍ 

    Setting Up Vault

    We have a couple of ways to set up Vault for our Kubernetes applications:

    1. Setting up Vault in a Kubernetes cluster which can be accessed across environments.
    2. Using a hosted version of Vault. 

    How to set up Vault on Kubernetes is beyond the scope of this blog, but below are some resources which you can use:

    1. https://www.hashicorp.com/blog/announcing-the-vault-helm-chart
    2. https://github.com/hashicorp/consul-helm

    Once you have Vault set up, it is possible to hook it up with your Kubernetes cluster. As stated previously, the goal is to make sure your pods can effectively authenticate with Vault, so the application can retrieve the secrets.  

    ‍ 

    Authenticating Kubernetes Pods with Vault

    First, create a Kubernetes service account with the following permissions.

    $ cat vault_sa.yaml
    apiVersion: rbac.authorization.k8s.io/v1beta1
    kind: ClusterRoleBinding
    metadata:
    &nbsp; name: role-tokenreview-binding
    &nbsp; namespace: default
    roleRef:
    &nbsp; apiGroup: rbac.authorization.k8s.io
    &nbsp; kind: ClusterRole
    &nbsp; name: system:auth-delegator
    subjects:
    - kind: ServiceAccount
    &nbsp; name: vault-auth
    &nbsp; namespace: default
    
    $ kubectl apply -f vault_sa.yaml 
    clusterrolebinding.rbac.authorization.k8s.io "role-tokenreview-binding" created

    Then, retrieve variables to enable Kubernetes auth in the Vault backend.

    Kubernetes host — an address that Vault can connect with.

    k8s_host="$(kubectl config view --minify | grep server | cut -f 2- -d ":" | tr -d " ")"

    Cluster authority data — a certificate to verify the connection.

    k8s_cacert="$(kubectl config view --raw --minify --flatten -o jsonpath='{.clusters[].cluster.certificate-authority-data}' | base64 --decode)"

    User token — a vault-auth service account with token reviewer role. Vault will interact with the cluster using this service account.

    secret_name="$(kubectl get serviceaccount vault-auth -o go-template='{{ (index .secrets 0).name }}')"
    
    account_token="$(kubectl get secret ${secret_name} -o go-template='{{ .data.token }}' | base64 --decode)"

    Once we have all of these values, we can enable Kubernetes auth in our Vault backend. 

    vault auth enable kubernetes
    vault write auth/kubernetes/config \
        token_reviewer_jwt=${account_token} \
        kubernetes_host=${k8s_host} \
        kubernetes_ca_cert=${k8s_cacert}

    Now we need to make sure that our newly launched pods can authenticate with our Vault server. For this, we bind a namespace to our Vault role and will later on use the service account JWT in that namespace for authentication.

    vault write auth/kubernetes/role/demo bound_service_account_names=vault-auth bound_service_account_namespaces=default policies=demo-policy ttl=1h

    Let’s test it! 

    Create a demo pod and try to authenticate with Vault.

    kubectl run -it --rm --image=ubuntu --serviceaccount=vault-auth test -- /bin/bash
    root$ apt-get update -y &amp;&amp; apt-get install vim curl jq mysql-client -y
    
    #Let's get the service account JWT token
    root$ JWT="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)"
    
    #Now we can use this to get the vault token
    root$ VAULT_TOKEN="$(curl --request POST --data '{"jwt": "'"$JWT"'", "role": "demo"}' -s -k https://${VAULT_ADDRESS}/v1/auth/kubernetes/login | jq -r '.auth.client_token')"

    We can now use VAULT_TOKEN to authenticate with Vault and retrieve our secrets. However, this is only half the battle. Our goal was to populate the environment variables with Vault secrets so that the application can consume them. 

    If your application has logic to read directly from Vault using the VAULT_TOKEN and VAULT_ADDRESS environment variables then you can completely skip this part. Otherwise you can do the following: 

    1. Integrate vaultenv with your Docker container. 
    2. Use vaultenv at the beginning of your entrypoint script and populate the environment.‍

    Storing Secrets in AWS Parameter Store 

    ‍ 

    What Is AWS Parameter Store?

    It is a highly scalable and secure service provided by AWS to store your secrets. You can read and write secrets using AWS CLI. If you are accessing it from an EC2 instance or an ECS task, appropriate IAM roles should be configured. 

     ‍

    What Is Chamber?

    Chamber is a command line utility which helps you read and write secrets from AWS Parameter Store. It supports a number of other commands to populate the execution environment with the secrets or to export them in various formats. 

    In order to get started with this you will need the following information handy:

    1. AWS_DEFAULT_REGION
    2. AWS_SECRET_KEY_ID
    3. AWS_SECRET_ACCESS_KEY

     ‍

    Installing Chamber Locally

    If you’re on a Mac, Chamber can be installed locally using the following commands.

    brew update 
    brew install chamber

    Make sure you have AWS CLI configured on your laptop.

    Use the following to write secrets to AWS Parameter Store.

    chamber write <service> <key> <value>

    Use the following to read secrets from the AWS Parameter Store.

    chamber read <service> <key>

    Integrating Chamber with Your Kubernetes Apps

    To integrate Chamber with your Kubernetes apps you will need to make some minor changes to the following:

    1. Dockerfile of your app
    2. Entrypoint of the script
    3. Deployment manifest file

    At the top your Dockerfile, add the following content.

    FROM golang:1.10.4 AS build
    
    RUN CGO_ENABLED=0 GOOS=linux go get -v github.com/segmentio/chamber
    
    FROM <existing base image>
    
    COPY --from=build /go/bin/chamber /chamber
    …

    This will build and integrate the Chamber binary into your container.

    In the entrypoint script, make the following changes.

    Before the main entrypoint logic kicks in, add the following statement to populate the environment variables.

    eval "$(chamber env $SERVICE)"

    The following environment variables need to be set in the manifest for the deployment.

    1. AWS_DEFAULT_REGION: this corresponds to the default region for your access credentials.
    2. SERVICE: this corresponds to the service whose secrets you want to read from the Parameter Store using Chamber. 

    Chamber also needs access to AWS_SECRET_KEY_ID and AWS_SECRET_ACCESS_KEY, however we do not recommend setting those in your pod manifest file. It is highly recommended that you use an IAM role based permission method to authenticate with AWS Parameter Store. In order to do that, make sure the IAM role assigned to your Kubernetes worker nodes has ssm:GetParameters action enabled. 

    Once you have all this set up, the container in your pod will read the secrets under the SERVICE environment variable. Then, it will authenticate with AWS Parameter Store to populate the container’s environment with all of the secrets needed for that service. 

     ‍

    Conclusion

    Kubernetes secrets, if managed correctly, can extremely simplify the deployment process. You can choose to inject them in your application’s execution environment or read them on the fly using custom built logic. 

    If you have deployed your Kubernetes cluster in AWS cloud, we highly recommend using the Chamber and AWS Parameter Store integration as it is easiest to get started with and very secure. 

    Also, you can manage access to Parameter Store using fine grain IAM access with the help of Kiam. This will make sure only certain pods in your cluster can retrieve and use secrets from AWS Parameter Store. However, that is a discussion for another time.

    When you need to monitor your Kubernetes setup, try out the MetricFire free trial, or book a demo and talk to us directly. A lot of our customers at MetricFire are monitoring Kubernetes clusters - we may have the expertise you're looking for.

    This post was written by our guest blogger Vaibhav Thakur. If you liked his stuff, check out his LinkedIn profile.


    Get similar stories in your inbox weekly, for free



    Share this story with your friends
    metricfire
    MetricFire

    MetricFire provides a complete infrastructure and application monitoring platform from a suite of open source monitoring tools. Depending on your setup, choose Hosted Prometheus or Graphite and view your metrics on beautiful Grafana dashboards in real-time.

    Latest stories


    GitHub Is Making Protocol Security Changes for SSH Users

    A report detailing changes being made by Git systems to the algorithm at GitHub.

    Different Reactions From the Cybersecurity Community Regarding the Ransomware Bill

    Insights into the proposed bill on ransomware attacks and the possible effect.

    WhiteSource’s Analysis to Improve Relations Among Developers and Security Teams

    A report from the insights released by WhiteSource on DevSecOps practices

    Security: Conti Ransomware Gangs Increase Attacks on International Organizations

    News on security agencies' investigation of the rising attacks on organizations by ransomware experts.

    An Insight Into How the Conti Gang Targets Microsoft Exchange Servers

    We look at how attackers can use PowerShell to orchestrate attacks on vulnerable servers and …

    What Developers Think About GitHub Copilot

    GitHub Copilot is a controversial developer assistant introduced to the public in June 2021. It …