Kubernetes Secrets Encrypting at Rest

Overview

Kubernetes has a built-in ‘secrets’ object which allows you to store your application’s sensitive information. This allows you to separate your backend database credentials, for example, from your pod spec.

Kubernetes secrets are stored in the cluster’s etcd database. While this provides your entire cluster with access, the database is not encrypted.

In this tutorial we will look at creating new secrets for your application, and then how your pods can fetch those secrets.

We will then look at improving the security of your secrets by ensuring that they are encrypted at rest.

Configure an Encryption Provider

1. Generate a 32 byte random key and base64 encode it.
   

   head -c 32 /dev/urandom | base64

   Your output will look something similar to the following example. Keep note of the output, as we will be using it in a config file.

   WxaW1k22mu3M/WYMIWYVOkAOrOTJ17+Q5+McAIqK3bM=

2. Create a new file called /etc/kubernetes/pki/secrets.yml
3. Add the following to its contents.
   

   kind: EncryptionConfiguration
   apiVersion: apiserver.config.k8s.io/v1
   resources:
      - resources:
        - secrets
        providers:
        - aescbc:
            keys:
            - name: key1
              secret: MnBTeg6QwRCS2/ZGynIAKtU48n+zgQHLldDIZibIuW0=
        - identity: {}

   Replace the secret value for key1 with the base64 encoded key we created earlier.

Add Encryption Provider Config to Kubernetes’ API Controller

1. Open the configuration file for the Kubernetes API server.
   

   vi /etc/kubernetes/manifests/kube-apiserver.yaml

2. Find the following section

   spec:
      containers:
      - command:
        - kube-apiserver
        - --authorization-mode=Node,RBAC
        [...]

   And add add the following line somewhere under - kube-apiserver.

       - --encryption-provider-config=/etc/kubernetes/pki/secrets.yaml

3. Changes saved to the manifest will automatically be loaded by the API controller. There is no need restart an services.

Rotating Keys

A general rule of thumb is to rotate your keys on a regular basis to ensure your secrets are protected. We do this by adding a second, newly generated key to our Encryption Provider configuration file, completing the process by removing the original key.

1. Generate a new base64 encoded key.

   head -c 32 /dev/urandom | base64

2. Open your encryption provider file.

   sudo vi /etc/kubernetes/pki/sercrets.yaml

3. Add a second key to your provider.

   kind: EncryptionConfiguration
   apiVersion: apiserver.config.k8s.io/v1
   resources:
      - resources:
        - secrets
        providers:
        - aescbc:
            keys:
            - name: key1
              secret: MnBTeg6QwRCS2/ZGynIAKtU48n+zgQHLldDIZibIuW0=
            - name: key2
              secret: MRP2QNqGum6UowzL/aqmOG+luwn+/kQdxVuJxDjvlJI=
        - identity: {}

4. Save your changes and then restart the API Controller.
5. Open your encryption provider file again, and then replace your first key with the second one and removing the second entry.

   kind: EncryptionConfiguration
   apiVersion: apiserver.config.k8s.io/v1
   resources:
      - resources:
        - secrets
        providers:
        - aescbc:
            keys:
            - name: key1
              secret: MRP2QNqGum6UowzL/aqmOG+luwn+/kQdxVuJxDjvlJI=
        - identity: {}

6. Once again, save your changes and then restart the API Controller.