Creating Kubernetes Secrets with kubectl

Many applications need access to secrets like passwords and cryptographic keys. Kubernetes secrets is a mechanism that gives applications this access, without storing the secrets in plain text in a CI pipeline or git repository.

The official documentation for kubernetes secrets can be found here: https://kubernetes.io/docs/concepts/configuration/secret/

Creating secrets

From literal data

Here is how to store the value l337 under the key password in a secret named my-service-password:

kubectl create secret generic my-service-password --from-literal=password=l337

A secret can contain more than one key/value pair, e.g. username and password. But you can only create secrets with a single pair using --from-literal.

From file

We can create secrets from the contents of one or more files. If we put hacker in the file username.txt and moarl337 in password.txt, we can create a secret with both like so:

kubectl create secret generic my-db-credentials \
--from-file=username.txt --from-file=password.txt

The filenames become keys, and the contents become values. Keys can also be given explicitly:

kubectl create secret generic my-db-credentials \
--from-file=username=foo.txt --from-file=password=bar.txt

caution

If a file ends in a newline, the newline will become part of the value. This is usually not what you intended.

On Linux,

echo -n <value> > <file>

works around the problem.

Using secrets

As files

Secrets can be exported as files to the containers. The secret is declared as a volume. The volume is then mounted to the container. Each key of the secret turns up as a file at the mount location. Extract from a pod spec:

spec:
  volumes:
  - name: foo
    secret:
      secretName: my-db-credentials
  containers:
  - image: nginx
    volumeMounts:
    - mountPath: /foo
      name: foo
      readOnly: true
    ...

This secret has the keys username and password, which become the files /foo/username and /foo/password in the container.

It is possible to control which paths the keys map to in the volume, and to only include a subset of the keys. See https://kubernetes.io/docs/concepts/configuration/secret/#projection-of-secret-keys-to-specific-paths.

As environment variables

Secrets can be exported as environment variables to the containers. Each environment variable is defined as an attribute of a container spec as shown below. MY_DB_USERNAME gets the value of the username key in the my-db-credentials secret.

    spec:
      containers:
      - image: foo
		..
        env:
        - name: MY_DB_USERNAME
          valueFrom:
            secretKeyRef:
              name: my-db-credentials
              key: username

Reading secrets

Our Kubernetes cluster is configured to allow users to create and replace secrets. For security reasons, it does not allow you to read them. This includes listing them with kubectl get secrets, since that operation also retrieves the contents of all the secrets.

Updating secrets

Since users cannot read secrets, it is not possible to update them using kubectl apply and kubectl edit secret NAME. The only way to update a secret with normal access to the cluster is to replace it. To actually modify an existing secret, you need help from an administrator of the cluster.

Updating by recreating/replacing the secret

If you know everything that is stored in the secret, you have all the information required to recreate it from scratch. This can be done by deleting the old secret and recreating it:

kubectl create secret generic my-service-password --from-literal=password=wrong-password
kubectl delete secret my-service-password
kubectl create secret generic my-service-password --from-literal=password=correct-password

If you create your secrets using YAML-files, you can also use the kubectl replace command. E.g. with a file my-service-password.yaml:

apiVersion: v1
kind: Secret
metadata:
  name: my-service-password
  namespace: my-namespace
type: Opaque
data:
  password: d3JvbmctcGFzc3dvcmQ=

You can replace an existing object using kubectl replace:

kubectl replace -f my-service-password.yaml

Modifying a secret

If you do not know the variables stored in a secret, but still need to update it, you need help from an administrators for the Platon PaaS cluster. First create a new secret containing the fields that you want in the original secret:

kubectl create secret generic my-service-password-update --from-literal=password=new-password

Then contact an administrator and ask that they update the existing secret using the data from the secret you just created. Include information about which project namespace the secret is in, and the name of the original secret and the secret containing the update. They can then log into the admin node for the cluster, retrieve data from the update secret and merge it into the original secret. E.g.:

# First get the updated fields
sudo kubectl -n project-namespace get secret my-service-password-update -o yaml
# Then load the original secret in an editor and add/update the existing fields
sudo EDITOR=nano kubectl -n project-namespace edit secret my-service-password
# Lastly delete the secret containing the update
sudo kubectl -n project-namespace delete secret my-service-password-update

When do updates take effect

Containers which have mounted the secret as a volume will see the change after a short delay, but those which consume it as an environment variable will not see the update until the pod is restarted.

Other kinds of secrets

Kubernetes secrets are also used for TLS, service accounts and for pulling from docker registries. kubectl create secret has subcommands for creating these kinds of secrets.

caution

Note that on the kubernetes worker host running the container, the secrets are available in plain text.