Syncing secrets from Vault with the Vault Secrets Operator

The Vault Secrets Operator (VSO) runs in Platon PaaS and copies secrets from Vault into Kubernetes Secrets in your namespace. Your application then uses them like any other Kubernetes Secret, as environment variables or mounted files.

Compared to fetching secrets in Gitlab CI/CD, secrets are kept in sync continuously. When you change a secret in Vault, the Kubernetes Secret is updated without a new pipeline run, and the operator can restart your pods for you.

What you get automatically

Every namespace belonging to your Gitlab project is set up for Vault access by the platform. You don't need to request anything from Platon, and you don't need to create any Vault roles or policies.

In each namespace the platform creates:

• A ServiceAccount named vault-auth.
• A VaultAuth resource named vault-auth, which tells the operator how to log in to Vault.

These resources are managed by the platform and can't be changed or deleted, neither with kubectl nor through your deployment manifests. If your deployment tries to create or modify them, it will be rejected.

What you can access

Your namespaces can read secrets stored under your Gitlab project's path in Vault, split into one folder per environment:

secret/gitlab/<group>/<project>/prod/...
secret/gitlab/<group>/<project>/dev/...
secret/gitlab/<group>/<project>/shared/...

<group>/<project> is the full path of your Gitlab project. For https://gitlab.sikt.no/my-group/my-subgroup/my-project this is secret/gitlab/my-group/my-subgroup/my-project/.

Each project has two namespaces, one for production and one for development. Which folders a namespace can read depends on which of the two it is:

Folder	Production namespace	Development namespace
prod/	✅	❌
dev/	❌	✅
shared/	✅	✅

This keeps production secrets away from the development namespace. Put secrets that differ between environments, such as database passwords, in prod/ and dev/, and secrets that are the same everywhere in shared/.

Secrets stored directly under your project's path, outside these three folders, are not available through VSO.

Moving existing secrets

Your Gitlab CI/CD pipelines use the same project path in Vault, but without the environment split. Secrets you have stored there for your pipelines must be moved into prod/, dev/ or shared/ before VSO can sync them. Remember to update the paths in your pipelines too if they read the same secrets.

info

Access is read-only. You add and update secrets through the Vault web UI or the Vault CLI.

Syncing a secret

To sync a secret, add a VaultStaticSecret to your deployment manifests. This is the only resource you need to create.

Say the secret my-app in Vault, at secret/gitlab/my-group/my-project/prod/my-app, contains the keys db-password and api-key:

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: my-app-secrets
spec:
  # The platform-provided VaultAuth
  vaultAuthRef: vault-auth
  mount: secret
  type: kv-v2
  # Path relative to the mount, without "secret/" or "data/"
  path: gitlab/my-group/my-project/prod/my-app
  # How often the operator checks Vault for changes
  refreshAfter: 60s
  destination:
    # Name of the Kubernetes Secret that will be created
    name: my-app-secrets
    create: true
  # Optional: restart these workloads when the secret changes
  rolloutRestartTargets:
    - kind: Deployment
      name: my-app

The fields you need to set:

Field	Value
vaultAuthRef	vault-auth, unless you read secrets outside the Gitlab tree.
mount	Always secret.
type	Always kv-v2.
path	gitlab/<group>/<project>/<prod|dev|shared>/<secret-name>. Must be in a folder the namespace's environment can read, see What you can access.
refreshAfter	How often to check Vault for updates, e.g. 60s or 5m.
destination.name	Name of the Kubernetes Secret to create in your namespace.
destination.create	true, so the operator creates the Secret for you.
destination.transformation	Optional. Rename, combine or filter keys, see Transforming the secret.
rolloutRestartTargets	Optional. Deployment, StatefulSet or DaemonSets to restart when the secret changes.

By default, every key in the Vault secret becomes a key in the Kubernetes Secret. Use one VaultStaticSecret per Vault secret you want to sync.

See the VaultStaticSecret API reference for all available options.

Transforming the secret

If your application expects a different format than what is stored in Vault, use destination.transformation to shape the Kubernetes Secret. You can:

• Template new keys from the Vault data with Go templates. The template name becomes the key in the Secret. Inside a template, .Secrets holds the keys from Vault, and functions like get, b64enc and b64dec are available.
• Filter keys with includes and excludes, which take lists of regular expressions. Excludes win over includes, and filters never apply to templated keys.

Say the Vault secret contains db-host, db-user and db-password, and your application wants a single connection string:

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: my-app-db
spec:
  vaultAuthRef: vault-auth
  mount: secret
  type: kv-v2
  path: gitlab/my-group/my-project/prod/database
  refreshAfter: 60s
  destination:
    name: my-app-db
    create: true
    transformation:
      # Drop all keys from Vault, keep only the templated ones
      excludes:
        - .*
      templates:
        database-url:
          text: |-
            {{- printf "postgresql://%s:%s@%s:5432/myapp"
                  (get .Secrets "db-user")
                  (get .Secrets "db-password")
                  (get .Secrets "db-host") -}}

The resulting Secret has a single key, database-url.

See Secret transformation and the Transformation API reference for all options and available template functions.

Using the secret in your application

The resulting Secret is a normal Kubernetes Secret, used the same way as described in Creating Kubernetes Secrets with kubectl:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  template:
    spec:
      containers:
      - name: my-app
        env:
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: my-app-secrets
              key: db-password
        - name: API_KEY
          valueFrom:
            secretKeyRef:
              name: my-app-secrets
              key: api-key

tip

Environment variables are only read when the container starts. If you consume secrets as environment variables, add your workload to rolloutRestartTargets so it picks up changes. Secrets mounted as files are updated in the running container after a short delay.

Checking the sync status

You can't read the contents of Secrets in the cluster, but you can see whether the sync works:

kubectl describe vaultstaticsecret my-app-secrets

The events at the bottom show whether the operator was able to read from Vault and write the Secret. Common problems:

• Permission denied - the path is outside your project's subtree, in a folder the namespace's environment can't read, or there's a typo in it. Check that it starts with gitlab/ followed by the full Gitlab path of the project the namespace belongs to, and then prod/, dev/ or shared/. The development namespace can't read from prod/, and the production namespace can't read from dev/.
• Secret not found - the secret doesn't exist in Vault yet, or the path includes secret/ or data/. Leave those out.

Secret outside the GitLab tree in Vault

If your secret is located somewhere else than the gitlab tree in Vault, you need Platon to create a special role for you. Let's say you need to read secrets in productareas/myPO/myApp/databases, or maybe service/myService/databases. You would then have to send an email to Platon via Sikt's internal help mail address, with subject "Platon: ny Vault rolle for lesing av sync secrets til k8s". Remember to supply the following information:

• the Vault path your project needs access to
• which GitLab projects will use the custom role
• a suitable name for the role, not conflicting with existing roles

Please bear in mind to keep the number of roles as few as security allows for your case. For example, maybe several projects (or entire GitLab group) can have access to the same subtree in Vault?

We will create a dedicated VaultAuth in your namespace that logs in with a custom Vault role with access to those secrets. Like vault-auth, it is managed by the platform and can't be changed by you. Reference it by name in vaultAuthRef in the VaultStaticSecrets that read from that path:

spec:
  # The VaultAuth created for you by the Platon team
  vaultAuthRef: <custom-vault-auth>
  mount: secret
  type: kv-v2
  path: <path-outside-gitlab-tree>/<secret-name>

Moving or renaming your Gitlab project

Vault access follows your Gitlab project's path. If you move or rename the project in Gitlab:

1. Move your secrets in Vault to the new path, secret/gitlab/<new-group>/<new-project>/.
2. Update path in your VaultStaticSecrets.
3. Synchronize the project in the PaaS Console. This updates your namespaces so they get access to the new path.

Until you synchronize the project in the PaaS Console, your namespaces keep reading from the old path.

caution

Don't skip the console sync. If someone later creates a new Gitlab project on your old path, your namespaces would be reading that project's secrets.

Using the Vault Agent instead

VSO is the recommended way to get secrets from Vault. If you need features like templating secrets into config files, see Vault Agent.