Creating secrets in PaaS from Gitlab CI/CD

Our Gitlab instance is the integration with Vault and a project has access to secrets in the /secret/gitlab/<group>/<project> tree by default. Additional access outside this path for Gitlab and user access to Vault need to be granted on request by Platon (see here).

Example - secrets using CI/CD variables and exposing them as env vars

When using secrets in the Gitlab CI/CD pipeline we want to make sure we don't expose them in the pipeline or when interacting with the resources in the cluster with kubectl.

The deploy script typically used with gitlab-ci-helpers.yml will redact Secret data values from being shown in the job console output, and using a Kubernetes Secret as an intermediary will avoid exposing the secrets when using kubectl to inspect Kubernetes resources.

The code snippets below show the parts of the CI/CD config and deployment files that are relevant for using secret from Vault, see the example projects for examples of complete files.

.gitlab-ci.yml

This defines a re-usable configuration section (called extends in Gitlab) .vault-secrets, which is then used in the production, staging and review steps.

.vault-secrets:
  id_tokens:
    VAULT_ID_TOKEN:
      aud: "https://vault.sikt.no:8200"
  secrets:
    MY_DB_PASSWORD:
      token: $VAULT_ID_TOKEN
      vault: "gitlab/my-group/my-project/my-db/password@secret"
      file: false
    MY_SECRET_API_KEY:
      token: $VAULT_ID_TOKEN
      vault: "gitlab/my-group/my-project/my-api/key@secret"
      file: false

production:
  extends:
    - .production
    - .vault-secrets
  stage: production
  variables:
    REPLICAS: "2"
  script:
    - deploy deployment.yaml

staging:
  extends:
    - .staging
    - .vault-secrets
  stage: staging
  script:
    - deploy deployment.yaml

review:
  extends:
    - .review
    - .vault-secrets
  stage: review
  script:
    - deploy deployment.yaml

deployment.yaml

The Deployment below show how to export the content of the Kubernetes Secret as environment variables to the containers. Note that it is not a complete and deployable Kubernetes resource. The Secret resource however is complete.

caution

Avoid special characters in secrets as it can break the yaml parsing. Alternatively base64 encode the data.

apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      containers:
      - name: my-app
        env:
        - name: API_KEY
          valueFrom:
            secretKeyRef:
                name: $CI_ENVIRONMENT_SLUG-app-secrets
                key: my-api-key
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
                name: $CI_ENVIRONMENT_SLUG-app-secrets
                key: my-db-pw

---
apiVersion: v1
kind: Secret
metadata:
  name: $CI_ENVIRONMENT_SLUG-app-secrets
  namespace: $KUBE_NAMESPACE
  annotations:
    app.gitlab.com/env: $CI_ENVIRONMENT_SLUG
    app.gitlab.com/app: $CI_PROJECT_PATH_SLUG
  labels:
    app: $CI_ENVIRONMENT_SLUG
    pipeline_id: "$CI_PIPELINE_ID"
    build_id: "$CI_JOB_ID"
type: Opaque
stringData:
  my-api-key: $MY_SECRET_API_KEY
  my-db-pw: $MY_DB_PASSWORD

Example - Secrets as files

This example show how you can store the secrets fetched from Vault as files and then use those to create a Kubernetes Secret.

This method:

• Supports multi-line secrets, such as private key files and certificates.
• Supports special characters, that would otherwise create syntax errors in YAML.

info

The command to actually deploy the secret is very complicated. The reason for this is that kubectl does not have a command to easily create or update a secret. kubectl can create a new secret with kubectl create secret, but that will fail if you try to update the secret.

To work around this, we use --dry-run=client --output=yaml to create a YAML manifest for the secret, and run kubectl apply -f - to apply that manifest.

.gitlab-ci.yml

This defines the same re-usable configuration section (called extends in Gitlab) .vault-secrets as above, but with file: true.

.vault-secrets:
  id_tokens:
    VAULT_ID_TOKEN:
      aud: "https://vault.sikt.no:8200"
  secrets:
    MY_TOKEN_FILE:
      token: $VAULT_ID_TOKEN
      vault: "gitlab/my-group/my-project/credentials/token@secret"
      file: true
    MY_PRIVATE_KEY:
      token: $VAULT_ID_TOKEN
      vault: "gitlab/my-group/my-project/credentials/key@secret"
      file: true

.create-secret:
  before_script:
    - |
      kubectl \
        -n "${KUBE_NAMESPACE}" \
        create secret generic \
        $CI_ENVIRONMENT_SLUG-some-secret \
        --from-file="private_key=${MY_TOKEN_FILE}" \
        --from-file="token=${MY_PRIVATE_KEY}" \
        --dry-run=client \
        --output=yaml \
      | kubectl apply -f -

production:
  extends:
    - .production
    - .vault-secrets
    - .create-secret
  stage: production
  variables:
    REPLICAS: "2"
  script:
    - deploy deployment.yaml

staging:
  extends:
    - .staging
    - .vault-secrets
    - .create-secret
  stage: staging
  script:
    - deploy deployment.yaml

review:
  extends:
    - .review
    - .vault-secrets
    - .create-secret
  stage: review
  script:
    - deploy deployment.yaml