Skip to main content
Gå til innhold

Troubleshooting

Pipeline failures​

ERROR: deploy repository path is not set​

The generate-manifests-* job fails straight away with this message.

Cause: The DEPLOY_REPOSITORY_PATH CI/CD variable is missing from your project, so the pipeline doesn't know which deploy repository to trigger. This happens when the project has not been bootstrapped yet.

Fix: Get your project bootstrapped (see Getting started) by asking the Platon team on #platon on Slack. Bootstrapping sets the variable for you.

CRITICAL: Found Secret in <file>​

CRITICAL: Found Secret in .../manifests-generated/secret.yaml. Secrets should not be directly deployed with Argo CD.

Cause: A kind: Secret resource was found in your rendered manifests. The pipeline blocks this to keep secrets out of Git.

Fix: Remove the Secret manifest and read the secret from Vault with a VaultStaticSecret instead. See Accessing Vault secrets.

ERROR: no manifests were generated from 'manifests'​

Cause: The manifests/ directory is missing, empty, or contains no .yaml or .yml files. The job refuses to continue, because an empty environment would make ArgoCD delete everything in it.

Fix: Check that your templates are committed in manifests/. If they live in another directory, set the templates-dir input on the component.

ERROR: <file> is not valid YAML after variable substitution​

Cause: A template became invalid YAML once its variables were filled in. A common cause is a value containing characters such as : or # that needs quoting, or a variable that was empty.

Fix: Download the manifests-generated/ artifact from the failed job and look at the file named in the error. Add quotes around the placeholder in the template if needed, for example value: "$MY_VAR".

Empty values in the rendered manifests​

Rendered manifests contain empty values where you expected a variable, for example image: :abc123.

Cause: envsubst replaces every placeholder in your templates, such as $CI_REGISTRY_IMAGE, with the value of that variable. If the variable is not set, the placeholder becomes an empty string instead of being left as it is.

Fix: Set the variable in one of two ways:

  • GitLab project settings: go to Settings → CI/CD → Variables and add it there. Use this for values that differ per environment.
  • .gitlab-ci.yml: define it under the variables key. Use this for non-sensitive, static values:
variables:
  MY_VAR: "my-value"
caution

Because every $ is treated as a variable, a literal $ in your manifests (for example in a shell script inside a ConfigMap) is replaced as well. Keep such content out of manifests/, or build it in a way that does not use $.

The pipeline does not run the deploy jobs​

Cause: The component's jobs only run in pipelines triggered by a push to a branch. Pushing a tag, scheduled pipelines and manually started pipelines do not deploy.

Fix: Push a commit to your branch.

No staging jobs in the pipeline​

Cause: Staging is off by default.

Fix: Set with-staging: true on the component include (see Configure your pipeline). If staging was not enabled when your project was bootstrapped, commit this change first, then have the project bootstrapped again so that the staging application is created in ArgoCD. Ask the Platon team to bootstrap the project again for you.

Downstream deploy job fails: the fetched artifact contains no manifests​

Cause: The deploy repository's pipeline fetches the rendered manifests from the latest successful generate-manifests-* job on your branch. The artifact it found was empty or had expired.

Fix: Re-run the pipeline on your branch.

ArgoCD issues​

Production or staging stays OutOfSync after the pipeline succeeded​

Cause: Production and staging don't sync automatically.

Fix: Open the application and click Sync → Synchronize. If the new commit doesn't show yet, click Refresh first: ArgoCD only checks the deploy repository every 3 minutes.

permission denied when syncing production​

Cause: Only maintainers of the GitLab group that owns the project can sync production. Developers can sync review and staging.

Fix: Ask a maintainer to run the sync, or ask a group owner to give you the Maintainer role in GitLab. See Permissions.

Review application does not appear​

Cause: ArgoCD may not have seen the new directory in the deploy repository yet, or the application failed to be created.

Fix:

  1. Wait a few minutes. ArgoCD checks the deploy repository every 3 minutes.
  2. Check the deploy repository (argocd-apps/gl-proj-<id>) for a rev-<branch-slug>/ directory. If it's missing, check the deploy-review job and its downstream pipeline.
  3. Very long branch names can make the namespace name longer than Kubernetes' 63-character limit. Use a shorter branch name.

ComparisonError: failed to load target state​

ArgoCD cannot read the manifests from the deploy repository.

Cause: Usually a manifest that is valid YAML but not a valid Kubernetes resource, for example a typo in apiVersion or kind.

Fix: Click the application to read the full error message, then check the matching file in the manifests-generated/ artifact of your pipeline.

Resource stuck in Progressing​

A Deployment or Pod stays in Progressing and never becomes Healthy.

Cause: The container is failing to start. Common reasons: wrong image tag, a missing environment variable or Secret, or a failing readiness probe.

Fix: Click the pod in the ArgoCD UI and open the Logs and Events tabs, or use kubectl:

kubectl logs -n gl-proj-<id>-<environment> <pod-name>
kubectl describe pod -n gl-proj-<id>-<environment> <pod-name>

Look for ImagePullBackOff, CrashLoopBackOff, or readiness probe failures in the output.

Degraded: Deployment does not have minimum availability​

Cause: One or more pods are crashing repeatedly (CrashLoopBackOff).

Fix:

kubectl get pods -n gl-proj-<id>-<environment>
kubectl logs -n gl-proj-<id>-<environment> <pod-name> --previous

The --previous flag shows the logs from the last crashed instance, which are often more useful than those of the current one.

Vault secret is not created​

The VaultStaticSecret exists, but the Kubernetes Secret it should create does not.

Cause: Usually a wrong path, or vaultAuthRef not set to vault-auth.

Fix: Read the events of the VaultStaticSecret to see why it fails. In the ArgoCD UI, open the application, click the VaultStaticSecret resource and select the Events tab. From the command line, run kubectl describe vaultstaticsecret -n <namespace> <name>. Check that the secret exists in Vault under gitlab/<group>/<project>/, which is the only path your project can read.

Manual kubectl changes are reverted​

Cause: Review environments have self-heal turned on, so ArgoCD keeps the cluster equal to Git.

Fix: Make the change in manifests/ and push. If you need to experiment directly in a review environment, see Sync policies.

Getting help​

If none of the above resolves your issue, reach out to the Platon team on #platon on Slack.