Skip to main content
Gå til innhold

Troubleshooting

Sometimes applications deployed to Kubernetes fail to work properly. This document contains some tips for troubleshooting them.

Kubeconfig, namespace​

The examples we give here assume that you have your kubeconfig file in its default location (~/.kube/config) or have the environment variable KUBECONFIG pointing to it. We also assume that you are accessing the namespace that is configured in your kubeconfig. For other namespaces, you have to add -n <namespace> to the command lines.

Viewing pods, deployments, etc.​

There are two ways to view objects, e.g. pods and deployments from the command line:

  • kubectl describe <objects>
  • kubectl get <objects>

Where object can be a large number of things. Try

kubectl get

to see the full list.

kubectl describe gives a structured, textual description. E.g.:

kubectl describe ingress basic

returning

    Name:                   basic
    Namespace:              my-namespace
    Address:                10.2.0.6,10.2.0.4,10.2.0.3
    Default backend:        default-http-backend:80 (<none>)
    Rules:
      Host                          Path    Backends
      ----                          ----    --------
      basic.temp.paas2.uninett.no
                                    /       basic:80 (<none>)
    Annotations:
    No events.

For pods and deployments, the description is typically much longer. The list of events at the bottom is useful for debugging.

kubectl get by default returns a list with one line per item. E.g.:

kubectl get pods

NAME READY STATUS RESTARTS AGE anders-ghost-3911434811-0mwhz 1/1 Running 0 55m basic-1025096757-721t0 1/1 Running 0 16m

There are other output formats, chosen with the -o option. -o wide adds more info to the single line format:

kubectl get pods -o wide

NAME READY STATUS RESTARTS AGE IP NODE anders-ghost-3911434811-0mwhz 1/1 Running 0 59m 10.11.87.148 paas2-worker-0 basic-1025096757-721t0 1/1 Running 0 20m 10.11.87.151 paas2-worker-0

There are many ways to choose which objects to include. Object name is straightforward:

kubectl get pods -o wide basic-1025096757-721t0

NAME READY STATUS RESTARTS AGE IP NODE basic-1025096757-721t0 1/1 Running 0 23m 10.11.87.151 paas2-worker-0

We can select by label, typically specified in the metadata of the deployment. The following will give include all pods in the deployment:

kubectl get pods -o wide -l app=basic

For single line output we also have grep, although we lose the heading. However, this does not work well for multiline output.

-o yaml gives the yaml description of the object. It includes attributes set by the system as well as those we specified when creating it. We can get the same info as json using -o json.

Other output formats:

Getting logs​

Here is how to get logs for an example pod:

kubectl logs basic-1025096757-6fgxk

This will include output to stdout and stderr. If the container logs to files, you have to shell into it and hunt.

kubectl logs -f tails the log. kubectl logs --previous shows logs from a crashed container.

If there is more than one container in the pod, you have to add the name of the container you are interested in. Run e.g.

    kubectl get pod complex-4241715055-2pjdr -o jsonpath=$'{.spec.containers[*].name}\n'

returning e.g.

hammer nail

and use the container name:

kubectl logs -f complex-4241715055-2pjdr nail

This get pod command uses the output format option jsonpath, which lets you select attributes. Details in the user guide.

Getting a shell inside a container​

Most container images, but not all, include a shell. If it does, you can get a shell on running pod like so:

kubectl exec -ti basic-1983954029-xjtl5 -- sh

Where basic-1983954029-xjtl5 is an example pod name. The image may include bash, but many do not, so it is a good idea to try sh first. More about kubectl exec here.

From the container, you can examine files and processes. If it includes the right tools (curl, netcat, ...), you can access ports of other containers in the same namespace. A container in namespace kube-system can access all namespaces.

You can run an image with a shell, and add the tools you need to it. alpine is a minimal Linux.

kubectl run -ti --image=alpine alpine sh

once inside, we install curl and run it:

apk update apk add curl curl 10.x.y.z

The connection to the shell may time out, in which case you can reconnect to the same shell:

kubectl attach -ti alpine-3835730047-j6fnx

or run a new shell in the same pod:

kubectl exec -ti alpine-3835730047-bq456 sh

Again, the actual pod name will vary. kubectl run created a deployment to keep an alpine instance running. You may want to delete it.

kubectl delete deployment alpine

More about these commands:

Kubectl debug​

kubectl debug is a powerful command introduced in Kubernetes to simplify the troubleshooting and debugging of running applications in a Kubernetes cluster.

kubectl debug can add ephemeral containers to running pods without disrupting the existing containers. These debug containers can run custom tools and scripts necessary for troubleshooting.

Attach a Debug Container to a Running Pod:

kubectl -n <namespace-name> debug <pod-name> -it --image=busybox

See the Kubernetes kubectl debug docs for more information.

Forwarding a port to you local machine​

You can forward a port from a pod to your local machine. Details here. Here is how to forward port 80 on the pod to 8080 on your machine:

kubectl port-forward basic-1025096757-8xz1x 8080:80 &