Skip to main content
Gå til innhold

Setting resources (requests and limits)

We are introducing a new node scheduler

Setting requests and limits is becoming more important, because we are introducing Karpenter, a service that chooses what nodes to add to the PaaS cluster depending on what requests are set on the workloads. There will be a lot less room for error when setting those, because the nodes will have very little overhead CPU or memory beyond the workloads.

Every container you run declares two numbers for CPU and for memory: a request and a limit.

  • The request is what Kubernetes reserves for you. It decides which node you land on, and it is the share you are guaranteed when that node is busy.
  • The limit is the ceiling. Cross it and Kubernetes either slows your container down or kills it, depending on which resource you crossed.

The rules​

  • Always set a CPU request. Without one you have no reserved share of the processor, and you are the last thing scheduled whenever the node is busy.
  • Always set a memory request. This is what Kubernetes reserves for you, and what protects you when the node runs short.
  • Set the memory limit equal to the memory request. Anything you use above your request is memory nobody reserved for you.
  • Do not set a CPU limit. A CPU request already guarantees your share. A limit only stops you using capacity that is sitting idle.
resources:
  requests:
    cpu: 100m       # 95th percentile of what you actually use
    memory: 256Mi   # peak usage, plus about 20%
  limits:
    memory: 256Mi   # the same as the request
    # no cpu limit, deliberately

Two situations justify breaking these rules, and both are described in when to break the rules.

What happens if you get it wrong​

Requests that are too low​

Kubernetes reserves what you ask for and nothing more. If you request 50m of CPU and routinely use 500m, you are borrowing the difference from whatever else happens to share your node, and you are only entitled to it for as long as nobody else wants it. When the node gets busy your workload will be slowed down and if it is memory you are borrowing, your workload will be evicted.

A container with no CPU request does not get an average share. It gets the smallest share the kernel allows, which puts it last in the queue whenever the node is busy.

Requests that are too high​

An inflated request is capacity reserved for you and used by nobody. Across the PaaS cluster, workloads currently request roughly five times the CPU they actually use. Sikt pays for all of that capacity.

A memory limit that is too low​

Memory cannot be slowed down — when your container tries to allocate past its memory limit, it is killed. You will see OOMKilled, and the container restarts. Under steady load it will happen again, and you get a restart loop.

Check this before assuming your application has a memory leak. If you never set a memory limit, yours is 200Mi, see what you get if you set nothing.

A CPU limit that binds​

CPU limits are not applied as an average over time. They are applied as a quota inside every 100 millisecond window, counted across all of your threads at once. An application running eight threads with a 500m limit can spend its entire budget in the first few milliseconds of a window, then sit frozen until the window rolls over.

The result is latency that does not match your CPU graph: the average looks low while your p99 is high. This is a common cause of unexplained slowness on Kubernetes.

Finding out what your app needs​

Both numbers are already being measured, so you do not have to estimate them.

Start with the dashboard​

In Grafana, switch to the Sikt organisation, then open Dashboards → Kubernetes → Kubernetes / Compute Resources / Pod and select your namespace and your pod. It shows CPU and memory usage, alongside the requests and limits currently set.

If you want a precise answer​

The dashboards can give you a good estimate of what you should set your requests to, but the most precise way is querying the metrics and get a numeric value.

Switch to your own Organisation in Grafana. Run the queries in Explore against the data source "Platon App Clusters", replacing my-namespace. Set the query type to Instant and the format to Table to get one row per pod with the number in its own column. The seven-day window is inside the query, so the panel's time range makes no difference.

95th percentile CPU, in millicores:

max by (pod, container) (
  quantile_over_time(0.95,
    rate(container_cpu_usage_seconds_total{namespace="my-namespace", container!=""}[5m])[7d:5m]
  )
) * 1000

Peak memory, in MiB:

max by (pod, container) (
  max_over_time(container_memory_working_set_bytes{namespace="my-namespace", container!=""}[7d])
) / 1024 / 1024

The worst throttling in the window, if you have set a CPU limit. A result of 0.05 means 5% of those 100 millisecond windows were cut short. An empty result or NaN means no CPU limit is set, so nothing is throttling you:

max by (pod, container) (
  max_over_time(
    (
      rate(container_cpu_cfs_throttled_periods_total{namespace="my-namespace", container!=""}[5m])
    / rate(container_cpu_cfs_periods_total{namespace="my-namespace", container!=""}[5m])
    )[7d:5m]
  )
)

Take the highest row for the workload you are sizing.

Turning measurements into values​

  • CPU request: the 95th percentile. You may burst above your CPU request whenever the node has room, so padding buys you nothing and costs a larger node.
  • Memory request and limit: the peak, plus about 20%. You may not burst above your memory limit — you are killed instead — so memory needs some margin.

Pad your CPU request and you waste money. Fail to pad your memory request and you get OOMKilled.

To check whether a container has already been killed for memory, find the pod first — the RESTARTS column tells you which one to look at:

kubectl -n <namespace> get pods
kubectl -n <namespace> describe pod <pod-name>

Look for Last State: Terminated with Reason: OOMKilled.

When to break the rules​

When a CPU limit is right​

  • Workloads that may consume whatever is available. Batch jobs, scheduled imports, and anything that will use a whole node if one is free.
  • When behaviour has to be identical everywhere. Benchmarking, or a workload whose timing must not depend on how busy its neighbours are.
  • Runtimes that size their thread pools from the CPU limit. With no limit, a JVM or Go application sizes itself to the node's core count rather than to your share of it, which on a large node means more threads — and more memory — than you intended. Pin it yourself with GOMAXPROCS or -XX:ActiveProcessorCount, or set a limit. The Go team's container-aware GOMAXPROCS explains the trade-off, and the JVM stopped inferring this from the CPU request in JDK 19.

If you set one, set it well above your request, and check the throttling query above afterwards to confirm it is not binding in normal operation.

When the memory limit should differ from the request​

If your application has a rare, brief peak — a monthly report, a bulk import — a limit above the request lets it survive that peak rather than be killed by it.

There is a trade-off. Memory above your request is not reserved for you: you may or may not get it, and while you are using it you are the first to be evicted if the node runs short. Setting the request at the real peak is usually the better choice. A gap makes sense when the peak is rare enough that reserving for it permanently would be wasteful.

What you get if you set nothing​

Every namespace has defaults:

  • Memory request 100Mi, memory limit 200Mi. Applied automatically to any container that does not set its own.
  • No CPU default at all. A container that does not request CPU gets no reservation, and therefore the smallest share the kernel allows.

The default memory resource settings are there so that an unconfigured workload still starts. You should set your own values based on actual usage.

Your namespace also carries a quota, by default 16Gi of memory requests and 22Gi of memory limits in total.

Learn more​