How to debug kubernetes probe issues?

Probe failures sit behind two of the most common Kubernetes symptoms. First, containers stuck in a restart loop, and second, pods that quietly stop receiving traffic. A failing probe is itself a symptom, though. The cause sits in the application, its resources, its dependencies, the network path, or the probe's own configuration. Here's how to work out which one you're looking at.

What is a Kubernetes probe?

A Kubernetes probe is a periodic health check the kubelet runs against a container. On each cycle, the kubelet calls a handler, which can be an HTTP request, a TCP connection attempt, a gRPC health check, or a command executed inside the container, and acts on the result. Probes feed two of Kubernetes' central decisions: which containers to restart, and which pods should receive traffic.

What are the three types of Kubernetes probes?

Production clusters use three probe types. Each asks a different question, fails with different consequences, and points at different causes.

Liveness probes ask "is this container stuck?" On failure, after failureThreshold consecutive misses (default: 3), the kubelet kills the container and the pod's restartPolicy decides whether it comes back. Repeated kills climb the restart count into CrashLoopBackOff. Typical causes are a genuine deadlock or hang, or a probe aggressive enough to kill a busy-but-healthy container.

Readiness probes ask "can this container handle requests right now?" On failure, the pod stays Running but drops out of the ready state. Its endpoint is marked unready in the Service's EndpointSlices, which pulls it out of load balancing. The container keeps running, so the symptom is dropped traffic rather than crashes. Typical causes are a slow dependency behind the health endpoint, incomplete warm-up, or the app shedding load on purpose.

Startup probes ask "has this container finished starting?" While one is configured and hasn't succeeded, liveness and readiness stay disabled. If it never succeeds within failureThreshold × periodSeconds, the container is killed before the other probes ever run. Typical causesare long initialization (migrations, cache warming, JIT) against thresholds set too low. Slow starters without a startup probe get killed by liveness instead.

Liveness and readiness failures both appear as Unhealthy events in kubectl describe pod, but they trigger different responses and usually have different root causes:

Warning  Unhealthy  Liveness probe failed: Get "http://10.1.2.3:8080/healthz": context deadline exceeded
Warning  Unhealthy  Readiness probe failed: HTTP probe failed with statuscode: 503

What mechanisms can probes use?

Each probe uses one of four mechanisms:

• HTTP GET: the kubelet sends a request to a path and port. Status codes 200 through 399 count as success, everything else fails.
• TCP socket: the kubelet tries to open a connection to a port. If the socket opens, the probe passes. That's the entire check, so TCP probes pass on applications that accept connections but can't serve anything, which makes them a weak liveness signal for hung apps.
• Exec: the kubelet runs a command inside the container. Exit code 0 is success, anything else fails.
• gRPC: the kubelet calls the container's gRPC health checking service on a port and treats a SERVING response as success.

gRPC probes

Native gRPC probes went stable in Kubernetes 1.27, after shipping as beta in 1.24. Before native support, the options were TCP probes, which prove nothing about gRPC health, or bundling grpc_health_probe into the image as an exec probe.

livenessProbe:
  grpc:
    port: 9090
  initialDelaySeconds: 10
  periodSeconds: 10

Two gotchas. The port field takes a number only, named ports don't work here. And the kubelet dials without TLS, so the health port needs to accept plaintext connections. There's also an optional service field, which lets a single port back separate liveness and readiness checks by name.

What are the common causes of probe failures?

A probe failure points at one of five places:

1. The application itself
2. Its resources
3. Its dependencies
4. The network path between the kubelet and the pod
5. The probe's configuration

In the first three, something real is wrong and the probe caught it, though a restart isn't always the right response. The last two are false positives. In those cases, the app is fine and the probe just can't see it.

The application is genuinely unhealthy

Sometimes the probe is right. A deadlocked process, a blocked event loop, or an exhausted thread pool. The container is up, the health endpoint can't answer, and a restart is the correct response. This is the case probes exist for, so rule it in before assuming a false alarm.

The tell is in the app's own telemetry rather than the probe config. If request latency climbed or throughput flatlined before the first Unhealthy event, the probe detected a real problem. The restart usually clears it; finding out why the app wedges is what stops the repeat.

The readiness version of this is a pod failing its readiness check under load to shed traffic, which is the mechanism working exactly as designed.

Resource starvation

When a container is starved of CPU, it may not respond within timeoutSeconds (default: 1 second). Either it's doing real work, or it's being throttled at its CPU limit. The application would answer in 2 seconds. The probe gives it 1. After three misses, the kubelet restarts the container, the surviving pods absorb its traffic, and their probes can start failing too.

Warning  Unhealthy  Liveness probe failed: Get "http://10.1.2.3:8080/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

Memory produces the same symptom by a different route. A heap running close to its ceiling drives constant garbage collection, and GC pauses stretch response times past the timeout even though nothing is deadlocked.

Get a usage snapshot from kubectl top pod and compare it against the limits in kubectl describe pod. Averaged usage near the CPU limit is a hint; the confirmation is the CFS throttling metrics (container_cpu_cfs_throttled_periods_total) climbing. Then raise timeoutSeconds to something the app can meet under load, or raise the limit.

One distinction worth keeping straight is that CPU limits throttle while memory limits kill. A container that dies with exit code 137 took a SIGKILL from somewhere, and Reason: OOMKilled in its last state is what confirms the memory limit rather than a probe kill.

Dependency problems behind the health endpoint

A readiness endpoint that checks everything it can reach (database, Redis, certificate expiry) turns one shared dependency blip into every pod failing readiness at the same moment. When no ready backends remain, whatever sits in front starts returning errors, typically the classic Service 503s at the ingress. The hang variant is worse. An endpoint that synchronously waits on a slow downstream times out instead of erroring, which looks identical to a stuck app.

Liveness probes really only need to answer is this process fundamentally broken? A liveness probe that reaches external systems will restart perfectly healthy containers when the actual problem lives somewhere else.

For readiness, decide dependency by dependency. Database connection pool exhausted should probably stop taking traffic. Downstream service running slow means you can probably keep serving.

Network interception between the kubelet and the pod

HTTP, TCP, and gRPC probes originate from the kubelet on the node, not from inside the pod. A standard NetworkPolicy can't block traffic from a pod's own node, so despite the common advice, your NetworkPolicy is rarely the culprit.

The usual suspect is a service mesh sidecar. Under Istio with strict mTLS, the kubelet's plaintext probe hits the Envoy sidecar and gets rejected. Istio's fix is probe rewriting, which is injection rewrites HTTP probes to the agent on port 15020, which forwards them to the app. If that rewrite is disabled, or the sidecar isn't up yet, probes fail while the application sits there healthy.

The test is to kubectl exec into the pod and curl the health endpoint twice, once on localhost and once on the pod IP, because the pod IP is what the kubelet actually probes. Localhost passing while the pod IP fails usually means the app is bound to 127.0.0.1. Both passing while probes still fail points outside the pod. Check the mesh first, then the CNI. And if the image has no curl, kubectl debug with an ephemeral container gets you a shell.

Probe misconfiguration

Probes start before the application is listening. If a container needs 30 seconds to load configuration, connect to databases, and warm caches, but the liveness probe starts after 5, the probe fails, the kubelet restarts the container, and the whole cycle repeats. Sometimes the process isn't even listening yet, so instead of a timeout you get "connection refused". Either way, the events look like:

Warning  Unhealthy  Liveness probe failed: connection refused
Normal   Killing    Container failed liveness probe, will be restarted

The fix is either a startup probe, or, for predictable startup times, an initialDelaySeconds past the worst case. The startup probe is usually the better tool, since a fast start proceeds the moment it passes while a slow one gets the whole budget. failureThreshold: 30 with periodSeconds: 10 gives the container up to 300 seconds to come up before liveness takes over.

Probes too aggressive for the workload. A periodSeconds of 1 with a timeoutSeconds of 1 means the kubelet expects an answer every second, within that second. Applications with variable latency, garbage collection pauses, or bursty load will fail that intermittently. Intermittent readiness failures make pods flap in and out of Service endpoints, which clients experience as random errors. Intermittent liveness failures cause pointless restarts. Each restart can also kills in-flight requests, which is a common source of 502 bad gateway errors at the ingress.

Think about what the probe protects against. Liveness exists for conditions that don't self-resolve, like deadlocks. A container that's slow for 5 seconds and then recovers doesn't need killing. failureThreshold: 5 with periodSeconds: 10 requires roughly 50 seconds of continuous failure before a restart, which catches real deadlocks while tolerating transient slowness.

Wrong port, path, or scheme. A probe aimed at port 8080 while the app listens on 8081 fails instantly with "connection refused". A probe pointing at /healthz when the app serves /health gets a 404. Two subtler versions of the same problem:

1. The app serves HTTPS but the probe's scheme is HTTP
2. The health endpoint sits behind auth middleware that returns 401 or 403, both of which land outside the 200 to 399 success range.

These mismatches creep in when application code changes without the manifest, or when manifests get copied between apps. Named ports add a layer of complexity as a probe can reference a containerPort by name, so the mismatch hides behind the name.

livenessProbe:
  httpGet:
    path: /healthz    # Does this path exist?
    port: web         # Which containerPort does "web" resolve to?
    scheme: HTTPS     # Only if the app actually serves TLS on this port

For HTTPS probes, the kubelet skips certificate verification, so self-signed certs on a health port are fine, though a probe can't present a client certificate, so an endpoint demanding mutual TLS still fails.

How do you diagnose probe failures?

Start with kubectl describe pod and read the Events section. Probe failures show up as Warning Unhealthy with a specific error.

kubectl describe pod my-app-xyz123 | grep -A 20 Events

The error message usually narrows it down fast:

• "connection refused": nothing is listening on the probe port. Wrong port, or the app hasn't started listening yet.
• "context deadline exceeded": no response within timeoutSeconds. The timeout is too short, or the app is too slow.
• "HTTP probe failed with statuscode: 503": the app answered, with an error. Look at what the health endpoint actually checks.
• "no such file or directory" (exec probes): the probe's command doesn't exist in the container image.

When liveness failures repeat, kubectl get pods shows the pod in CrashLoopBackOff with a climbing restart count. CrashLoopBackOff means the kubelet is backing off exponentially between restarts; the Unhealthy events above tell you why the restarts keep happening. And if the pod is stuck in Pending, its probes haven't run at all and the problem sits earlier in the lifecycle, in scheduling, image pulls, or volumes, not in the health checks.

If the pod is running but probes fail, test the endpoint by hand:

kubectl exec -it my-app-xyz123 -- curl -v http://localhost:8080/healthz

Success here with failing probes points to timing (probes fire before the app is ready), timeout (curl will wait longer than the 1-second default), or interception (the network bucket above).

Then check resource pressure and the probe configuration itself:

kubectl top pod my-app-xyz123
kubectl get pod my-app-xyz123 -o yaml | grep -A 15 livenessProbe

Compare initialDelaySeconds against real startup time, and timeoutSeconds against the endpoint's real latency under load.

How do you tune probe configuration?

Five timing and threshold fields control when probes run and how many results count:

1. initialDelaySeconds (default: 0)
2. periodSeconds (10)
3. timeoutSeconds (1)
4. failureThreshold (3)
5. successThreshold (1)

Reasonable starting points that usually need adjustment somewhere. Liveness and startup probes also accept a probe-level terminationGracePeriodSeconds, for when the kill itself needs a different grace period than the pod's.

successThreshold is the one people forget. It defaults to 1 and has to stay 1 for liveness and startup probes. On readiness probes, raising it to 2 or 3 requires that many consecutive passes before the pod re-enters the ready state, which stops a marginal pod from flapping in and out of load balancing.

For slow starters, a startup probe buys time without loosening liveness:

startupProbe:
  httpGet:
    path: /healthz
    port: web
  failureThreshold: 30
  periodSeconds: 10

Here's a full manifest that puts the rest together:

apiVersion: v1
kind: Pod
metadata:
  name: my-app
  labels:
    app: my-app
spec:
  containers:
    - name: my-app
      image: registry.example.com/my-app:1.4.2
      ports:
        - name: web
          containerPort: 8080
      readinessProbe:
        httpGet:
          path: /ready        # Checks dependencies
          port: web
        periodSeconds: 5
        timeoutSeconds: 3
        successThreshold: 2
      livenessProbe:
        httpGet:
          path: /healthz      # Process health only
          port: web
        initialDelaySeconds: 30
        periodSeconds: 10
        timeoutSeconds: 5
        failureThreshold: 3

This is a standalone Pod, so the apiVersion, kind, and metadata shown are the simplest case; in a Deployment or StatefulSet, the same probe blocks sit inside the pod template, under spec.template.spec.containers. Both probes reference the containerPort by its name, web, so the port number lives in exactly one place.

The two endpoints check different things. /ready can inspect dependencies and pull the pod out of rotation, while /healthz only confirms the process responds, so a dependency outage never triggers restarts. Readiness asks "should this pod receive traffic right now?" Liveness asks "is this container beyond saving?" They can share a port, but they usually shouldn't share an answer.

How Resolve AI approaches probe failures

Probe failures often look like application problems but originate elsewhere. A container timing out on health checks might be CPU-throttled because node-level pressure rose after a deployment to a different service. Readiness might be failing because a dependency three hops away started returning errors.

Diagnosing that by hand means correlating pod events, node metrics, deployment history, and dependency health, each living in a different tool with different query languages.

When a probe failure pattern emerges, Resolve AI traces backward from the failing pod to node resource utilization, recent deployments, and the health of whatever the endpoint checks. Root causes that never show up from a single vantage point surface when those domains sit together. Each failure signal points the investigation at different sources:

Failure signal	How Resolve AI investigates it
Restarts / CrashLoopBackOff (liveness)	Kubernetes pod state plus Warning events (Liveness probe failed, BackOff) and restart counts
NotReady / dropped endpoints (readiness)	Kubernetes events (Readiness probe failed), the Loki pod-event stream, endpoint state
Killed during startup (startup)	Event timeline plus the probe configuration in Helm
OOM or CPU throttling underneath	Prometheus: memory against the limit, throttling metrics, restart rate
Dependency hang	Traces and downstream service logs, correlated to probe timing
Misconfigured probe	Probe definitions in the Helm chart or manifest repo

That context also settles the tuning question. An app that reliably starts in 20 seconds but occasionally hits 45 under cluster load needs a different fix than an app whose startup got slower after a specific commit. Telling those apart requires knowing what changed and when, which is exactly the cross-system correlation that's tedious manually and fast with visibility across code, infrastructure, and telemetry.