> For the complete documentation index, see [llms.txt](https://docs.openiam.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.openiam.com/troubleshooting/deployment/kubernetes.md).

# Troubleshooting a Kubernetes cluster

Working through an OpenIAM cluster on Kubernetes from the nodes inward, and collecting a log excerpt to send with a ticket.

| Symptom                                             | Likely cause                                          | Resolution                                                       |
| --------------------------------------------------- | ----------------------------------------------------- | ---------------------------------------------------------------- |
| Several services are misbehaving at once            | A node or the cluster beneath them, not the services  | [Work inward](#work-inward-in-this-order), starting at the nodes |
| One pod will not start or keeps restarting          | That component — its own description names the reason | `kubectl describe pod`, then its log                             |
| A component starts, stays up, and still cannot work | It cannot reach another one, or it is being starved   | [Connectivity and resources](#connectivity-and-resources)        |

When a cluster-based installation fails, the fastest route to the cause is to work **from the nodes inward** — cluster, then pods, then one pod's log — because a problem at a lower layer shows up as several services misbehaving at once. Confirm each layer before moving in.

## Work inward, in this order

{% stepper %}
{% step %}
**Are all the nodes up?** Every node should be **Ready**. A node that is **NotReady** is a host or network problem, and nothing above it will make sense until it is fixed.

```bash
kubectl get nodes
```

{% endstep %}

{% step %}
**Are all the pods running?** A pod that is not running tells you which component to look at, and its description names the reason it will not schedule or start.

```bash
kubectl get pods -A
kubectl describe pod <pod_name>
```

{% endstep %}

{% step %}
**What does that component's log say?** This is where the actual error is. Follow it live if the pod is restarting.

```bash
kubectl logs <pod_name>
kubectl logs -f <pod_name>
```

{% endstep %}

{% step %}
**If the logs are clean, check connectivity and resources.** A component that starts, passes its own checks and still cannot work is usually unable to reach another one, or is being starved.
{% endstep %}

{% step %}
**Only then restart anything.** A restart that fixes a symptom without a known cause hides the cause until it recurs.

```bash
kubectl delete pod <pod_name>
```

{% endstep %}
{% endstepper %}

## Connectivity and resources

Between nodes:

```bash
ping <node_ip>
```

To a service:

```bash
curl http://<service_ip>:<port>
```

And the host's own headroom:

```bash
top       # CPU and memory, live
df -h     # free disk
free -m   # available memory
```

## Looking further in

| Check                     | Command                                            |
| ------------------------- | -------------------------------------------------- |
| Cluster events            | `kubectl get events -A`                            |
| A shell inside a pod      | `kubectl exec -it <pod_name> -- /bin/sh`           |
| What is actually deployed | `kubectl get deployment <deployment_name> -o yaml` |
| Storage claims            | `kubectl get pvc -A`                               |

{% hint style="info" %}
If the cause is still not visible, read the node's own system log and check what changed recently — an update, a configuration change, or a certificate that expired. A cluster that worked yesterday and does not today has usually had something done to it.
{% endhint %}

## Sharing a log excerpt

Logs grow large enough to be awkward to send. When support asks for the relevant part, send the end of it rather than the whole thing — the most recent activity is where the error is.

|                                | Command                              |
| ------------------------------ | ------------------------------------ |
| The last 100 lines from a pod  | `kubectl logs --tail 100 <pod_name>` |
| The last 100 lines of a file   | `tail -n 100 application.log`        |
| Follow a file as it is written | `tail -f application.log`            |

{% hint style="info" %}
**A pod usually has no log file to tail.** Containers write to `stdout` and `stderr`, which the runtime collects, so `kubectl logs --tail` is the equivalent of `tail -n` rather than something you run inside the pod.
{% endhint %}

[Getting help and escalating](/troubleshooting/getting-help.md) covers what else to collect before opening a ticket.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.openiam.com/troubleshooting/deployment/kubernetes.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.