> For the complete documentation index, see [llms.txt](https://docs.openiam.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.openiam.com/installation/kubernetes/troubleshoot.md).

# Troubleshoot

Symptom, meaning and action for an OpenIAM deployment on Kubernetes.

## First, orient yourself

**Set the namespace once**, so nothing below has an unexplained placeholder:

{% code overflow="wrap" %}

```bash
export NS=<the namespace you deployed into>
export DB_HOST=<your database endpoint>
export DB_PORT=<3306, 5432, or 1433>
export DIAG_IMAGE=<a small utility image with a shell, reachable from your cluster>
```

{% endcode %}

**Pin the diagnostic image to a version.** `busybox:1.36` is a common choice. On a restricted network, mirror one into your own registry and use that rather than pulling `latest` from a public registry.

{% code overflow="wrap" %}

```bash
kubectl get pods -n "$NS"
kubectl get pvc -n "$NS"
kubectl get jobs -n "$NS"
kubectl get events -n "$NS" --sort-by=.lastTimestamp | tail -20
```

{% endcode %}

Those four answer most questions before you read a single log. **Events are the most under-used of them** — scheduling failures, mount failures and image pull failures all appear there and nowhere else.

**For any pod that is not running, `describe` before `logs`.** A pod that never started has no log worth reading.

{% code overflow="wrap" %}

```bash
kubectl describe pod -n "$NS" <pod>
kubectl logs -n "$NS" <pod> --previous
```

{% endcode %}

***

## Pods

### A pod stays in `Pending`

Almost always storage or capacity.

{% code overflow="wrap" %}

```bash
kubectl describe pod -n "$NS" <pod> | tail -20
```

{% endcode %}

* **`pod has unbound immediate PersistentVolumeClaims`** — the claim is not bound. `kubectl get pvc -n "$NS"`, then [*Prepare the cluster*](/installation/kubernetes/prepare-the-cluster.md) step 3.
* **`Insufficient memory` or `Insufficient cpu`** — the cluster is short of capacity. Add nodes or move to a larger node size.

### A pod stays in `Init`

It is waiting for something below it — usually a volume, or a tier that has not started. Check the claim first, then the component it depends on.

### A pod is in `CrashLoopBackOff`

**During the first thirty minutes of a deployment this is normal.** Releases are created without waiting for each other, so services start before the tier beneath them is ready and restart until it is.

After that window:

{% code overflow="wrap" %}

```bash
kubectl logs -n "$NS" <pod> --previous | tail -50
```

{% endcode %}

The three causes worth checking first: the secret store is not reachable or is sealed, the database is not reachable, or the search tier could not start because `vm.max_map_count` is too low on the node.

### `ImagePullBackOff`

{% code overflow="wrap" %}

```bash
kubectl describe pod -n "$NS" <pod> | grep -A5 Events
```

{% endcode %}

Either the pull secret is wrong or the cluster has no route to the registry. Confirm the credentials your account team issued, then the cluster's outbound rules.

***

## Storage

### A claim stays `Pending`

{% code overflow="wrap" %}

```bash
kubectl get pvc -n "$NS"
kubectl describe pvc -n "$NS" <claim>
```

{% endcode %}

* **No default storage class** — the claims that do not name one cannot bind.
* **A `ReadWriteMany` claim against a class that does not support it** — several OpenIAM components need shared volumes. EFS on EKS, Azure Files on AKS, NFS or a distributed filesystem on your own cluster.
* **A claim asking for a class named `nfs` that does not exist** — the class name in `values/infra/pvc.values.yaml` was not changed to yours. Change it, run `./setup.sh`, and redeploy.

***

## The database and the schema migration

### The schema migration job failed

{% code overflow="wrap" %}

```bash
kubectl get jobs -n "$NS"
kubectl logs -n "$NS" job/flyway-job | tail -40
```

{% endcode %}

Confirm the database is reachable from the cluster before anything else:

{% code overflow="wrap" %}

```bash
kubectl run db-check -n "$NS" --rm -it --restart=Never --image="$DIAG_IMAGE" -- \
  sh -c 'nc -zv $DB_HOST $DB_PORT'
```

{% endcode %}

Then confirm the engine parameters — [*Managed databases*](/installation/rpm/cloud/managed-databases.md). On MySQL and MariaDB the migration creates stored functions and needs `log_bin_trust_function_creators` set.

To re-run it after a fix, delete **that job only** and redeploy:

{% code overflow="wrap" %}

```bash
kubectl delete job flyway-job -n "$NS"
```

{% endcode %}

Then re-run the deployment — `./setup-no-tf.sh` on the Helm route, `terraform apply` on the Terraform route.

**Do not delete every job in the namespace.** The secret store's bootstrap job and the message broker's keystore generator also live there, they run once, and they do not re-create themselves on a redeploy. Losing them means rebuilding the deployment rather than repairing it.

***

## The secret store

### Everything fails at once

Check whether the store is sealed. Sealed means no component can read its credentials, which presents as many unrelated failures.

{% code overflow="wrap" %}

```bash
kubectl exec -n "$NS" "$(kubectl get pod -n "$NS" | grep -i vault | head -1 | awk '{print $1}')" -- \
  sh -c 'wget -qO- --no-check-certificate https://localhost:8200/v1/sys/seal-status'
```

{% endcode %}

Unseal with the keys recorded at deployment. **Keep those keys somewhere separate** — they are not held anywhere you can recover them from.

***

## Access

### The proxy answers but the console does not load

The reverse proxy has no working backend. The symptom is specific: the proxy returns a response, the page's HTML loads, and everything it requests returns `404`.

Confirm the interface router is deployed and that the proxy names it:

{% code overflow="wrap" %}

```bash
kubectl get svc -n "$NS" | grep -E 'nginx|rproxy'
```

{% endcode %}

**Expected:** both a reverse proxy service and an interface router service. If the router is absent, the deployment did not create it.

### The service has no external address

{% code overflow="wrap" %}

```bash
kubectl get svc -n "$NS" | grep rproxy
```

{% endcode %}

A service of type `LoadBalancer` with no external address means the cluster has no load balancer integration, or the provider has not finished assigning one. On your own cluster, expose it with an ingress instead.

### Links in email point at the wrong address

The content provider's domain pattern is not the published address. Edit it in the console — [*First sign-in*](/installation/rpm/rpm-single-node/first-sign-in.md) step 3.

***

## Logs

| What                            | Where                                                                    |
| ------------------------------- | ------------------------------------------------------------------------ |
| A service                       | `kubectl logs -n "$NS" <pod>`, or `--previous` for the run that failed   |
| A job                           | `kubectl logs -n "$NS" job/<job-name>`                                   |
| Scheduling, mounts, image pulls | `kubectl describe pod -n "$NS" <pod>`, and `kubectl get -n "$NS" events` |
| The deployment run              | The terminal you ran it in — keep the session or capture it to a file    |

Collect everything for a support case at once:

{% code overflow="wrap" %}

```bash
kubectl get -n "$NS" pods,pvc,jobs,svc -o wide > openiam-state.txt
kubectl get -n "$NS" events --sort-by=.lastTimestamp > openiam-events.txt
for p in $(kubectl get -n "$NS" pods --no-headers -o custom-columns=:metadata.name); do
  kubectl logs -n "$NS" "$p" --tail=500 > "logs-$p.txt" 2>/dev/null
done
```

{% endcode %}

**Review before sending.** Service logs can contain identity data from your environment. [*Diagnostics*](/installation/diagnostics.md) covers what else to include in a case.

## Related

* [*Verify a Kubernetes deployment*](/installation/kubernetes/verify.md)
* [*EKS with Terraform*](/installation/kubernetes/terraform-eks.md) and [*AKS with Terraform*](/installation/kubernetes/terraform-aks.md)
* [*Prepare the cluster*](/installation/kubernetes/prepare-the-cluster.md)
* [*Diagnostics*](/installation/diagnostics.md)

*Applies to OpenIAM 2026.8.1.*


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.openiam.com/installation/kubernetes/troubleshoot.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.