> For the complete documentation index, see [llms.txt](https://docs.openiam.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.openiam.com/installation/kubernetes/troubleshoot.md).

# Troubleshoot

Symptom, meaning and action for an OpenIAM deployment on Kubernetes.

## First, orient yourself

**Set the namespace once**, so nothing below has an unexplained placeholder:

{% code overflow="wrap" %}

```bash
export NS=<the namespace you deployed into>
export DB_HOST=<your database endpoint>
export DB_PORT=<3306, 5432, or 1433>
export DIAG_IMAGE=<a small utility image with a shell, reachable from your cluster>
```

{% endcode %}

**Pin the diagnostic image to a version.** `busybox:1.36` is a common choice. On a restricted network, mirror one into your own registry and use that rather than pulling `latest` from a public registry.

{% code overflow="wrap" %}

```bash
kubectl get pods -n "$NS"
kubectl get pvc -n "$NS"
kubectl get jobs -n "$NS"
kubectl get events -n "$NS" --sort-by=.lastTimestamp | tail -20
```

{% endcode %}

Those four answer most questions before you read a single log. **Events are the most under-used of them** — scheduling failures, mount failures and image pull failures all appear there and nowhere else.

**For any pod that is not running, `describe` before `logs`.** A pod that never started has no log worth reading.

{% code overflow="wrap" %}

```bash
kubectl describe pod -n "$NS" <pod>
kubectl logs -n "$NS" <pod> --previous
```

{% endcode %}

***

## Pods

### A pod stays in `Pending`

Almost always storage or capacity.

{% code overflow="wrap" %}

```bash
kubectl describe pod -n "$NS" <pod> | tail -20
```

{% endcode %}

* **`pod has unbound immediate PersistentVolumeClaims`** — the claim is not bound. `kubectl get pvc -n "$NS"`, then [*Prepare the cluster*](https://docs.openiam.com/installation/kubernetes/prepare-the-cluster.md) step 3, *Storage*.
* **`Insufficient memory` or `Insufficient cpu`** — the cluster is short of capacity. Add nodes or move to a larger node size.

### A pod stays in `Init`

It is waiting for something below it — usually a volume, or a tier that has not started. Check the claim first, then the component it depends on.

### A pod is in `CrashLoopBackOff`

**During the first thirty minutes of a deployment this is normal.** Releases are created without waiting for each other, so services start before the tier beneath them is ready and restart until it is.

After that window:

{% code overflow="wrap" %}

```bash
kubectl logs -n "$NS" <pod> --previous | tail -50
```

{% endcode %}

The three causes worth checking first: the secret store is not reachable or is sealed, the database is not reachable, or the search tier could not start because `vm.max_map_count` is too low on the node.

### `ImagePullBackOff`

{% code overflow="wrap" %}

```bash
kubectl describe pod -n "$NS" <pod> | grep -A5 Events
```

{% endcode %}

Either the pull secret is wrong or the cluster has no route to the registry. Confirm the credentials your account team issued, then the cluster's outbound rules.

**If you side-loaded the image onto the nodes yourself** ([Bake the CA into the ESB image](#bake-the-ca-into-the-esb-image)), there is a third cause, and it reports the same way — `ErrImagePull` with *"pull access denied, repository does not exist or may require authorization"* even though the image is on the node, or `ErrImageNeverPull` under `pullPolicy: Never`. The name the chart renders does not match the name you imported. The chart builds it as `{registry}{registryPrefixSeparator}{prefix}/<service>:{version}-{environment}`, so each of these produces a different name and a failed pull:

| Mistake                                      | Produces                                                      |
| -------------------------------------------- | ------------------------------------------------------------- |
| `openiam-esb` rather than `openiam/esb`      | `docker.io/library/openiam-esb`                               |
| `registryPrefixSeparator: "-"`               | `docker.io/library-openiam/esb`                               |
| An `environment` that is not what you tagged | `…/esb:2026.10.2-cacerts` where the node has `…-prod-cacerts` |
| A version that is not the one you built      | `…/esb:2026.6.2-prod` against a `2026.10.2-prod` image        |

Compare the two directly and make them identical — the `Image:` line against what the node holds:

{% code overflow="wrap" %}

```bash
kubectl describe pod -n "$NS" esb-0 | grep -i '^\s*Image:'
ctr -n k8s.io images list | grep esb        # on the node itself
```

{% endcode %}

**Images must be imported into containerd's `k8s.io` namespace.** `kubelet` cannot see an image loaded with `docker load`, or imported into any other namespace — the pod fails to pull something that is demonstrably on the host.

***

## Storage

### A claim stays `Pending`

{% code overflow="wrap" %}

```bash
kubectl get pvc -n "$NS"
kubectl describe pvc -n "$NS" <claim>
```

{% endcode %}

* **No default storage class** — the claims that do not name one cannot bind.
* **A `ReadWriteMany` claim against a class that does not support it** — several OpenIAM components need shared volumes. EFS on EKS, Azure Files on AKS, NFS or a distributed filesystem on your own cluster.
* **A claim asking for a class named `nfs` that does not exist** — the class name in `values/infra/pvc.values.yaml` was not changed to yours. Change it, run `./setup.sh`, and redeploy.

***

## The database and the schema migration

### The schema migration job failed

{% code overflow="wrap" %}

```bash
kubectl get jobs -n "$NS"
kubectl logs -n "$NS" job/flyway-job | tail -40
```

{% endcode %}

Confirm the database is reachable from the cluster before anything else:

{% code overflow="wrap" %}

```bash
kubectl run db-check -n "$NS" --rm -it --restart=Never --image="$DIAG_IMAGE" -- \
  sh -c 'nc -zv $DB_HOST $DB_PORT'
```

{% endcode %}

Then confirm the engine parameters — [Managed databases](https://docs.openiam.com/installation/rpm/cloud/managed-databases.md). On MySQL and MariaDB the migration creates stored functions and needs `log_bin_trust_function_creators` set.

To re-run it after a fix, delete **that job only** and redeploy:

{% code overflow="wrap" %}

```bash
kubectl delete job flyway-job -n "$NS"
```

{% endcode %}

Then re-run the deployment — `./setup-no-tf.sh` on the Helm route, `terraform apply` on the Terraform route.

**Do not delete every job in the namespace.** The secret store's bootstrap job and the message broker's keystore generator also live there, they run once, and they do not re-create themselves on a redeploy. Losing them means rebuilding the deployment rather than repairing it.

***

## The secret store

### Everything fails at once

Check whether the store is sealed. Sealed means no component can read its credentials, which presents as many unrelated failures.

{% code overflow="wrap" %}

```bash
kubectl exec -n "$NS" "$(kubectl get pod -n "$NS" | grep -i vault | head -1 | awk '{print $1}')" -- \
  sh -c 'wget -qO- --no-check-certificate https://localhost:8200/v1/sys/seal-status'
```

{% endcode %}

Unseal with the keys recorded at deployment. **Keep those keys somewhere separate** — they are not held anywhere you can recover them from.

***

## TLS to the search tier

### Services fail with `PKIX path validation failed: signature check failed`

The service logs an `SSLHandshakeException` with `CertPathValidatorException: signature check failed` — or *"Signature does not match"* — while reaching a search index. On the ESB the Spring context then fails with a cascade of `UnsatisfiedDependencyException` and bean-creation errors; those are consequences, not the fault. Other services simply land in `CrashLoopBackOff`.

**The cause is a certificate with the right subject and the wrong key.** Three copies of the search tier's CA have to agree, and a redeployment can regenerate the certificates without the other two following:

<table><thead><tr><th width="50" align="center">#</th><th>Copy</th><th>Where it lives</th></tr></thead><tbody><tr><td align="center">1</td><td>What the running search pod actually serves</td><td><code>opensearch.ca.pem</code> in the pod</td></tr><tr><td align="center">2</td><td>What the services mount</td><td>the <code>opensearch-certificates</code> secret</td></tr><tr><td align="center">3</td><td>What a custom ESB image trusts</td><td><code>cacerts</code> baked into the image</td></tr></tbody></table>

Compare the SHA-256 fingerprints; all three must be identical:

{% code overflow="wrap" expandable="true" %}

```bash
# 1 — the CA the running pod serves
kubectl exec -n "$NS" opensearch-cluster-master-0 -c opensearch -- \
  cat /usr/share/opensearch/config/opensearch.ca.pem | openssl x509 -noout -subject -fingerprint -sha256

# 2 — the CA in the secret the services mount
kubectl get secret -n "$NS" opensearch-certificates -o jsonpath='{.data.opensearch\.ca\.pem}' \
  | base64 -d | openssl x509 -noout -subject -fingerprint -sha256

# 3 — the CA inside a custom ESB truststore
keytool -list -v -keystore ./cacerts -storepass changeit 2>/dev/null \
  | grep -A12 'Alias name: opensearch-ca' | grep SHA256:
```

{% endcode %}

**If 2 differs from 1**, rebuild the secret from the running pod. Check each key and certificate pair matches before applying it — the two hashes must be equal, or the tier will not start at all:

{% code overflow="wrap" expandable="true" %}

```bash
kubectl get secret -n "$NS" opensearch-certificates -o yaml > opensearch-certificates.backup.yaml
mkdir -p /tmp/os-certs && cd /tmp/os-certs
for f in opensearch.pem opensearch.key.pem opensearch.ca.pem admin.pem admin.key.pem; do
  kubectl exec -n "$NS" opensearch-cluster-master-0 -c opensearch -- \
    cat "/usr/share/opensearch/config/$f" > "$f"
done

openssl x509 -in opensearch.pem -noout -pubkey | sha256sum
openssl pkey  -in opensearch.key.pem -pubout   | sha256sum

kubectl create secret generic opensearch-certificates -n "$NS" \
  --from-file=opensearch.pem --from-file=opensearch.key.pem --from-file=opensearch.ca.pem \
  --from-file=admin.pem --from-file=admin.key.pem \
  --dry-run=client -o yaml | kubectl apply -f -
```

{% endcode %}

**If 3 differs from 1**, replace the entry in the truststore, then rebuild and reload the image.

{% hint style="danger" %}
**Delete the alias before importing, or the old certificate stays.** `keytool -importcert` refuses an alias that already exists — it reports *"alias already exists"* and leaves the previous certificate in place. The import appears to have run, the fingerprints still disagree, and the handshake still fails.
{% endhint %}

{% code overflow="wrap" %}

```bash
cp cacerts cacerts.bak
keytool -delete     -alias opensearch-ca -keystore ./cacerts -storepass changeit
keytool -importcert -alias opensearch-ca -file opensearch.ca.pem -keystore ./cacerts \
        -storepass changeit -noprompt
```

{% endcode %}

{% hint style="warning" %}
**Trusting the CA is not always sufficient.** The generated `opensearch.pem` is self-signed with the same distinguished name as the CA — `openssl verify` reports *"self-signed certificate"* — so a client may need the exact certificate the tier serves, not just its issuer. Where the ESB still reports `PKIX` after the CA is in place, import the served certificate too, under its own alias, and rebuild:

```bash
keytool -importcert -alias opensearch-server -file opensearch.pem -keystore ./cacerts \
        -storepass changeit -noprompt
```

{% endhint %}

Restart the affected pods afterwards and watch one come up before restarting the rest:

{% code overflow="wrap" %}

```bash
kubectl delete pod -n "$NS" idm-0
kubectl logs -n "$NS" idm-0 -f | grep -iE 'PKIX|SSLHandshake|Started'
```

{% endcode %}

`keytool -cacerts` reads the truststore of whatever machine it runs on, which is rarely the one you mean. Always name the file: `keytool -list -keystore ./cacerts`. The search pod has no `openssl`, so read its files out with `kubectl exec … cat` and pipe them to `openssl` where you are working.

### Bake the CA into the ESB image

The fix above edits the truststore inside a running pod, which a restart or a rescheduled pod undoes. Where the ESB has to carry the CA permanently — a restore into an environment built separately, or a rebuild that keeps regenerating the search tier's certificates — build the truststore into the image instead.

The rest of this page assumes you have done this when it talks about an image you side-loaded yourself.

{% stepper %}
{% step %}
**Clear a stale alias from the truststore you are about to ship.** An export taken from an older environment carries that environment's Vault certificate, and `keytool -importcert` will not overwrite an alias that already exists.

{% code overflow="wrap" %}

```bash
keytool -list   -keystore cacerts -storepass changeit | grep vault
keytool -delete -alias <old-environment>-vault -keystore cacerts -storepass changeit
keytool -list   -keystore cacerts -storepass changeit | grep vault
```

{% endcode %}
{% endstep %}

{% step %}
**Build the image with the corrected `cacerts`.** Put the file in an empty directory with a `.dockerignore` containing `*.tar`, so earlier image tars are not sent into the build context.

{% code overflow="wrap" expandable="true" %}

```bash
mkdir -p /root/esb-custom-image && cd /root/esb-custom-image
# copy the corrected cacerts here, then:
cat > .dockerignore <<'EOF'
*.tar
EOF

cat > Dockerfile <<'EOF'
FROM registry.openiam.com/openiam_service/esb:2026.x.x-prod
USER root
# keep the original alongside it
RUN cp /usr/lib/jvm/default-jvm/lib/security/cacerts \
       /usr/lib/jvm/default-jvm/lib/security/cacerts.orig
COPY cacerts /usr/lib/jvm/default-jvm/lib/security/cacerts
RUN chmod 664 /usr/lib/jvm/default-jvm/lib/security/cacerts && \
    chown openiam:openiam /usr/lib/jvm/default-jvm/lib/security/cacerts
USER openiam
EOF

docker build -t docker.io/library/openiam/esb:2026.x.x-prod-cacerts .
docker save  docker.io/library/openiam/esb:2026.x.x-prod-cacerts \
             -o openiam-esb-2026.x.x-prod-cacerts.tar
```

{% endcode %}

{% hint style="warning" %}
**Tag it exactly as the chart renders the name**, `docker.io/library/openiam/esb:<version>-<environment>`. `openiam-esb` with a hyphen does not work — the chart always puts a `/` between the prefix and the service name, and the mismatch surfaces later as a failed pull rather than as a naming error.
{% endhint %}
{% endstep %}

{% step %}
**Confirm the CA is actually in the image** before shipping it to the nodes.

{% code overflow="wrap" %}

```bash
docker run --rm --entrypoint sh docker.io/library/openiam/esb:2026.x.x-prod-cacerts -c \
  'keytool -list -v -cacerts -storepass changeit 2>/dev/null | grep -A12 "Alias name: opensearch-ca" | grep "SHA256:"'
```

{% endcode %}

The fingerprint must match the one the search tier serves.
{% endstep %}

{% step %}
**Import it on every node where an ESB pod can be scheduled.** The ESB runs three replicas, so a node you skip schedules a pod that cannot pull.

{% code overflow="wrap" expandable="true" %}

```bash
kubectl get nodes
kubectl debug node/<node-name> -it --image=busybox
kubectl cp openiam-esb-2026.x.x-prod-cacerts.tar \
  <debug-pod>:/host/var/tmp/images/openiam-esb-2026.x.x-prod-cacerts.tar

kubectl exec -it <debug-pod> -- sh
chroot /host bash

# remove any image already carrying this tag, or the import is a no-op
ctr -n k8s.io images rm     docker.io/library/openiam/esb:2026.x.x-prod-cacerts
ctr -n k8s.io images import /var/tmp/images/openiam-esb-2026.x.x-prod-cacerts.tar
ctr -n k8s.io images list | grep esb
```

{% endcode %}

If the imported name differs, retag it rather than rebuilding:

{% code overflow="wrap" %}

```bash
ctr -n k8s.io images tag <imported-name> docker.io/library/openiam/esb:2026.x.x-prod-cacerts
```

{% endcode %}

Check the digest against the build. An unchanged digest means the old tar was imported.
{% endstep %}

{% step %}
**Point the chart at it.** From this release the image is configured in the values file, not in the StatefulSet template.

{% code overflow="wrap" expandable="true" %}

```bash
vi /usr/local/openiam/kubernetes-docker-configuration/values/services/esb.values.yaml
```

{% endcode %}

{% code overflow="wrap" %}

```yaml
image:
  environment: "prod-cacerts"
  pullPolicy: "IfNotPresent"   # "Never" while debugging
  version: "2026.x.x"
  prefix: "openiam"
  registry: "docker.io/library"
  registryPrefixSeparator: "/"
```

{% endcode %}

Then render the name and read it before deploying — this is the step that catches a tag mismatch while it is still cheap:

{% code overflow="wrap" %}

```bash
helm template esb <chart-path> -f values/services/esb.values.yaml | grep image:
# expected: docker.io/library/openiam/esb:2026.x.x-prod-cacerts
```

{% endcode %}
{% endstep %}
{% endstepper %}

Replace `2026.x.x` with the release you are deploying, in every command above and in the values file.

***

## Access

### The proxy answers but the console does not load

The reverse proxy has no working backend. The symptom is specific: the proxy returns a response, the page's HTML loads, and everything it requests returns `404`.

Confirm the interface router is deployed and that the proxy names it:

{% code overflow="wrap" %}

```bash
kubectl get svc -n "$NS" | grep -E 'nginx|rproxy'
```

{% endcode %}

**Expected:** both a reverse proxy service and an interface router service. If the router is absent, the deployment did not create it.

### The service has no external address

{% code overflow="wrap" %}

```bash
kubectl get svc -n "$NS" | grep rproxy
```

{% endcode %}

A service of type `LoadBalancer` with no external address means the cluster has no load balancer integration, or the provider has not finished assigning one. On your own cluster, expose it with an ingress instead.

### Links in email point at the wrong address

The content provider's domain pattern is not the published address. Edit it in the console — [First sign-in](https://docs.openiam.com/installation/rpm/rpm-single-node/first-sign-in.md) step 3, Create the first content provider.

***

## Logs

| What                            | Where                                                                    |
| ------------------------------- | ------------------------------------------------------------------------ |
| A service                       | `kubectl logs -n "$NS" <pod>`, or `--previous` for the run that failed   |
| A job                           | `kubectl logs -n "$NS" job/<job-name>`                                   |
| Scheduling, mounts, image pulls | `kubectl describe pod -n "$NS" <pod>`, and `kubectl get -n "$NS" events` |
| The deployment run              | The terminal you ran it in — keep the session or capture it to a file    |

Collect everything for a support case at once:

{% code overflow="wrap" %}

```bash
kubectl get -n "$NS" pods,pvc,jobs,svc -o wide > openiam-state.txt
kubectl get -n "$NS" events --sort-by=.lastTimestamp > openiam-events.txt
for p in $(kubectl get -n "$NS" pods --no-headers -o custom-columns=:metadata.name); do
  kubectl logs -n "$NS" "$p" --tail=500 > "logs-$p.txt" 2>/dev/null
done
```

{% endcode %}

**Review before sending.** Service logs can contain identity data from your environment. [Diagnostics](https://docs.openiam.com/installation/diagnostics.md) covers what else to include in a case.

## Related

* [*Verify a Kubernetes deployment*](https://docs.openiam.com/installation/kubernetes/verify.md)
* [*EKS with Terraform*](https://docs.openiam.com/installation/kubernetes/terraform-eks.md) and [*AKS with Terraform*](https://docs.openiam.com/installation/kubernetes/terraform-aks.md)
* [*Prepare the cluster*](https://docs.openiam.com/installation/kubernetes/prepare-the-cluster.md)
* [*Diagnostics*](https://docs.openiam.com/installation/diagnostics.md)


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.openiam.com/installation/kubernetes/troubleshoot.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
