> For the complete documentation index, see [llms.txt](https://docs.openiam.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.openiam.com/installation/rpm/rpm-ha/troubleshoot.md).

# Troubleshoot

Symptom, meaning and action for a three-node cluster — including the checks that compare the three hosts against each other.

Everything in [*Troubleshoot*](/installation/rpm/rpm-single-node/troubleshoot.md) for a single node applies to each host in a cluster. This page covers what is specific to running three.

## Set these first

{% code overflow="wrap" %}

```bash
export NODE1=<first node>
export NODE2=<second node>
export NODE3=<third node>
export NODES="$NODE1 $NODE2 $NODE3"
export NODE=$NODE3          # the node you are currently investigating

V=/usr/local/openiam/utils/vault/vault.fetch.property.sh
export REDIS_PASSWORD="$(sudo $V vault.secret.redis.password)"
export OPENSEARCH_USER="$(sudo $V vault.secret.opensearch.username):$(sudo $V vault.secret.opensearch.password)"
```

{% endcode %}

## Start here: compare the three

**On a cluster, the fastest diagnosis is almost always the difference between the node that works and the node that does not.** Run this from the first node before anything else:

{% code overflow="wrap" expandable="true" %}

```bash
for h in $NODES; do
  echo "=== $h"
  ssh "$h" '
    echo "  services: $(pgrep -c -u openiam java)"
    echo "  conf:     $(ls /usr/local/openiam/conf/properties 2>/dev/null | wc -l) files"
    echo "  mounts:   $(mount | grep -c openiam)"
    echo "  memory:   $(free -g | awk "/^Mem:/ {print \$7}") GB available"
    echo "  disk:     $(df -h / | awk "NR==2 {print \$4}") free"
    echo "  proxy:    $(systemctl is-active httpd)"
    echo "  haproxy:  $(systemctl is-active haproxy)"'
done
```

{% endcode %}

[*Diagnostics*](/installation/diagnostics.md) packages this and more into a single collection script.

**Expected:** three nearly identical blocks. Whatever differs is where to look.

***

## Cluster membership

### A tier is missing a node

Each tier reports its own membership. Ask the tier directly rather than inferring from the platform:

{% code overflow="wrap" %}

```bash
consul members                     # secret store backend
rabbitmqctl cluster_status         # message broker
cluster nodetool status            # graph storage
curl -sk -u "$OPENSEARCH_USER" https://localhost:9200/_cat/nodes    # search
redis-cli -a "$REDIS_PASSWORD" info replication                     # cache
```

{% endcode %}

A node absent from **one** tier is usually that component on that host. A node absent from **every** tier is the host — check that it is up, that its clock matches the others, and that nothing between the hosts started filtering the inter-node ports.

### The cache does not fail over, or the watchers disagree

{% code overflow="wrap" %}

```bash
getenforce
```

{% endcode %}

**Expected: `Disabled` on all three hosts.** A three-node deployment requires it — the watchers rewrite their own configuration as the topology changes, and an enforcing policy prevents that. The symptom is usually not an error at all: failover does not happen, and users are signed out when a node is lost. [*Prepare the hosts*](/installation/rpm/rpm-ha/prepare-the-hosts.md) step 3 has the reason and the fix.

{% code overflow="wrap" %}

```bash
redis-cli -p 26379 -a "$REDIS_PASSWORD" sentinel masters | head -4
```

{% endcode %}

**Expected:** the same primary named by all three watchers, with a quorum of three.

### A node rejoins but does not stay

Almost always the clock. Cluster protocols and certificate validation both depend on time agreeing across the three:

{% code overflow="wrap" %}

```bash
cluster date -Is
cluster timedatectl status | grep -i synchronized
```

{% endcode %}

**Expected:** times within a second or two, and synchronisation active on all three.

### Graph storage shows a node as `DN`

{% code overflow="wrap" %}

```bash
cluster nodetool status
```

{% endcode %}

Start that node's storage, allow a minute, and re-check before starting anything above it. **This tier is the slowest to settle and the most sensitive to being started after the services that use it.** When bringing a node back after maintenance, let graph storage rejoin the ring before starting the platform on that host.

***

## Shared storage

### One node sees different configuration from the others

{% code overflow="wrap" %}

```bash
cluster ls /usr/local/openiam/conf/properties | head -3
cluster mount | grep openiam
```

{% endcode %}

**Expected:** the same listing and three mounts from every node. A node missing them has lost the shared mount, and everything it does from then on is based on stale or absent configuration.

{% code overflow="wrap" %}

```bash
ssh "$NODE" mount -a
ssh "$NODE" ls /usr/local/openiam/conf/properties
```

{% endcode %}

Then restart that node's services.

### The mounts do not return after a restart

The entries are in `/etc/fstab` on each host. Confirm they are present:

{% code overflow="wrap" %}

```bash
cluster grep openiam /etc/fstab
```

{% endcode %}

If the shared-storage host is slower to come back than its peers, a node can start before the export is available. `mount -a` and a service restart on that node resolves it; if it recurs, start the shared-storage host first when bringing the cluster up from cold.

***

## Load balancing

### The external load balancer keeps a failed node in rotation

Its health check is not strict enough. Check a real application path rather than a TCP connection or the proxy's home page — [*Configure the load balancer*](/installation/rpm/rpm-ha/load-balancer.md) step 2.

### One node's HAProxy shows a backend down

{% code overflow="wrap" %}

```bash
cluster systemctl is-active haproxy
```

{% endcode %}

HAProxy on each node health-checks the tier behind it and removes what is not answering. **A backend marked down is HAProxy working, not HAProxy failing** — find out why that component on that host is not answering, using the tier's own check above.

{% code overflow="wrap" %}

```bash
cluster journalctl -u haproxy --since '30 min ago' --no-pager | tail -20
```

{% endcode %}

### Users are signed out when a node fails

Session state is shared through the cache tier. Confirm it is clustered:

{% code overflow="wrap" %}

```bash
redis-cli -a "$REDIS_PASSWORD" info replication | grep -E 'role|connected_slaves'
```

{% endcode %}

**Expected:** one primary with two replicas. If each node reports itself as a primary with no replicas, the tier is not clustered and each node is holding its own sessions.

Also review session affinity at the load balancer — affinity that pins a user to one node for the life of their session prevents the failover the cache tier makes possible.

***

## Starting and stopping

### The cluster does not come back cleanly after a full stop

Bring it up in the order the tiers depend on each other:

1. Shared storage host and mounts
2. Secret store backend, then the secret store
3. Graph storage, then the graph engine
4. Broker, cache, search
5. `cluster openiam-cli start`

### The secret store is sealed on a node

{% code overflow="wrap" %}

```bash
ssh "$NODE" curl -sk https://localhost:8200/v1/sys/seal-status | jq
```

{% endcode %}

Unseal it with the keys recorded at the build, then start the platform on that node. **Stop the platform with `openiam-cli stop` before restarting a host** — the store seals on an unclean stop, and services cannot read their credentials while it is sealed.

### A command only affected one node

`openiam-cli` and `systemctl` act locally. Prefix with `cluster` to run across all three:

{% code overflow="wrap" %}

```bash
openiam-cli start           # this node
cluster openiam-cli start   # all three
```

{% endcode %}

***

## Logs

<table><thead><tr><th width="236.39999389648438">What</th><th>Where</th></tr></thead><tbody><tr><td>The cluster build</td><td><code>install.log</code> in the cluster tooling directory on the first node</td></tr><tr><td>A service</td><td><code>/usr/local/openiam/logs/&#x3C;name>.log</code> on each node. The unit-to-log-name mapping is in <a href="/installation/commands.md"><em>Commands</em></a></td></tr><tr><td>Reverse proxy</td><td><code>/var/log/httpd/mod_openiam_error_log</code> on each node</td></tr><tr><td>Component load balancing</td><td><code>journalctl -u haproxy</code> on each node</td></tr><tr><td>Interface router</td><td><code>/var/log/nginx/</code> on each node</td></tr><tr><td>Host services</td><td><code>journalctl -u openiam-&#x3C;service></code> on each node</td></tr></tbody></table>

Read the same log on all three at once:

{% code overflow="wrap" %}

```bash
cluster openiam-cli log openiam-esb
```

{% endcode %}

**Treat the build log as sensitive.** It records the configuration of the deployment; keep a copy somewhere appropriate and remove it from the host.

## Related

* [*Troubleshoot*](/installation/rpm/rpm-single-node/troubleshoot.md) — everything that applies per host
* [*The clustered tiers*](/installation/rpm/rpm-ha/form-the-clusters.md) — the healthy state of each tier
* [*Start and verify*](/installation/rpm/rpm-ha/start-and-verify.md)
* [*Configure the load balancer*](/installation/rpm/rpm-ha/load-balancer.md)

*Applies to OpenIAM 2026.8.1.*


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.openiam.com/installation/rpm/rpm-ha/troubleshoot.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.