> For the complete documentation index, see [llms.txt](https://docs.openiam.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.openiam.com/installation/rpm/rpm-ha/start-and-verify.md).

# Start and verify

Confirm all three nodes are serving, every tier is genuinely clustered, and the platform survives the loss of a node.

## What this covers, and how long it takes

<table><thead><tr><th width="152.3999786376953"></th><th></th></tr></thead><tbody><tr><td><strong>You get</strong></td><td>Evidence that the cluster is running, clustered, and able to lose a node</td></tr><tr><td><strong>Time</strong></td><td>45 minutes, plus up to 20 minutes of waiting on a cold start</td></tr><tr><td><strong>Hosts</strong></td><td>All three, driven from the first node</td></tr><tr><td><strong>Downtime</strong></td><td>None, until the last section — which stops one node deliberately</td></tr><tr><td><strong>Reversible</strong></td><td>Yes. Everything before the node-loss test only reads</td></tr></tbody></table>

## Where this page sits in the sequence

This page runs in two parts, because the load balancer has to exist before the platform can be reached the way users will reach it.

1. **Steps 1 to 5 — now.** Per-node and per-tier checks. Nothing outside the three hosts is needed.
2. [*Configure the load balancer*](/installation/rpm/rpm-ha/load-balancer.md).
3. [*Enable HTTPS*](/installation/rpm/rpm-single-node/enable-https.md), then [*First sign-in*](/installation/rpm/rpm-single-node/first-sign-in.md) through the load balancer's name.
4. **Step 6 and the node-loss test — after that.** Return here.

## Before you start

The build has finished. Give a cold cluster **up to 20 minutes** to settle — the services wait for the infrastructure tiers beneath them, and on three hosts there is more to wait for than on one.

Work from the first node. The `cluster` command runs what follows it on all three hosts in turn, which is what makes most of this page a single command rather than three.

## Set these first

Fill this in once, on the first node.

{% code overflow="wrap" %}

```bash
export NODE1=<first node>
export NODE2=<second node>
export NODE3=<third node>
export NODES="$NODE1 $NODE2 $NODE3"
export OPENIAM_URL=<the load balancer's name>       # needed from step 6 onwards

V=/usr/local/openiam/utils/vault/vault.fetch.property.sh
export REDIS_PASSWORD="$(sudo $V vault.secret.redis.password)"
export OPENSEARCH_USER="$(sudo $V vault.secret.opensearch.username):$(sudo $V vault.secret.opensearch.password)"
```

{% endcode %}

`unset` the last two, or close the session, when you have finished — see [*Commands*](/installation/commands.md#reading-a-credential-from-the-secret-store).

***

## Verify it

### 1. Ask the cluster

{% code overflow="wrap" %}

```bash
cluster openiam-cli status
```

{% endcode %}

**Expected:** every service listed on each of the three nodes. The checks that follow confirm the same picture tier by tier, and are what you keep with your build record.

### 2. Every node is running the platform

{% code overflow="wrap" %}

```bash
cluster pgrep -c -u openiam java
```

{% endcode %}

**Expected: twenty or more on each of the three nodes, and comparable counts.** A node well below its peers has services that did not start — go to step 6 before continuing.

### 3. Every service is running on every node

{% code overflow="wrap" %}

```bash
for h in $NODES; do
  echo "=== $h"
  ssh "$h" 'for j in openiam-esb idm idp auth-manager workflow synchronization; do
    printf "  %-16s %s\n" "$j" "$(pgrep -f "/$j.jar\|/$j.war" >/dev/null && echo yes || echo NO)"
  done'
done
```

{% endcode %}

**Expected: `yes` on every row, on all three nodes.** A service missing on one node while running on the other two usually means that node was not ready when it started — restart that service on that host and re-check.

The names above are the component names, which differ from the systemd unit names for several services — [*Commands*](/installation/commands.md) has the full mapping.

### 4. Every interface answers, on every node

{% code overflow="wrap" %}

```bash
for h in $NODES; do
  echo "=== $h"
  ssh "$h" 'for p in 8082:webconsole 8086:openiam-ui-static 8081:idp \
           8083:selfservice 8084:selfservice-ext 8085:reportviewer; do
    port=${p%%:*}; app=${p##*:}
    printf "  %-22s %s\n" "$app" \
      "$(curl -s -m 5 "http://localhost:$port/$app/actuator/health" | jq -r ".status // \"NO RESPONSE\"")"
  done'
done
```

{% endcode %}

**Expected: `UP` on all six, on all three nodes.** Give the full 20 minutes before treating anything else as a failure.

### 5. Every tier is genuinely clustered

This is the check that distinguishes a cluster from three separate deployments that happen to be running. Work down [*The clustered tiers*](/installation/rpm/rpm-ha/form-the-clusters.md) — one command per tier — or run the summary here.

**Set the two credential variables first.** The cache and search checks read them from the secret store, which keeps the values off your command line:

{% code overflow="wrap" %}

```bash
V=/usr/local/openiam/utils/vault/vault.fetch.property.sh
export REDIS_PASSWORD="$(sudo $V vault.secret.redis.password)"
export OPENSEARCH_USER="$(sudo $V vault.secret.opensearch.username):$(sudo $V vault.secret.opensearch.password)"
```

{% endcode %}

`unset` them, or close the session, when you have finished — see [*Commands*](/installation/commands.md#reading-a-credential-from-the-secret-store).

{% code overflow="wrap" %}

```bash
echo "shared storage:"; cluster ls /usr/local/openiam/conf/properties | head -3
echo "secret store backend:"; consul members
echo "search:"; curl -sk -u "$OPENSEARCH_USER" https://localhost:9200/_cluster/health \
  | jq '{status, number_of_nodes}'
echo "broker:"; rabbitmqctl cluster_status | grep -A4 'Running Nodes'
echo "cache:"; redis-cli -a "$REDIS_PASSWORD" info replication | grep -E 'role|connected_slaves'
echo "graph storage:"; cluster nodetool status
```

{% endcode %}

**Expected, in order:** the same configuration listing from all three; three `alive` members; `green` with three nodes; three running brokers; one primary with two replicas; three `UN` nodes.

**`green` is the correct search state on a cluster.** On a single node `yellow` is expected; here it means an index is not fully replicated and is worth resolving before going further.

Take credentials from the secret store rather than typing them on a command line — anything on a command line is visible to every local user through the process list.

***

## Return here after the load balancer and first sign-in

Steps 1 to 5 confirm the cluster itself. The rest of this page confirms it end to end, and needs [*Configure the load balancer*](/installation/rpm/rpm-ha/load-balancer.md) and [*First sign-in*](/installation/rpm/rpm-single-node/first-sign-in.md) to be complete.

### 6. The platform answers through the load balancer

{% code overflow="wrap" %}

```bash
for i in 1 2 3 4 5 6; do
  curl -s -o /dev/null -w '%{http_code} ' "http://$OPENIAM_URL/idp/login"
done; echo
```

{% endcode %}

**Expected: `200` six times.** Then confirm the requests actually spread across the three hosts by watching each node's proxy log while you repeat it:

{% code overflow="wrap" %}

```bash
cluster tail -5 /var/log/httpd/mod_openiam_access_log
```

{% endcode %}

**Confirm you can sign in** at the load balancer's name. If you have not completed [*First sign-in*](/installation/rpm/rpm-single-node/first-sign-in.md) yet, do it now, setting the content provider's **Domain Pattern** to the load balancer's name rather than any individual host's.

### 7. If a check failed

Work upward from the bottom of the stack, because each tier depends on the ones below it:

1. **Shared storage.** A node that has lost its mount sees stale configuration and behaves unlike its peers. `cluster mount | grep openiam`.
2. **The secret store and its backend.** Almost everything fetches credentials from it, and a failure here produces confusing symptoms everywhere else.
3. **The remaining infrastructure tiers** — graph storage, cache, broker, search.
4. **The services**, once everything below them is confirmed.

Restart one service on one node:

{% code overflow="wrap" %}

```bash
ssh "$NODE3" systemctl restart "openiam-$SERVICE"   # substitute the node and unit you are fixing
```

{% endcode %}

Its log names the reason if it does not come back:

{% code overflow="wrap" %}

```bash
ssh "$NODE3" openiam-cli log "$SERVICE"
```

{% endcode %}

**Compare the three nodes rather than reading one in isolation.** On a cluster the fastest diagnosis is almost always the difference between the node that works and the node that does not:

{% code overflow="wrap" %}

```bash
cluster systemctl is-active openiam-esb
cluster df -h /usr/local/openiam/conf
cluster free -g
```

{% endcode %}

***

## Prove it survives a node loss

**This is the test the whole shape exists for.** Do it before the deployment carries anything real, and record what you see.

### 1. Take one node out

**Choose a node that is not the shared-storage host.** The first node exports the filesystem the other two mount, so stopping it tests something different — see [*Scope and availability*](/installation/scope-and-availability.md). This test measures what the clustered tiers do when they lose a member.

{% code overflow="wrap" %}

```bash
ssh "$NODE3" openiam-cli stop
```

{% endcode %}

**Test the shared-storage node separately, and deliberately.** Stopping it removes the shared configuration from all three hosts at once. If your deployment must survive that, provide the shared filesystem from resilient storage you already run, and then this test applies to any node.

### 2. Confirm the platform keeps serving

{% code overflow="wrap" %}

```bash
for i in $(seq 1 10); do
  curl -s -o /dev/null -w '%{http_code} ' "http://$OPENIAM_URL/idp/login"
done; echo
```

{% endcode %}

**Expected: `200` throughout.** A short run of errors while the load balancer notices is normal; a sustained failure is not.

**Confirm a session established before the stop still works** — sign in first, then stop the node, then continue using the console. The cache tier holds session state across all three nodes, so a user whose node disappears should continue without signing in again.

### 3. Confirm each tier is degraded but serving

{% code overflow="wrap" %}

```bash
consul members                                    # two alive, one failed
rabbitmqctl cluster_status                        # two running
nodetool status                                   # two UN, one DN
redis-cli -a "$REDIS_PASSWORD" info replication   # a primary still present
```

{% endcode %}

**If the stopped node was holding the cache primary, another node will now report `role:master`.** That is the failover working.

### 4. Bring it back

{% code overflow="wrap" %}

```bash
ssh "$NODE3" openiam-cli start
```

{% endcode %}

Then repeat step 5 of *Verify it*. **Every tier should return to three members without intervention.**

### 5. Record the result

| Record                                                                      | Why                                        |
| --------------------------------------------------------------------------- | ------------------------------------------ |
| How long the load balancer took to stop sending traffic to the stopped node | This is your user-visible failover time    |
| Whether existing sessions survived                                          | The answer your service owner will ask for |
| How long each tier took to return to three members after the restart        | Sets expectations for planned maintenance  |

Keep this with your build record. Handing over a cluster with its node-loss test attached saves the next person — often you — from re-establishing what healthy looked like on day one.

## If it goes wrong

| What you see                                              | What it means                          | What to do                                                                                                                        |
| --------------------------------------------------------- | -------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- |
| One node has far fewer processes than its peers           | Its services did not all start         | Compare `cluster df -h /usr/local/openiam/conf` and `cluster free -g` first; a lost mount or exhausted memory explains most cases |
| Search reports `yellow` on the cluster                    | An index is not fully replicated       | Confirm all three search nodes are in the cluster; replication settles on its own once they are                                   |
| Graph storage shows a node as `DN`                        | That node's storage is not in the ring | Start it, allow a minute, and re-check. Bring this tier up before the services that use it                                        |
| The cache reports no primary                              | The watchers have not agreed on one    | Confirm all three watchers are running and can reach each other on their port                                                     |
| Nodes disagree about configuration                        | One has lost the shared mount          | `mount -a` on that host, then restart its services                                                                                |
| Users are signed out when a node fails                    | Session state is not being shared      | Confirm the cache tier is clustered — step 5                                                                                      |
| The load balancer keeps sending traffic to a stopped node | Its health check is not strict enough  | [*Configure the load balancer*](/installation/rpm/rpm-ha/load-balancer.md) step 2                                                 |

## What to do next

The cluster is running and resilient. From here:

* **Complete first sign-in** — [*First sign-in*](/installation/rpm/rpm-single-node/first-sign-in.md), using the load balancer's name as the published address.
* **Connect your first managed system, define policy and synchronise users** — the *Administration Guide*.
* **Size the deployment for your load.** Every value here is a shipped default; sizing is in the [Solution Architecture and Design Guide](/solution-architecture/deployment/sizing.md).
* **Take a backup** — [*Back up and restore*](/installation/rpm/rpm-single-node/back-up-and-restore.md).

## What this procedure does not cover

Configuring the platform, and the availability of the database, the load balancer and the shared filesystem — each is arranged separately, as [*Scope and availability*](/installation/scope-and-availability.md) sets out. [*First sign-in*](/installation/rpm/rpm-single-node/first-sign-in.md) and [*Configure*](/installation/rpm/rpm-single-node/configure.md) come next.

## Related

* [*The clustered tiers*](/installation/rpm/rpm-ha/form-the-clusters.md) — the per-tier detail behind step 4
* [*Configure the load balancer*](/installation/rpm/rpm-ha/load-balancer.md)
* [*Troubleshoot*](/installation/rpm/rpm-ha/troubleshoot.md)
* [*Configure*](/installation/rpm/rpm-single-node/configure.md) — what to review now it is up

*Applies to OpenIAM 2026.8.1.*


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.openiam.com/installation/rpm/rpm-ha/start-and-verify.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.