> For the complete documentation index, see [llms.txt](https://docs.openiam.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.openiam.com/installation/rpm/rpm-ha/form-the-clusters.md). # The clustered tiers What the cluster build produced, tier by tier — how each is clustered, where it is configured, and the one command that confirms it. Every tier in a three-node deployment is clustered, and the installation forms each cluster for you. This page is the map: what each tier does across three hosts, where it is configured, and the single command that tells you it is healthy. **Read it once after the build**, working down the table, and keep it for the day something needs looking at. The order below is the order the installation builds them, which is also the order to check them in — each tier depends on the ones above it. ## How a request travels in a cluster {% code overflow="wrap" %} ``` your load balancer 80 / 443 │ ┌──────────────────────┼──────────────────────┐ node 1 node 2 node 3 │ │ │ httpd httpd httpd 80 / 443 │ │ │ ┌────┴────┐ ┌────┴────┐ ┌────┴────┐ router HAProxy router HAProxy router HAProxy 8080 9081 8080 9081 8080 9081 │ │ │ that node's └──── round-robins across all three service buses ────┘ interfaces ``` {% endcode %} Each node serves its own interfaces locally and reaches the shared tiers through HAProxy, which is installed on every node. That is what lets any node answer any request: if a component on one host is unavailable, that host's HAProxy sends the work to the same component on another. **Several checks below read a credential from the secret store.** Set them once, before you start: {% code overflow="wrap" %} ```bash V=/usr/local/openiam/utils/vault/vault.fetch.property.sh export REDIS_PASSWORD="$(sudo $V vault.secret.redis.password)" export OPENSEARCH_USER="$(sudo $V vault.secret.opensearch.username):$(sudo $V vault.secret.opensearch.password)" ``` {% endcode %} ## The tiers | Tier | How it is clustered | Confirm with | | ---------------------------- | ----------------------------------------------------------------------------------------------- | ------------------------------------------- | | **Shared storage** | Three directories exported by the first node and mounted on all three | `cluster mount \| grep openiam` | | **Secret store backend** | A three-server cluster with encrypted gossip | `consul members` | | **Secret store** | Three instances sharing that backend, one active | `cluster systemctl is-active openiam-vault` | | **Component load balancing** | HAProxy on every node, one front end per tier | `cluster systemctl is-active haproxy` | | **Search** | A three-node cluster, indexes held on all three | `curl … /_cluster/health` | | **Message broker** | Three brokers in one cluster, queues replicated by quorum | `rabbitmqctl cluster_status` | | **Cache** | One primary with two replicas, and three watchers that promote a replica if the primary is lost | `redis-cli info replication` | | **Database** | A three-node Galera cluster, or your own external database | `SHOW STATUS LIKE 'wsrep_cluster_size'` | | **Graph storage** | A three-node ring, every keyspace held three times | `cluster nodetool status` | | **Graph engine** | One instance per node, fronted by HAProxy | `cluster systemctl is-active janusgraph` | | **Services and interfaces** | The full set on every node | `cluster openiam-cli status` | The rest of this page takes them one at a time. *** ## Shared storage Three directories are common to the whole cluster, so that every node sees the same generated configuration, the same uploaded files, and the same secret-store certificates:
DirectoryHolds
/usr/local/openiam/confThe generated configuration and service properties
/usr/local/openiam/uploadFiles uploaded through the platform
/usr/local/openiam/vault/certsThe secret store's certificates
The first node exports them; all three mount them, with entries in `/etc/fstab` so the mounts return after a restart. The export is restricted to the network you set as `NETWORK` in `infra.conf`. {% code overflow="wrap" %} ```bash cluster ls /usr/local/openiam/conf/properties ``` {% endcode %} **Expected:** the same listing from all three nodes. An empty listing on one node means its mount is not in place — `mount -a` on that host. **This is the tier to check first when nodes disagree about anything.** A node that has lost the mount sees stale or missing configuration and behaves unlike its peers. ## Secret store, and its backend The secret store keeps every credential the platform uses. Its backend is a three-server cluster holding the sealed data, so the store survives the loss of a node. {% code overflow="wrap" %} ```bash consul members ``` {% endcode %} **Expected:** three members, all `alive`. {% code overflow="wrap" %} ```bash cluster systemctl is-active openiam-vault ``` {% endcode %} **Expected:** `active` on all three. The instances share one backend, so one is active for writes at a time and another takes over if it is lost — that is normal and needs nothing from you. Applications reach the store through HAProxy on the port set as `HAPROXY_VAULT_PORT`, which means a service never has to know which instance is currently serving. **Keep the unseal keys and the root token from the build somewhere safe and separate.** Without them a sealed store cannot be opened, and the platform cannot start without its credentials. ## Component load balancing HAProxy runs on every node with one front end per clustered tier:
Front endFrontsDefault port
InterfacesThe interface tier on all three nodes8087
Service busThe service bus on all three nodes9081
Secret storeThe secret store instances8200
Graph engineThe graph engine on all three nodes8183
SearchThe search tier9201
Message brokerThe broker5673
Broker managementThe broker's management interface5002
Configuration is in `/etc/haproxy/conf.d/`, one file per tier, generated from the values in `infra.conf`. The application and interface front ends health-check their backends against the platform's own health endpoints, so a node that stops answering is taken out of rotation and put back when it recovers. {% code overflow="wrap" %} ```bash cluster systemctl is-active haproxy ``` {% endcode %} **Expected:** `active` on all three. ## Search A three-node cluster, with indexes replicated so that every index survives the loss of a node. {% code overflow="wrap" %} ```bash curl -sk -u "$OPENSEARCH_USER" https://localhost:9200/_cluster/health | jq '{status, number_of_nodes}' ``` {% endcode %} **Expected:** `green`, and three nodes. Take the credentials from the secret store rather than typing them on a command line. **`green` is the correct state here** — unlike a single-node deployment, where `yellow` is expected because replicas have nowhere to go. ## Message broker Three brokers in one cluster, named `openiam`. Queues are replicated across the cluster by quorum, which means a queue keeps working while a majority of its members are up. {% code overflow="wrap" %} ```bash rabbitmqctl cluster_status ``` {% endcode %} **Expected:** all three nodes listed as running. Remote connectors reach the broker through HAProxy on `HAPROXY_RABBIT_PORT`, so a connector on another machine keeps working when a node is lost. ## Cache One primary with two replicas, plus a watcher on each of the three nodes. If the primary is lost, the watchers agree on a replica and promote it; the platform's cache client follows the promotion without being reconfigured. {% code overflow="wrap" %} ```bash redis-cli -a "$REDIS_PASSWORD" info replication | grep -E 'role|connected_slaves|slave[0-9]' ``` {% endcode %} **Expected:** one node reporting `role:master` with two connected replicas, and the other two reporting `role:slave`. {% code overflow="wrap" %} ```bash redis-cli -p 26379 -a "$REDIS_PASSWORD" sentinel masters | head -4 ``` {% endcode %} **Expected:** the primary named, with three watchers in agreement. **Which node is the primary changes over time**, and that is the point. Do not treat the first node holding it as a fixed fact. The watchers persist the topology they agree on back into their own configuration file, which is why a three-node deployment runs with SELinux disabled — see [*Prepare the hosts*](/installation/rpm/rpm-ha/prepare-the-hosts.md) step 3. If failover is not working, that is the first thing to check. ## Database If you asked the build for one, the three hosts run a Galera cluster — every node accepts writes, and every write is replicated to all three before it is acknowledged. {% code overflow="wrap" %} ```bash mysql -e "SHOW STATUS LIKE 'wsrep_cluster_size'; SHOW STATUS LIKE 'wsrep_local_state_comment';" ``` {% endcode %} **Expected:** a cluster size of `3` and a state of `Synced`. If you supplied your own database server, the cluster points at it and its availability is arranged on your side. That is the more common production choice — see [*Use a remote database*](/installation/rpm/rpm-single-node/use-a-remote-database.md). ## Graph storage and the graph engine Graph storage is a three-node ring. Both of the platform's keyspaces are held on all three nodes, so any node can answer for any part of the graph. {% code overflow="wrap" %} ```bash cluster nodetool status ``` {% endcode %} **Expected:** three nodes, each `UN` — up and normal. {% code overflow="wrap" %} ```bash nodetool status janusgraph ``` {% endcode %} **Expected:** three nodes, each owning 100% — that is what a three-way copy looks like. The graph engine runs on every node and is reached through HAProxy on `HAPROXY_JANUS_PORT`. {% code overflow="wrap" %} ```bash cluster systemctl is-active janusgraph ``` {% endcode %} **Expected:** `active` on all three. **This tier is the one to start first and stop last.** It is the slowest to settle, and services that depend on it start more cleanly when it is already up. ## Services and interfaces Every node runs the full set of platform services and all six user interfaces. There is no node with a special role in normal operation. {% code overflow="wrap" %} ```bash cluster pgrep -c -u openiam java ``` {% endcode %} **Expected:** a comparable count on all three nodes, twenty or more each. The reverse proxy on each node routes to that node's own interfaces, and to the service bus through HAProxy — which spreads bus work across all three nodes rather than keeping it local. Your load balancer decides which node's proxy a user reaches; [*Configure the load balancer*](/installation/rpm/rpm-ha/load-balancer.md) covers what it needs to do. *** ## Starting and stopping the cluster {% code overflow="wrap" %} ```bash cluster openiam-cli start # all three nodes cluster openiam-cli stop # all three nodes openiam-cli start # this node only ``` {% endcode %} **Stop the platform on a node before restarting that host.** The secret store seals on an unclean stop, and the services on that node cannot read their credentials from a sealed store. ## Related * [*Start and verify*](/installation/rpm/rpm-ha/start-and-verify.md) — the full verification pass, including losing a node on purpose * [*Configure the load balancer*](/installation/rpm/rpm-ha/load-balancer.md) * [*Troubleshoot*](/installation/rpm/rpm-ha/troubleshoot.md) * [*Ports*](/installation/rpm/rpm-ha/ports.md) — what each tier uses between hosts *Applies to OpenIAM 2026.8.1.* --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://docs.openiam.com/installation/rpm/rpm-ha/form-the-clusters.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.