> For the complete documentation index, see [llms.txt](https://docs.openiam.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.openiam.com/installation/rpm/rpm-ha/component-load-balancing.md).

# Component load balancing

HAProxy on every node — what each front end fronts, how the health checks work, how to change a port, and how to read the state when a backend is marked down.

Your own load balancer spreads **user traffic** across the three hosts. Inside the cluster, a second layer spreads **component traffic**: HAProxy runs on every node, and each node reaches the shared tiers through its own local HAProxy rather than addressing a specific host.

That is what lets a node keep working when a component elsewhere in the cluster is unavailable. If the search tier on node 2 stops answering, node 2's HAProxy sends search traffic to node 1 or node 3 instead, and puts node 2 back in rotation when it recovers. The application never knows.

The cluster installation configures all of this. This page is what you need when you change a port, diagnose a backend that has been marked down, or explain the design to a reviewer.

## What each front end fronts

Every node runs the same set. A service on that node connects to `localhost:<front-end port>`, and HAProxy distributes round-robin across all three hosts.

<table><thead><tr><th width="127.59999084472656">Front end</th><th width="221.79998779296875">Setting</th><th width="99.199951171875" align="center">Default</th><th width="220.20001220703125">Backends</th><th>Mode</th></tr></thead><tbody><tr><td>Interfaces</td><td><code>HAPROXY_UI_PORT</code></td><td align="center"><code>8087</code></td><td>The interface router on all three nodes, port 8080</td><td>HTTP</td></tr><tr><td>Service bus</td><td><code>HAPROXY_ESB_PORT</code></td><td align="center"><code>9081</code></td><td>The service bus on all three nodes, port 9080</td><td>HTTP</td></tr><tr><td>Secret store</td><td><code>HAPROXY_VAULT_PORT</code></td><td align="center"><code>8200</code></td><td>The secret store instances, port 8202</td><td>TCP</td></tr><tr><td>Graph engine</td><td><code>HAPROXY_JANUS_PORT</code></td><td align="center"><code>8183</code></td><td>The graph engine on all three nodes, port 8182</td><td>TCP</td></tr><tr><td>Search tier</td><td><code>HAPROXY_ES_PORT</code></td><td align="center"><code>9201</code></td><td>The search tier on all three nodes, port 9200</td><td>HTTP</td></tr><tr><td>Message broker</td><td><code>HAPROXY_RABBIT_PORT</code></td><td align="center"><code>5673</code></td><td>The broker on all three nodes, port 5672</td><td>TCP</td></tr><tr><td>Broker management</td><td><code>HAPROXY_RABBIT_UI_PORT</code></td><td align="center"><code>5002</code></td><td>The broker's management interface, port 15672</td><td>HTTP</td></tr></tbody></table>

**The cache is the exception.** It is not fronted by HAProxy — the platform's cache client talks to the three watchers directly and follows a promotion when the primary changes. See [*The clustered tiers*](/installation/rpm/rpm-ha/form-the-clusters.md).

## Where the configuration lives

<table><thead><tr><th width="356.800048828125">Path</th><th>Contains</th></tr></thead><tbody><tr><td><code>/etc/haproxy/haproxy.cfg</code></td><td>The global and default settings</td></tr><tr><td><code>/etc/haproxy/conf.d/vault.cfg</code></td><td>One file per tier, generated at installation</td></tr><tr><td><code>/etc/haproxy/conf.d/openiam-ui.cfg</code></td><td></td></tr><tr><td><code>/etc/haproxy/conf.d/openiam-esb.cfg</code></td><td></td></tr><tr><td><code>/etc/haproxy/conf.d/janusgraph.cfg</code></td><td></td></tr><tr><td><code>/etc/haproxy/conf.d/elasticsearch.cfg</code></td><td></td></tr><tr><td><code>/etc/haproxy/conf.d/rabbit.cfg</code></td><td></td></tr><tr><td><code>/etc/haproxy/conf.d/rabbitmq-ui.cfg</code></td><td></td></tr></tbody></table>

The three node addresses in each file come from `CLUSTER_IPS` in the cluster tooling's `infra.conf`, and the front-end port from the matching `HAPROXY_*` setting. **The same files are placed on all three hosts** — every node has the full picture of the cluster.

## How the health checks work

The two front ends that carry application traffic check a real application endpoint rather than a TCP connection, so a backend that is listening but not serving is still taken out of rotation.

<table><thead><tr><th width="173.1999969482422">Front end</th><th>Check</th></tr></thead><tbody><tr><td>Interfaces</td><td><code>GET /idp/actuator/health</code>, expecting a status in the response body</td></tr><tr><td>Service bus</td><td><code>GET /openiam-esb/actuator/health</code>, expecting a status in the response body</td></tr></tbody></table>

Both use the same profile: **a backend is marked down after 10 errors observed at layer 7, and must pass 10 consecutive checks a second apart before it returns** — then it is eased back in over 20 seconds rather than taking a full share immediately. Requests that arrive for a backend that has just gone down are re-dispatched to a healthy one, up to three times.

The effect is that a node restarting does not receive traffic until it is genuinely serving, and a node flapping does not take the cluster with it.

The remaining front ends use a TCP check, which is the appropriate check for those protocols. The broker's front end holds connections open for up to three hours with keep-alive, because a connector's connection is long-lived by design.

## Read the current state

{% code overflow="wrap" %}

```bash
cluster systemctl is-active haproxy
```

{% endcode %}

**Expected: `active` on all three.**

To see which backends a node currently considers healthy:

{% code overflow="wrap" %}

```bash
journalctl -u haproxy --since '1 hour ago' --no-pager | grep -iE 'is (UP|DOWN)'
```

{% endcode %}

**A backend marked down is HAProxy doing its job.** It means that component on that host stopped answering its check — the next step is the tier's own check in [*The clustered tiers*](/installation/rpm/rpm-ha/form-the-clusters.md), not HAProxy itself.

Confirm a front end is answering on the node you are on:

{% code overflow="wrap" %}

```bash
curl -s -o /dev/null -w '%{http_code}\n' http://localhost:8087/idp/actuator/health
curl -s -o /dev/null -w '%{http_code}\n' http://localhost:9081/openiam-esb/actuator/health
```

{% endcode %}

**Expected: `200` from both.** A `503` means every backend for that front end is currently marked down — check the tier, then all three hosts.

## Change a front-end port

Do this **before the build** wherever possible, by setting the matching `HAPROXY_*` value in `infra.conf`.

Afterwards, a port has to change in two places, because the platform's own configuration records where it expects to find each tier:

1. The front end, in `/etc/haproxy/conf.d/<tier>.cfg` on all three hosts.
2. The property that names it, under `/usr/local/openiam/conf/properties/` — this directory is shared across the cluster, so it changes once.

Then, on all three hosts:

{% code overflow="wrap" %}

```bash
haproxy -c -f /etc/haproxy/haproxy.cfg && systemctl reload haproxy
```

{% endcode %}

and restart the services that use that tier.

## Add a node, or replace one

The node addresses are baked into every `conf.d` file at installation. To point the cluster at a replacement host, update `CLUSTER_IPS` in `infra.conf`, regenerate the affected files, and distribute them to all three hosts. Validate before reloading:

{% code overflow="wrap" %}

```bash
haproxy -c -f /etc/haproxy/haproxy.cfg
```

{% endcode %}

**Expected: `Configuration file is valid`.** Reload rather than restart, so established connections are not dropped:

{% code overflow="wrap" %}

```bash
systemctl reload haproxy
```

{% endcode %}

## If it goes wrong

| What you see                                        | What it means                                                                   | What to do                                                                                                                             |
| --------------------------------------------------- | ------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- |
| `503 Service Unavailable` from a front end          | Every backend for that tier is marked down                                      | Check the tier itself — [*The clustered tiers*](/installation/rpm/rpm-ha/form-the-clusters.md) — on all three hosts                    |
| One backend permanently down, the other two fine    | That component on that host is not answering its health check                   | `ssh <node> systemctl status <unit>`, then its log                                                                                     |
| A backend flaps up and down                         | The component is answering intermittently, or the host is resource-starved      | `cluster free -g` and the component's log. The check requires 10 consecutive passes, so flapping means real intermittency              |
| HAProxy will not start after an edit                | A syntax error                                                                  | `haproxy -c -f /etc/haproxy/haproxy.cfg` names the file and line                                                                       |
| A service cannot reach its tier, HAProxy is healthy | The backend is refusing the connection, or a network firewall between the hosts | Confirm the tier answers on the backend host directly, then check the inter-node rules in [*Ports*](/installation/rpm/rpm-ha/ports.md) |
| Connectors drop and reconnect repeatedly            | The broker front end is being reached on the wrong port                         | Remote connectors use `HAPROXY_RABBIT_PORT`, not 5672 — [*Ports*](/installation/rpm/rpm-ha/ports.md)                                   |

## Related

* [*The clustered tiers*](/installation/rpm/rpm-ha/form-the-clusters.md) — what sits behind each front end
* [*Configure the load balancer*](/installation/rpm/rpm-ha/load-balancer.md) — the layer in front of the three hosts
* [*Ports*](/installation/rpm/rpm-ha/ports.md)
* [*Troubleshoot*](/installation/rpm/rpm-ha/troubleshoot.md)

*Applies to OpenIAM 2026.8.1.*


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.openiam.com/installation/rpm/rpm-ha/component-load-balancing.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.