> For the complete documentation index, see [llms.txt](https://docs.openiam.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.openiam.com/troubleshooting/message-broker/1-rabbitmq-reinit.md).

# RabbitMQ cluster went out of order

Re-initialising a RabbitMQ cluster that has lost its state, and the vhost and permission set the OpenIAM services need back afterwards.

| Symptom                                                             | Likely cause                                                                         | Resolution           |
| ------------------------------------------------------------------- | ------------------------------------------------------------------------------------ | -------------------- |
| The RabbitMQ management UI cannot be reached                        | The broker is not serving, or the cluster has no quorum                              | Re-initialise, below |
| A service log complains that it cannot create or connect to a queue | The vhost the service needs is missing, or the node holding it is out of the cluster | Re-initialise, below |

The usual causes are an unexpected network change, a node's hostname changing, or a node running out of disk space — all three leave the cluster unable to agree on its own membership.

{% hint style="danger" %}
**Re-initialising discards every message currently in RabbitMQ.** It is the recovery for a cluster that cannot form, not a way to clear a backlog. Everything in flight is lost, so confirm the cluster is genuinely broken before starting.
{% endhint %}

{% hint style="warning" %}
**Do not enable `rabbitmq_delayed_message_exchange` while re-initialising.** From 2026.9.2 OpenIAM does not use it — the plugin has no build past RabbitMQ 4.2, and enabling it pins the broker to a version that can no longer be upgraded. Older copies of the script below include the line; remove it.
{% endhint %}

## Re-initialise the cluster

{% stepper %}
{% step %}
**Stop RabbitMQ on every node.**

```bash
cluster systemctl stop rabbitmq-server
```

{% endstep %}

{% step %}
**Remove the Mnesia directory on every node.** This is the state that has gone bad.

```bash
cluster rm -rf /var/lib/rabbitmq/mnesia/
```

{% endstep %}

{% step %}
**Start RabbitMQ on every node**, then join the others to the first from that first node.

```bash
cluster systemctl start rabbitmq-server
```

{% code overflow="wrap" %}

```bash
ssh node2 rabbitmqctl stop_app
ssh node2 rabbitmqctl join_cluster rabbit@node1
ssh node2 rabbitmqctl start_app
ssh node3 rabbitmqctl stop_app
ssh node3 rabbitmqctl join_cluster rabbit@node1
ssh node3 rabbitmqctl start_app
```

{% endcode %}
{% endstep %}

{% step %}
**Recreate the vhosts, the `openiam` user and its permissions.** An empty broker has none of them, and a service whose vhost is missing cannot start.

{% code overflow="wrap" expandable="true" %}

```bash
#!/bin/bash
#### RABBITMQ INITIALIZATION SCRIPT
. /usr/local/openiam/env.conf
export VAULT_CERTS="$HOME_DIR/vault/certs/"
export JAVA_HOME="$HOME_DIR/jdk"
export VAULT_HOME="$HOME_DIR/utils/vault/"

. ${VAULT_HOME}validate.vault.sh

export RABBITMQ_PASSWORD=$(. ${VAULT_HOME}vault.fetch.property.sh vault.secret.rabbitmq.password)

if [ -z "$RABBITMQ_PASSWORD" ] || [ "$RABBITMQ_PASSWORD" == "null" ]; then
	echo "cannot get vault.secret.rabbitmq.password property from vault"
	exit 1;
fi

rabbitmq-plugins enable rabbitmq_management

rabbitmqctl add_user openiam $RABBITMQ_PASSWORD
rabbitmqctl set_user_tags openiam administrator

VHOSTS="openiam_am openiam_idm openiam_audit openiam_common openiam_connector
openiam_activiti openiam_user openiam_groovy_manager openiam_machine_learning
openiam_synchronization openiam_bulk_synchronization openiam_ext_log
openiam_reconciliation openiam_bulk_reconciliation openiam_business_rule
openiam_sas openiam_pbac openiam_nhi openiam_privacy"

for VHOST in $VHOSTS; do
	rabbitmqctl add_vhost "$VHOST"
	rabbitmqctl set_permissions -p "$VHOST" openiam ".*" ".*" ".*"
done
```

{% endcode %}

{% hint style="warning" %}
**Confirm the vhost list against your own release before running it.** The set grows as subsystems are added — PBAC, non-human identities, privacy and SAS all arrived after the 4.2.1.x line, and a copy of this script predating them re-initialises the broker and leaves those services unable to connect. The deployment's own definition is the authority; `rabbitmqctl list_vhosts` afterwards is how you check you have them all.
{% endhint %}
{% endstep %}

{% step %}
**Start the OpenIAM services** and confirm each one connects. The services declare their own queues on the vhosts above, so there is nothing further to create by hand.
{% endstep %}
{% endstepper %}

## High availability needs no policy

{% hint style="info" %}
**Do not apply `ha-all` / `ha-mode` policies.** Older copies of this procedure end with a script of `rabbitmqctl set_policy … '{"ha-mode":"all"}'` lines. That was classic queue mirroring, **removed in RabbitMQ 4.0** — OpenIAM runs 4.2, so those commands are rejected rather than applied.

Replication is now a property of the queue itself: OpenIAM declares its queues as **quorum** queues, which are replicated across the cluster by design. Fanout queues are the one exception and are declared classic deliberately. Nothing has to be set after a re-initialisation to get HA back.
{% endhint %}


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.openiam.com/troubleshooting/message-broker/1-rabbitmq-reinit.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.