RabbitMQ cluster went out of order

Issue overview

Problem with RabbitMQ can occur due to various reasons, most common are network unexpected changes, change of hostname of one (or more) of the node, running out of diskspace at one (ore more) node. In these cases you might have issue with connecting to RabbitMQ UI manager or one of OpenIAM service ends up with error and in log of service you can observe complains about not able to create/connect with queue.

Important note: following steps provided below will lead to losing current messages in RabbitMQ.


Solution is to re-initialize RabbitMQ cluster.

  1. stop RabbitMQ service on all nodes
cluster systemctl stop rabbitmq-server
  1. remove directory /var/lib/rabbitmq/mnesia/ from all nodes
  2. start RabbitMQ service on all nodes, and join nodes in cluster from first node
cluster systemctl start rabbitmq-server
ssh node2 rabbitmqctl stop_app
ssh node2 rabbitmqctl join_cluster rabbit@node1
ssh node2 rabbitmqctl start_app
ssh node3 rabbitmqctl stop_app
ssh node3 rabbitmqctl join_cluster rabbit@node1
ssh node3 rabbitmqctl start_app
  1. Create hosts, user and assign permissions to it by running following script.
##set -e
. /usr/local/openiam/env.conf
export VAULT_CERTS="$HOME_DIR/vault/certs/"
export JAVA_HOME="$HOME_DIR/jdk"
export VAULT_HOME="$HOME_DIR/utils/vault/"
. ${VAULT_HOME}validate.vault.sh
export RABBITMQ_PASSWORD=$(. ${VAULT_HOME}vault.fetch.property.sh vault.secret.rabbitmq.password)
if [ -z "$RABBITMQ_PASSWORD" ] || [ "$RABBITMQ_PASSWORD" == "null" ]; then
echo "cannot get vault.secret.rabbitmq.password property from vault"
exit 1;
rabbitmq-plugins enable rabbitmq_delayed_message_exchange
rabbitmq-plugins enable rabbitmq_management
rabbitmqctl add_vhost openiam_am
rabbitmqctl add_vhost openiam_idm
rabbitmqctl add_vhost openiam_audit
rabbitmqctl add_vhost openiam_common
rabbitmqctl add_vhost openiam_connector
rabbitmqctl add_vhost openiam_activiti
rabbitmqctl add_vhost openiam_user
rabbitmqctl add_vhost openiam_groovy_manager
rabbitmqctl add_vhost openiam_synchronization
rabbitmqctl add_vhost openiam_ext_log
rabbitmqctl add_vhost openiam_bulk_synchronization
rabbitmqctl add_vhost openiam_reconciliation
rabbitmqctl add_vhost openiam_bulk_reconciliation
rabbitmqctl add_vhost openiam_business_rule
rabbitmqctl add_user openiam $RABBITMQ_PASSWORD
rabbitmqctl set_user_tags openiam administrator
rabbitmqctl set_permissions -p openiam_am openiam ".*" ".*" ".*"
rabbitmqctl set_permissions -p openiam_idm openiam ".*" ".*" ".*"
rabbitmqctl set_permissions -p openiam_audit openiam ".*" ".*" ".*"
rabbitmqctl set_permissions -p openiam_common openiam ".*" ".*" ".*"
rabbitmqctl set_permissions -p openiam_connector openiam ".*" ".*" ".*"
rabbitmqctl set_permissions -p openiam_activiti openiam ".*" ".*" ".*"
rabbitmqctl set_permissions -p openiam_user openiam ".*" ".*" ".*"
rabbitmqctl set_permissions -p openiam_groovy_manager openiam ".*" ".*" ".*"
rabbitmqctl set_permissions -p openiam_synchronization openiam ".*" ".*" ".*"
rabbitmqctl set_permissions -p openiam_ext_log openiam ".*" ".*" ".*"
rabbitmqctl set_permissions -p openiam_bulk_synchronization openiam ".*" ".*" ".*"
rabbitmqctl set_permissions -p openiam_reconciliation openiam ".*" ".*" ".*"
rabbitmqctl set_permissions -p openiam_bulk_reconciliation openiam ".*" ".*" ".*"
rabbitmqctl set_permissions -p openiam_business_rule openiam ".*" ".*" ".*"
  1. Run commands to apply high availability policies by running following script.
rabbitmqctl set_policy -p openiam_am ha-all ".*" '{"ha-mode":"all"}'
rabbitmqctl set_policy -p openiam_idm ha-all ".*" '{"ha-mode":"all"}'
rabbitmqctl set_policy -p openiam_audit ha-all ".*" '{"ha-mode":"all"}'
rabbitmqctl set_policy -p openiam_common ha-all ".*" '{"ha-mode":"all"}'
rabbitmqctl set_policy -p openiam_connector ha-all ".*" '{"ha-mode":"all"}'
rabbitmqctl set_policy -p openiam_activiti ha-all ".*" '{"ha-mode":"all"}'
rabbitmqctl set_policy -p openiam_user ha-all ".*" '{"ha-mode":"all"}'
rabbitmqctl set_policy -p openiam_groovy_manager ha-all ".*" '{"ha-mode":"all"}'
rabbitmqctl set_policy -p openiam_synchronization ha-all ".*" '{"ha-mode":"all"}'
rabbitmqctl set_policy -p openiam_ext_log ha-all ".*" '{"ha-mode":"all"}'
rabbitmqctl set_policy -p openiam_bulk_synchronization ha-all ".*" '{"ha-mode":"all"}'
rabbitmqctl set_policy -p openiam_reconciliation ha-all ".*" '{"ha-mode":"all"}'
rabbitmqctl set_policy -p openiam_bulk_reconciliation ha-all ".*" '{"ha-mode":"all"}'
rabbitmqctl set_policy -p openiam_business_rule ha-all ".*" '{"ha-mode":"all"}'