Issue overview

Problem with RabbitMQ can occur due to various reasons, most common are network unexpected changes, change of hostname of one (or more) of the node, running out of diskspace at one (ore more) node. In these cases you might have issue with connecting to RabbitMQ UI manager or one of OpenIAM service ends up with error and in log of service you can observe complains about not able to create/connect with queue.

Important note: following steps provided below will lead to losing current messages in RabbitMQ.

Solution

Solution is to re-initialize RabbitMQ cluster.

1. stop RabbitMQ service on all nodes

   cluster systemctl stop rabbitmq-server

2. remove directory /var/lib/rabbitmq/mnesia/ from all nodes
3. start RabbitMQ service on all nodes, and join nodes in cluster from first node

   cluster systemctl start rabbitmq-server

ssh node2 rabbitmqctl stop_app
ssh node2 rabbitmqctl join_cluster rabbit@node1
ssh node2 rabbitmqctl start_app
ssh node3 rabbitmqctl stop_app
ssh node3 rabbitmqctl join_cluster rabbit@node1
ssh node3 rabbitmqctl start_app

4. Create hosts, user and assign permissions to it by running following script.

#!/bin/bash

##set -e
#### THIS IS RABBITMQ INITIALIZATION SCRIPT
. /usr/local/openiam/env.conf
export VAULT_CERTS="$HOME_DIR/vault/certs/"
export JAVA_HOME="$HOME_DIR/jdk"
export VAULT_HOME="$HOME_DIR/utils/vault/"


. ${VAULT_HOME}validate.vault.sh


export RABBITMQ_PASSWORD=$(. ${VAULT_HOME}vault.fetch.property.sh vault.secret.rabbitmq.password)


if [ -z "$RABBITMQ_PASSWORD" ] || [ "$RABBITMQ_PASSWORD" == "null" ]; then
    echo "cannot get vault.secret.rabbitmq.password property from vault"
    exit 1;
fi

rabbitmq-plugins enable rabbitmq_delayed_message_exchange
rabbitmq-plugins enable rabbitmq_management

rabbitmqctl add_vhost openiam_am
rabbitmqctl add_vhost openiam_idm
rabbitmqctl add_vhost openiam_audit
rabbitmqctl add_vhost openiam_common
rabbitmqctl add_vhost openiam_connector
rabbitmqctl add_vhost openiam_activiti
rabbitmqctl add_vhost openiam_user
rabbitmqctl add_vhost openiam_groovy_manager
rabbitmqctl add_vhost openiam_synchronization
rabbitmqctl add_vhost openiam_ext_log
rabbitmqctl add_vhost openiam_bulk_synchronization
rabbitmqctl add_vhost openiam_reconciliation
rabbitmqctl add_vhost openiam_bulk_reconciliation
rabbitmqctl add_vhost openiam_business_rule

rabbitmqctl add_user openiam $RABBITMQ_PASSWORD
rabbitmqctl set_user_tags openiam administrator

rabbitmqctl set_permissions -p openiam_am openiam ".*" ".*" ".*"
rabbitmqctl set_permissions -p openiam_idm openiam ".*" ".*" ".*"
rabbitmqctl set_permissions -p openiam_audit openiam ".*" ".*" ".*"
rabbitmqctl set_permissions -p openiam_common openiam ".*" ".*" ".*"
rabbitmqctl set_permissions -p openiam_connector openiam ".*" ".*" ".*"
rabbitmqctl set_permissions -p openiam_activiti openiam ".*" ".*" ".*"
rabbitmqctl set_permissions -p openiam_user openiam ".*" ".*" ".*"
rabbitmqctl set_permissions -p openiam_groovy_manager openiam ".*" ".*" ".*"
rabbitmqctl set_permissions -p openiam_synchronization openiam ".*" ".*" ".*"
rabbitmqctl set_permissions -p openiam_ext_log openiam ".*" ".*" ".*"
rabbitmqctl set_permissions -p openiam_bulk_synchronization openiam ".*" ".*" ".*"
rabbitmqctl set_permissions -p openiam_reconciliation openiam ".*" ".*" ".*"
rabbitmqctl set_permissions -p openiam_bulk_reconciliation openiam ".*" ".*" ".*"
rabbitmqctl set_permissions -p openiam_business_rule openiam ".*" ".*" ".*"

5. Run commands to apply high availability policies by running following script.

#!/bin/bash
rabbitmqctl set_policy -p  openiam_am  ha-all ".*" '{"ha-mode":"all"}'
rabbitmqctl set_policy -p  openiam_idm  ha-all ".*" '{"ha-mode":"all"}'
rabbitmqctl set_policy -p  openiam_audit  ha-all ".*" '{"ha-mode":"all"}'
rabbitmqctl set_policy -p  openiam_common  ha-all ".*" '{"ha-mode":"all"}'
rabbitmqctl set_policy -p  openiam_connector  ha-all ".*" '{"ha-mode":"all"}'
rabbitmqctl set_policy -p  openiam_activiti  ha-all ".*" '{"ha-mode":"all"}'
rabbitmqctl set_policy -p  openiam_user  ha-all ".*" '{"ha-mode":"all"}'
rabbitmqctl set_policy -p  openiam_groovy_manager  ha-all ".*" '{"ha-mode":"all"}'
rabbitmqctl set_policy -p  openiam_synchronization  ha-all ".*" '{"ha-mode":"all"}'
rabbitmqctl set_policy -p  openiam_ext_log  ha-all ".*" '{"ha-mode":"all"}'
rabbitmqctl set_policy -p  openiam_bulk_synchronization  ha-all ".*" '{"ha-mode":"all"}'
rabbitmqctl set_policy -p  openiam_reconciliation  ha-all ".*" '{"ha-mode":"all"}'
rabbitmqctl set_policy -p  openiam_bulk_reconciliation  ha-all ".*" '{"ha-mode":"all"}'
rabbitmqctl set_policy -p  openiam_business_rule  ha-all ".*" '{"ha-mode":"all"}'