RabbitMQ cluster went out of order
Issue overview
Problem with RabbitMQ can occur due to various reasons, most common are network unexpected changes, change of hostname of one (or more) of the node, running out of diskspace at one (ore more) node. In these cases you might have issue with connecting to RabbitMQ UI manager or one of OpenIAM service ends up with error and in log of service you can observe complains about not able to create/connect with queue.
Important note: following steps provided below will lead to losing current messages in RabbitMQ.
Solution
Solution is to re-initialize RabbitMQ cluster.
- stop RabbitMQ service on all nodes
cluster systemctl stop rabbitmq-server
- remove directory /var/lib/rabbitmq/mnesia/ from all nodes
- start RabbitMQ service on all nodes, and join nodes in cluster from first node
cluster systemctl start rabbitmq-server
ssh node2 rabbitmqctl stop_appssh node2 rabbitmqctl join_cluster rabbit@node1ssh node2 rabbitmqctl start_appssh node3 rabbitmqctl stop_appssh node3 rabbitmqctl join_cluster rabbit@node1ssh node3 rabbitmqctl start_app
- Create hosts, user and assign permissions to it by running following script.
#!/bin/bash##set -e#### THIS IS RABBITMQ INITIALIZATION SCRIPT. /usr/local/openiam/env.confexport VAULT_CERTS="$HOME_DIR/vault/certs/"export JAVA_HOME="$HOME_DIR/jdk"export VAULT_HOME="$HOME_DIR/utils/vault/". ${VAULT_HOME}validate.vault.shexport RABBITMQ_PASSWORD=$(. ${VAULT_HOME}vault.fetch.property.sh vault.secret.rabbitmq.password)if [ -z "$RABBITMQ_PASSWORD" ] || [ "$RABBITMQ_PASSWORD" == "null" ]; thenecho "cannot get vault.secret.rabbitmq.password property from vault"exit 1;firabbitmq-plugins enable rabbitmq_delayed_message_exchangerabbitmq-plugins enable rabbitmq_managementrabbitmqctl add_vhost openiam_amrabbitmqctl add_vhost openiam_idmrabbitmqctl add_vhost openiam_auditrabbitmqctl add_vhost openiam_commonrabbitmqctl add_vhost openiam_connectorrabbitmqctl add_vhost openiam_activitirabbitmqctl add_vhost openiam_userrabbitmqctl add_vhost openiam_groovy_managerrabbitmqctl add_vhost openiam_synchronizationrabbitmqctl add_vhost openiam_ext_lograbbitmqctl add_vhost openiam_bulk_synchronizationrabbitmqctl add_vhost openiam_reconciliationrabbitmqctl add_vhost openiam_bulk_reconciliationrabbitmqctl add_vhost openiam_business_rulerabbitmqctl add_user openiam $RABBITMQ_PASSWORDrabbitmqctl set_user_tags openiam administratorrabbitmqctl set_permissions -p openiam_am openiam ".*" ".*" ".*"rabbitmqctl set_permissions -p openiam_idm openiam ".*" ".*" ".*"rabbitmqctl set_permissions -p openiam_audit openiam ".*" ".*" ".*"rabbitmqctl set_permissions -p openiam_common openiam ".*" ".*" ".*"rabbitmqctl set_permissions -p openiam_connector openiam ".*" ".*" ".*"rabbitmqctl set_permissions -p openiam_activiti openiam ".*" ".*" ".*"rabbitmqctl set_permissions -p openiam_user openiam ".*" ".*" ".*"rabbitmqctl set_permissions -p openiam_groovy_manager openiam ".*" ".*" ".*"rabbitmqctl set_permissions -p openiam_synchronization openiam ".*" ".*" ".*"rabbitmqctl set_permissions -p openiam_ext_log openiam ".*" ".*" ".*"rabbitmqctl set_permissions -p openiam_bulk_synchronization openiam ".*" ".*" ".*"rabbitmqctl set_permissions -p openiam_reconciliation openiam ".*" ".*" ".*"rabbitmqctl set_permissions -p openiam_bulk_reconciliation openiam ".*" ".*" ".*"rabbitmqctl set_permissions -p openiam_business_rule openiam ".*" ".*" ".*"
- Run commands to apply high availability policies by running following script.
#!/bin/bashrabbitmqctl set_policy -p openiam_am ha-all ".*" '{"ha-mode":"all"}'rabbitmqctl set_policy -p openiam_idm ha-all ".*" '{"ha-mode":"all"}'rabbitmqctl set_policy -p openiam_audit ha-all ".*" '{"ha-mode":"all"}'rabbitmqctl set_policy -p openiam_common ha-all ".*" '{"ha-mode":"all"}'rabbitmqctl set_policy -p openiam_connector ha-all ".*" '{"ha-mode":"all"}'rabbitmqctl set_policy -p openiam_activiti ha-all ".*" '{"ha-mode":"all"}'rabbitmqctl set_policy -p openiam_user ha-all ".*" '{"ha-mode":"all"}'rabbitmqctl set_policy -p openiam_groovy_manager ha-all ".*" '{"ha-mode":"all"}'rabbitmqctl set_policy -p openiam_synchronization ha-all ".*" '{"ha-mode":"all"}'rabbitmqctl set_policy -p openiam_ext_log ha-all ".*" '{"ha-mode":"all"}'rabbitmqctl set_policy -p openiam_bulk_synchronization ha-all ".*" '{"ha-mode":"all"}'rabbitmqctl set_policy -p openiam_reconciliation ha-all ".*" '{"ha-mode":"all"}'rabbitmqctl set_policy -p openiam_bulk_reconciliation ha-all ".*" '{"ha-mode":"all"}'rabbitmqctl set_policy -p openiam_business_rule ha-all ".*" '{"ha-mode":"all"}'