> For the complete documentation index, see [llms.txt](https://docs.openiam.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.openiam.com/troubleshooting/message-broker/4-audit-queue-poison-message.md).

# ESB restarts in a loop over one audit log message

This section describes the audit log queue's dead-letter queue and how to inspect a message that no consumer can process

| Symptom                                                                                                  | Likely cause                                                                          | Resolution                                                   |
| -------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- | ------------------------------------------------------------ |
| A consumer of the audit log queue restarts continuously, and restarting it or the broker changes nothing | One malformed message the consumer cannot get past, redelivered on every reconnection | Upgrade; on an older release, [purge the queue](#resolution) |

## Symptom

A service that consumes the audit log queue — the ESB, or the audit log manager where that service is deployed separately — restarts continuously. Restarting it, and restarting RabbitMQ, changes nothing: it comes up, processes for a moment, and dies again. The log carries a `StackOverflowError` on the audit log listener thread rather than an ordinary exception:

{% code overflow="wrap" %}

```
Exception in thread "auditLogListener-1" java.lang.StackOverflowError
	at org.openiam.idm.srvc.audit.dto.AuditLogBuilder.build(AuditLogBuilder.java)
	at org.openiam.idm.srvc.audit.dto.AuditLogBuilder.build(AuditLogBuilder.java)
	...
```

{% endcode %}

## Why it happens

One malformed audit log event in the queue cannot be processed. An audit log event carries child events, and an event whose child referenced one of its own ancestors made the code that flattens the tree recurse until the thread's stack was exhausted.

A `StackOverflowError` is an `Error`, not an exception, so it was not caught: it killed the consumer thread while the message was still unacknowledged. RabbitMQ therefore treated the message as undelivered and handed it back on the next connection — the same message, breaking the same thread, on every restart. The service could not get past it and no amount of restarting cleared it.

## Resolution

Upgrade. Three changes remove the loop:

* the flattening walk now visits each event at most once, so a cyclic reference cannot exhaust the stack;
* the message listener now answers an `Error` with a failure response instead of letting it kill the consumer thread, so the message is acknowledged and the caller is told;
* the audit log queue now dead-letters to `AuditLogDeadLetterQueue`, so a message the broker gives up on is kept for inspection rather than redelivered indefinitely or discarded.

On a version that predates these changes, purge the audit log queue to get the service running again.

{% hint style="danger" %}
**Purging discards every audit log message still in the queue, not just the poison one.** Those events are gone — they were never written to the audit store. Do it to get the service running again, and record the window you lost.
{% endhint %}

{% code overflow="wrap" %}

```
rabbitmqctl purge_queue -p openiam_audit AuditLogQueue
```

{% endcode %}

## Inspecting a dead-lettered message

Nothing consumes `AuditLogDeadLetterQueue`. Read it in the RabbitMQ management UI — select the `openiam_audit` vhost, open the queue, and use **Get messages** — or over the API:

{% code overflow="wrap" %}

```
curl -u $RABBITMQ_USERNAME:$RABBITMQ_PASSWORD \
  -H "content-type:application/json" \
  -X POST http://<rabbitmq-host>:15672/api/queues/openiam_audit/AuditLogDeadLetterQueue/get \
  -d '{"count":10,"ackmode":"ack_requeue_true","encoding":"auto"}'
```

{% endcode %}

Messages there expire after seven days, and the queue holds at most ten thousand, so a repeating failure cannot fill the broker's disk. Both limits are set when the queue is first created, from `org.openiam.rabbitmq.audit.dlq.message.ttl.millis` and `org.openiam.rabbitmq.audit.dlq.max.length`.

## Verifying the dead-letter routing

The exchange and the dead-letter queue are created by the services on startup, but the policy that points the audit log queue at them is applied by the installer. Confirm it is present:

{% code overflow="wrap" %}

```
rabbitmqctl list_policies -p openiam_audit
```

{% endcode %}

The `audit-log-dead-letter` policy should be listed. If it is missing — an upgrade that skipped the installer, or a manually built broker — apply it by hand:

{% code overflow="wrap" %}

```
rabbitmqctl set_policy -p openiam_audit --priority 1 --apply-to queues audit-log-dead-letter '^AuditLogQueue$' \
  '{"dead-letter-exchange":"AUDIT_DEAD_LETTER_EXCHANGE","dead-letter-routing-key":"AuditLogDeadLetterQueue","delivery-limit":20}'
```

{% endcode %}

Two things to be aware of when setting it by hand:

* The policy cannot be set as a queue argument instead. RabbitMQ rejects a declaration that gives an existing queue arguments it did not have, so a queue already in service can only be changed this way.
* RabbitMQ applies only the highest-priority policy that matches a queue — it does not merge their definitions. Kubernetes installations carry a `max-message-size-policy` on the `openiam_audit` vhost whose pattern is `.*`, so it matches every queue there. On those the dead-letter policy also has to repeat `max-length-bytes` and `overflow`, or `AuditLogQueue` loses its size cap. The shipped values are `134217728` bytes and `reject-publish`, overridable at install time.

Check what else matches before setting it:

{% code overflow="wrap" %}

```
rabbitmqctl list_policies -p openiam_audit
```

{% endcode %}


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.openiam.com/troubleshooting/message-broker/4-audit-queue-poison-message.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.