> For the complete documentation index, see [llms.txt](https://docs.openiam.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.openiam.com/administration-guide/governance/certification/diagnose-a-stalled-campaign.md). # Diagnose a stalled campaign Work a campaign that has stopped moving from the outside in — service health, queue consumers, preview state, item rows, then Activiti — and tell a stuck queue from a stuck preview from a stuck campai ## What this is, and when you need it A campaign that has stopped moving is more often a service or a queue than a misconfigured campaign, and it is diagnosed with different tools from the ones on the other governance pages. This is the ordered path: prove the service is up, prove its queue is being consumed, then narrow to the preview, the item rows, or the reviewer workflow. Work it in order and each step rules out a whole class of cause. Use it when a certification screen errors, a progress bar has frozen, a launch reported `LAUNCHING` and never `LAUNCHED`, reviewers say they have no task, or a report never appeared. The architecture the diagnosis rests on is [Operate certification services](/administration-guide/governance/certification/operate-certification-services.md). ## At a glance | | | | ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **You will end up with** | The stall attributed to exactly one of: the service, the `CertificationQueue`, the preview store, item generation, or Activiti — with the object to fix named | | **Where** | `${confpath}/health`, the RabbitMQ management UI on vhost `openiam_am`, then `/webconsole/access-certification-edit?id=` and the `certification` database | | **You need first** | Sight of the health directory or the deployment owner on the call; RabbitMQ management access; the `CERTIFICATION_ID` and `CAMPAIGN_ID` you are chasing | | **Takes** | 10 to 20 minutes to attribute the stall. Fixing it is a separate job, and some fixes are deployment work | | **Reversible** | Yes — every step here is a read. The only write suggested is deleting the three ephemeral preview indexes, which the next preview recreates | ## Before you begin Two correlation rules, or you will chase the wrong object. **Join on ids, never on names** — `CERTIFICATION_ID`, `CAMPAIGN_ID`, `PREVIEW_ID`, `ACCESS_REVIEW_ITEM.ID`, `CURRENT_PROC_INST_ID`. And `BATCH_CONFIG.PARAM_1` is the only link from a batch task row back to its certification: the row's `NAME` is a copy of the certification's name taken at save time and drifts after a rename. ## Work it from the outside in {% stepper %} {% step %} ### Confirm the service is reporting Every OpenIAM service writes two files to `${confpath}/health` every `org.openiam.health.check.sweeptime` milliseconds (default **30000**): `certification-manager.simple.health`, containing just the status code, and `certification-manager.full.health`, containing the full health descriptor. Those two files are the check. A `simple.health` file whose mtime is more than 30 seconds old is not reporting at all. One that reads `DOWN` while the process is still alive means the service's internal scheduler has stopped, which is a different and worse problem than a crash, because nothing restarts it. There is no certification health tile in the Webconsole. If you cannot reach `${confpath}` yourself, the observable proxy is the one the user sees: with `certification-manager` down, **every certification screen errors after the reply timeout rather than loading empty**. The timeout is `org.openiam.mq.broker.reply.timeout`, which the shipped deployment sets to `120000` ms. If the service is down, stop here — getting it running is deployment work and belongs to the [Installation Guide](/installation/installation.md). {% endstep %} {% step %} ### Confirm `CertificationQueue` has a consumer In the RabbitMQ management UI, vhost `openiam_am`, open `CertificationQueue`. A consumer count of **0**, or a ready count that climbs and never falls, is the whole explanation for screens that time out. Two neighbouring queues matter and one does not: | Queue | vhost | What its idleness means | | ------------------------------- | ---------------- | --------------------------------------------------------------------------------------------------- | | `CertificationQueue` | `openiam_am` | Every certification request and reply. Zero consumers is a fault | | `CertificationReportChunkQueue` | `openiam_am` | One message per 5,000-item report page. No consumer means reports never assemble | | `RiskScoreChangeQueue` | `openiam_common` | `SCORE_CHANGED` batches from `risk-manager`. No consumer means item risk stays at its launch values | | `AccessReviewQueue` | `openiam_am` | **Nothing consumes it in any release.** Zero consumers and zero messages is expected, not a fault | | {% endstep %} | | | {% step %} ### If a preview is the thing that stopped, read its progress key A running preview writes its progress map and status to `CERT_PREVIEW_PROGRESS:` in Redis, and runs on a fixed pool whose threads are prefixed `cert-preview-`. Three distinguishable states: * **Progress map not advancing while status is `RUNNING`** — the compute is stuck, not the queue. Check the `cert-preview-` threads. * **The preview never started and the error mentions capacity** — the pool (`org.openiam.certification.preview.executor.max.pool.size`, **4**) and its queue (`...executor.queue.capacity`, **50**) are both full, so OpenIAM failed the preview outright with *"Preview capacity is full; too many previews are running. Try again shortly."* rather than queueing it. * **Requesting a preview says another preview is already active** — `CERT_PREVIEW_ACTIVE:` points at a live preview. Re-open the **Preview** tab, which re-attaches to it, and either reuse that preview or **Cancel preview** and request a new one. {% endstep %} {% step %} ### If a launch stopped, count the item rows `CERT_PREVIEW_LAUNCH:` holds the launch-progress pointer, and then: {% code overflow="wrap" %} ```sql SELECT COUNT(*) FROM ACCESS_REVIEW_ITEM WHERE CAMPAIGN_ID = ?; ``` {% endcode %} Run it twice a minute apart. A count that is still climbing means items are being created batch by batch and the launch is working, just slowly — inserts are batched, which is why the number climbs in steps. A count that has flattened while the status is still `LAUNCHING` is a stalled launch. {% hint style="warning" %} **The shipped heap is sized for a demo.** `certification-manager` ships with a 512 MB Java heap, which is not enough for a launch that pages hundreds of thousands of review items — expect the service to slow and then fail during generation. Raising it is a deployment change and belongs to the Installation and Tuning Guide; the symptom to hand your deployment owner is exactly the one above, `LAUNCHING` never becoming `LAUNCHED` with `ACCESS_REVIEW_ITEM` row growth flattened. {% endhint %} {% endstep %} {% step %} ### If items exist but nobody has a task, look at reviewer resolution and Activiti Read `ACCESS_REVIEW_ITEM.CURRENT_PROC_INST_ID`. A populated process instance id with no visible reviewer task points at Activiti in the `workflow` service; check its vhost queues. An **empty** set of items for a user you expected, on the other hand, is not a workflow problem at all — reviewer resolution held them out, and the row that says so is in `CAMPAIGN_PREVIEW_EXCLUSION` with category `REVIEWER`. See [Plan and scope a campaign](/administration-guide/governance/certification/plan-and-scope-a-campaign.md) for every reason and its fix. {% endstep %} {% step %} ### Reconcile the counters before you conclude anything `CAMPAIGN.TOTAL_COUNT`, `COMPLETED_COUNT` and `NOT_STARTED_COUNT` are denormalized columns, not live counts, and the dashboard reads them. Reconcile before you treat a discrepancy as a stall: {% code overflow="wrap" %} ```sql SELECT REVIEW_OUTCOME, COUNT(*) FROM ACCESS_REVIEW_ITEM WHERE CAMPAIGN_ID = ? GROUP BY REVIEW_OUTCOME; ``` {% endcode %} `COMPLETED_COUNT` should equal the number of rows whose `REVIEW_OUTCOME` is not null. A persistent gap is a counter defect worth reporting, not a stalled campaign. {% endstep %} {% endstepper %} ## How a stale preview presents Three distinct failure shapes, and they look similar on screen. **Aged, or past freshness.** Past 24 hours the status sets `ageWarning` and launch is still allowed; past the freshness window launch is refused outright. Both windows and their properties are on [Preview and launch a campaign](/administration-guide/governance/certification/preview-and-launch.md#preview-statuses). Re-run the preview; there is no override short of raising the property. **Snapshot gone, pointer left.** OpenIAM drops the metadata document when a preview reaches a terminal status, and deleting a certification removes every preview row for it plus the progress tracker and the active-preview slot. If the metadata is missing, launch fails with `Cannot launch: no completed preview found for id ` — even though the Preview tab may still show cached numbers in the browser. A hard refresh of the tab will show you which of the two you have. {% hint style="warning" %} There is a fourth shape that is not staleness at all and is easy to misdiagnose: **the tiles show non-zero counts and the Excluded tab is empty.** The tiles come from the metadata document, which is read by id and therefore realtime, while the tabs are searches over `certification_preview_item`, whose documents only become visible after the index refreshes. OpenIAM closes that gap by refreshing the item index once, when it saves `COMPLETED` metadata, so the mismatch should be momentary. If it persists, the cause is an index that was created **dynamically** rather than with OpenIAM's mapping: a dynamic mapping types every string as analysed text, while every preview query is an exact-term query on a keyword field, so the searches return zero rows — and delete-by-query deletes nothing — even though the documents are there. Delete the three `certification_preview_*` indexes and let the next preview recreate them with the correct mapping. {% endhint %} ## Verification | Claim | Evidence | | ------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | | The service is up and scheduling | `${confpath}/health/certification-manager.simple.health` reads `UP` and its mtime is within 30 s | | The queue is being consumed | `CertificationQueue` on vhost `openiam_am` has at least one consumer and no growing ready count | | Its own database is migrated | `SELECT ID, VERSION FROM certification.OPENIAM_VERSION` includes the running release | | A preview snapshot exists | Documents in `certification_preview_item` with that `previewId` and `docType` `INCLUDED` or `EXCLUDED` | | A launch is progressing rather than stalled | `SELECT COUNT(*) FROM ACCESS_REVIEW_ITEM WHERE CAMPAIGN_ID = ?` is still climbing between two reads | | The exclusion slice was captured at launch | `SELECT COUNT(*) FROM CAMPAIGN_PREVIEW_EXCLUSION WHERE CAMPAIGN_ID = ?` matches `CAMPAIGN.EXCLUDED_USER_COUNT + EXCLUDED_ACCESS_COUNT` | | Counters are honest | `COMPLETED_COUNT` equals `SELECT COUNT(*) FROM ACCESS_REVIEW_ITEM WHERE CAMPAIGN_ID = ? AND REVIEW_OUTCOME IS NOT NULL` | | Risk write-back is live | `ACCESS_REVIEW_ITEM.RISK_LAST_COMPUTED` moving on open items while `RISK_SCORE_AT_LAUNCH` stays fixed | | Review finished, whatever the status says | `CAMPAIGN.ACTUAL_COMPLETED_DATE` is set and `COMPLETED_COUNT = TOTAL_COUNT` | ## When it goes wrong Configuration and deployment causes — a missing datasource key, an unmigrated schema, a stale helper script, a node-local upload directory — are on [Operate certification services](/administration-guide/governance/certification/operate-certification-services.md#when-it-goes-wrong). These are the ones that present as a stall. | Symptom | Cause | Fix | | ----------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Every certification screen errors | `certification-manager` is not consuming `CertificationQueue`, so each request waits out `org.openiam.mq.broker.reply.timeout` (shipped `120000` ms) then errors | Have the service started; confirm at least one consumer on `CertificationQueue`, vhost `openiam_am` | | Preview fails immediately with a capacity message | The preview pool rejected the request — pool 4, queue 50, both full | Retry, or raise `org.openiam.certification.preview.executor.max.pool.size` and `...queue.capacity` | | `Another preview is already active` | `CERT_PREVIEW_ACTIVE:` holds a live preview id | Re-open the **Preview** tab to re-attach to it, then reuse it or **Cancel preview** | | Launch refused, *"older than the 24h freshness window"* | The preview's `completedAt` is older than its `validityHours`, or than `org.openiam.certification.preview.freshness.hours` | Re-run the preview | | Launch refused, *"no completed preview found for id"* | The preview's metadata document is gone — the preview reached a terminal status, or the certification was deleted | Start a new preview | | Preview tiles show counts, tabs show no rows | The `certification_preview_*` index was created with a dynamic mapping, so exact-term queries on analysed text match nothing | Delete the three `certification_preview_*` indexes; the next preview recreates them with the correct mapping | | `LAUNCHING` never becomes `LAUNCHED` and row growth has flattened | The shipped 512 MB heap is too small for the item volume | A deployment change: [Installation Guide](/installation/installation.md) | | Reviewers have no task although items exist | A populated `CURRENT_PROC_INST_ID` with no visible task points at Activiti in the `workflow` service | Check the Activiti vhost queues | | A reviewer submitted but none of their decisions were saved | The reviewer does not hold the review task for at least one item in the submission, so the whole submission was rejected. The audit log shows a failed `MAKE_ACCESS_REVIEW_ITEM_DECISION` with *"is not authorized to decide access-review item"* | Compare the item's reviewers in `ACCESS_REVIEW_ITEM_CUR_REV` with the task's assignee and candidate users in Activiti, correct the assignment with delegation, then have the reviewer submit again | | A user you expected has no items at all | Reviewer resolution held them out; the row is in `CAMPAIGN_PREVIEW_EXCLUSION` with category `REVIEWER` | Fix the relationship the reviewer type resolves against, then re-preview | | Counters disagree with the item table | The dashboard reads the denormalized `CAMPAIGN.*_COUNT` columns | Reconcile against `ACCESS_REVIEW_ITEM`; a persistent gap is a counter defect worth reporting | | Risk on open items frozen at launch values | `RiskScoreChangeQueue` on `openiam_common` has no consumer; the publisher gates on one being present, so there is no backlog to see | Check `risk-manager` is deployed; the `0 30 3 * * *` reconciliation sweep is the backstop | | A report never appears | `CertificationReportChunkQueue` has no consumer, or `/upload/.parts` is node-local | Check the queue first, then the storage — see [Campaign history and evidence](/administration-guide/governance/certification/campaign-history-and-evidence.md) | | An idle `AccessReviewQueue` looks like a broken consumer | Nothing consumes that queue in any release | Ignore it; monitor `CertificationQueue` | | A campaign sits in `AWAITING_SIGN_OFF` forever | Nothing in this build invokes sign-off, so review activity alone never sets `COMPLETED` | Not a stall — use `ACTUAL_COMPLETED_DATE` and the complete report | ## Related * [Operate certification services](/administration-guide/governance/certification/operate-certification-services.md) — the architecture this diagnosis rests on, and the retention position * [Certification database tables](/administration-guide/reference/certification-tables.md) — every table and preview field the queries above touch * [Plan and scope a campaign](/administration-guide/governance/certification/plan-and-scope-a-campaign.md) — the preview statuses and every exclusion reason * [Operate a live campaign](/administration-guide/governance/certification/operate-a-live-campaign.md) — the sign-off gate from the campaign side * [Campaign history and evidence](/administration-guide/governance/certification/campaign-history-and-evidence.md) — the chunked report build and `/upload` *Applies to OpenIAM 2026.9.1. Last reviewed 27 September 2026.* --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://docs.openiam.com/administration-guide/governance/certification/diagnose-a-stalled-campaign.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.