> For the complete documentation index, see [llms.txt](https://docs.openiam.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.openiam.com/administration-guide/reports/job-monitor.md).
# Job Monitor
The Job Monitor — what is running and queued across all users, what failed in the last day, how the queue orders runs, and how a super administrator cancels, retries and spots a lost worker.
## What this is, and when you need it
The Job Monitor is the report queue seen as a whole: every run of every user, on every report service instance, with the workers that serve them. It is for super administrators; anyone else who opens Webconsole → **Audit & Reports** → **Reports** → **Job Monitor** sees *Administrators only — Only administrators can open the Job Monitor.* Read this page when a run has been **Queued** for too long, when a report owner reports a failure you need to look into, or when you want to stop a run that is eating the database.
For one report's own runs, its **Run history** tab is the better view: [Run a report and download the file](/administration-guide/reports/run-and-download.md#history). For the properties behind the numbers here, see [Operate the report service](/administration-guide/reports/operate-the-report-service.md#operate-queue).
## At a glance
| What | Value |
| -------------------- | ---------------------------------------------------------------------------- |
| Who may open it | Super administrators (the `SUPER_ADMIN` resource) |
| Refresh | Every 5 seconds while the browser tab is visible |
| Workers per instance | 4 (`report.worker.pool-size`); the tile sums all instances |
| Queue priority | Manual 30, then Scheduled 20, then Export 10; oldest first within a priority |
| Heartbeat | Every 15 seconds; a run with none for 90 seconds is failed as `WORKER_LOST` |
## The tiles
Four tiles under the title, subtitle *Report runs and exports across all users*:
| Tile | What it shows |
| ------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Workers** | *2 of 8 workers busy*: runs in progress over the worker threads of every instance. The total is the pool size of one instance multiplied by the instances the broker sees; with one instance it is 4 |
| **Queued jobs** | Runs waiting for a worker. Click the tile to open the **Queued** tab |
| **Failed in the last 24 hours** | Runs that ended **Failed** since this time yesterday. Click to open **Recent** filtered to failures |
| **Average duration** | Mean duration of completed runs over the window named in the hint (*Last 7 days*) |
Under the tiles a banner states the installation's caps: *Limits: 8 concurrent jobs, 1,000,000 rows per export, 500 MB per file.* and, when a store cap is set, *Storage: 3.1 GB of 20 GB used.* The caps themselves are explained under [Limits](/administration-guide/reports/overview.md#limits).
## The tabs
| Tab | Rows | Columns |
| ----------- | ---------------------------------------------------------- | --------------------------------------------------------------------- |
| **Running** | Runs a worker is executing now | Report, Owner, Trigger, Started, Duration, Rows (written so far) |
| **Queued** | Runs waiting for a worker, in the order they will be taken | Position, Report, Owner, Trigger, Priority, Queued, Jobs ahead |
| **Recent** | Runs that finished, newest first | Report, Owner, Trigger, Started, Duration, Rows, Output, Size, Status |
Each tab reads *No jobs running*, *No jobs in the queue* or *No recent jobs* when empty. A run started through the API without a report shows **Ad hoc query** as its report.
### Filters
Above every table: a report name search, **Owner**, **Trigger** (Manual, Scheduled, Export), **Status** (on Recent), a **Started** or **Queued** date range, and **Reset**. Clicking a tile sets the matching filter for you.
## How the queue orders runs
The queue is the run table of the reporting database, shared by every report service instance. Each instance polls it every five seconds and claims, in one atomic statement, the run with the highest priority and, within that, the oldest queued time, until it has four runs in flight. Runs are therefore ordered:
| Priority | Trigger | Why |
| -------- | --------------------------- | ------------------------------- |
| 30 | Manual (**Run**, **Retry**) | Somebody is waiting for it |
| 20 | Scheduled | Due now, but nobody is watching |
| 10 | Export | A one-off in another format |
The **Position** and **Jobs ahead** columns on the Queued tab are computed from exactly this order, so a new manual run can appear ahead of exports that were queued earlier. Each instance also keeps a bounded queue of fifty runs it has accepted but not started; one more is refused and reported as a failed submission rather than silently dropped.
## The row menu
| Action | Shown when | Does |
| -------------------------------- | ----------------------------------------- | -------------------------------------------------------------------------------------------------------- |
| **View report** | The run belongs to a report | Opens the report's details page |
| **Download** | The run produced a file that still exists | Streams the file |
| **Keep file** / **Stop keeping** | The run has a file | Pins or releases it; see [Keep a file](/administration-guide/reports/run-and-download.md#retention-keep) |
| **View error** | Status **Failed** | Opens the error sheet below |
| **Cancel** | Status **Queued** or **Running** | Stops the run; see below. Greyed out once a cancel is already requested |
| **Delete file** | The run has a file | Removes the file; the run stays in history |
### Cancel a run
**Cancel** asks **Cancel this run?** — *The query is stopped and no file is produced.* — with **Cancel run** and **Keep running**.
* A **Queued** run is cancelled at once; no worker ever takes it.
* A **Running** run gets a cancel flag that its worker reads at the next progress write, every thousand rows, so the status changes within seconds for a fast query and after the current database page for a slow one. The partial file is deleted.
* A run whose worker has stopped sending heartbeats would never read the flag, so it is cancelled at once instead.
The requester of a run may cancel it from their own Run history through the API; the Job Monitor is where a super administrator cancels anyone's. Each cancel is an audit entry naming the report and the person who requested the run.
### The error sheet and Retry
**View error** opens **Run failed** with **Error**, the code, and **Details**, the message the service recorded (*No details were recorded.* when there is none). The codes and what to do about each are in [the error codes](/administration-guide/reports/run-and-download.md#history-error-codes).
**Retry**, on the sheet, queues a new run with the same report, parameters and file format, as a manual run with priority 30, and audits it as a retry naming the person who requested the original. Retry is for super administrators only; report owners run the report again from the details page instead. A retry of a run whose definition no longer validates fails the same way; fix the report first.
## Heartbeats and lost workers
While a run is **Running**, its instance refreshes the run's heartbeat every fifteen seconds on its own thread, independent of row progress, so a slow query still beats. At every poll, every instance looks for running runs whose heartbeat is older than ninety seconds, whichever instance claimed them, and fails them with `WORKER_LOST`: the partial file is deleted, the run's schedule may fire again, and the row appears on **Recent** as **Failed**. Ninety seconds is six heartbeats, so one slow database round trip never fails a live run.
An instance that restarts under an explicitly configured instance name (`report.worker.instance-id`) fails its own leftover runs as `WORKER_RESTARTED` as soon as it is up. By default an instance has no fixed name (`-`), so a restarted instance's runs are caught by the heartbeat sweep instead, up to ninety seconds later.
Both codes mean the same thing to the person waiting: **Retry**. A burst of `WORKER_LOST` across many runs means an instance died or lost its database connection; see [Operate the report service](/administration-guide/reports/operate-the-report-service.md#operate-troubleshooting).
## Verification
* The **Workers** tile shows the number of instances you expect times four. One instance, four workers.
* Start a run from any report: it appears on **Queued** within five seconds, then on **Running**, then on **Recent** as **Completed**.
* **Cancel** on a running run turns it **Cancelled** within one progress interval.
## When it goes wrong
| Symptom | Likely cause |
| ----------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| *Administrators only* | Your account is not entitled to the `SUPER_ADMIN` resource; see [Access and permissions](/administration-guide/reports/access-and-permissions.md#access-super-admin) |
| The **Workers** tile reads *0 of 0* | No report service instance is connected to the broker; the service is down or cannot reach RabbitMQ |
| Runs stay **Queued** while workers are idle | The instances cannot read the reporting database, or they poll but every claim fails; read the service log |
| A run stays **Running** with the same row count for minutes | Its worker is gone; the heartbeat sweep fails it as `WORKER_LOST` within ninety seconds. If the row count is still growing it is a slow query, not a lost worker |
| Many runs fail with `WORKER_LOST` at once | An instance died or lost its database connection; check the instance and the database before retrying |
| **Cancel** does nothing for a long time | The worker reads the flag every thousand rows; a slow source database can take a page's worth of time to get there |
| Submitting a run says it was refused | The instance's bounded queue of fifty accepted runs is full; wait or add an instance |
## Related
* [Run a report and download the file](/administration-guide/reports/run-and-download.md) — one report's Run history, the error codes, retention
* [Operate the report service](/administration-guide/reports/operate-the-report-service.md) — the worker, heartbeat and queue properties, logs, troubleshooting
* [Access and permissions](/administration-guide/reports/access-and-permissions.md#access-super-admin) — what makes a super administrator
* [Report builder overview](/administration-guide/reports/overview.md#limits) — the caps the banner shows
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.
Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:
```
GET https://docs.openiam.com/administration-guide/reports/job-monitor.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
