> For the complete documentation index, see [llms.txt](https://docs.openiam.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.openiam.com/administration-guide/automation/batch-tasks/diagnose-stalled-batch-tasks.md).

# Restart batch tasks that have all stopped

Every enabled batch task stopped at the same moment with no failed audit record. Find the task whose schedule backlog aborted the cycle, clear it and confirm recovery.

## What this is, and when you need it

Enabled tasks simply stop running. `LAST_EXEC_TIME` stops advancing on all of them at once, and no failure appears anywhere in the audit trail — because no task was ever started, so there is no `BATCH_TASK_EXECUTE` record to fail. Nothing on any screen will tell you.

The cause is almost always one task with an enormous backlog in `BATCH_SCHEDULE`, the queue between the scheduler's two timers. The run-marking for a task is done in one database statement carrying one parameter per due row, and when that statement exceeds the database's parameter limit the exception abandons the rest of the cycle — so **one task with too many due schedule rows stops every batch task on the installation**, on that cycle and every cycle after it.

## At a glance

|                          |                                                                                                                              |
| ------------------------ | ---------------------------------------------------------------------------------------------------------------------------- |
| **You will end up with** | The offending task disabled, its surplus schedule rows gone, and every other enabled task's `LAST_EXEC_TIME` advancing again |
| **Where**                | `/webconsole/batchTaskSearch` — **Administration → Batch Tasks**, and each task's **Scheduled Tasks** grid                   |
| **You need first**       | The batch-task-manager service log, and the list of enabled tasks sorted by `LAST_EXEC_TIME`                                 |
| **Takes**                | 10 minutes to identify; clearing thousands of rows is database work and takes as long as its owner needs                     |
| **Reversible**           | Yes for the disable — but deleted schedule rows are gone, and any run they would have performed simply does not happen       |

## Find the offender and clear its backlog

Order matters: confirm the shape of the failure, then clear the rows before you re-enable anything, or the next cycle hits the same limit.

{% stepper %}
{% step %}

### Confirm every enabled task is frozen at the same moment

Sort the batch task list by `LAST_EXEC_TIME`. One task stalled is a task problem; **all** of them stalled at one moment is the cycle aborting before any task runs, which is this failure and not any individual task's.
{% endstep %}

{% step %}

### Read the three log lines, in order

In the batch-task-manager log, `Exception occurred during batch tasks execution` repeats every `org.openiam.batch.task.execute.time` (shipped `20000`, 20 seconds), with two lines above it naming the parameter cap:

{% code overflow="wrap" %}

```
HHH000443: Dialect [org.hibernate.dialect.SQLServer2016Dialect] limits the number of elements in an IN predicate to 2100 entries. However, the given parameter list [param0] contained 142649 entries, which will likely cause failures to execute the query in the database
The incoming request has too many parameters. The server supports a maximum of 2100 parameters. Reduce the number of parameters and resend the request.
Exception occurred during batch tasks execution
```

{% endcode %}

The entry count in the first line is the size of the backlog you are about to find.
{% endstep %}

{% step %}

### Find the task whose Scheduled Tasks grid is enormous

Open the enabled tasks one at a time; the offender is the one whose grid holds thousands of pending rows, usually sharing a single `NEXT_SCHEDULED_RUN`. This is the tedious part — there is no cross-task view of the schedule table in the product, so if you need the count per task in one answer, that comes from the database and is a question for whoever owns it. See the [Installation Guide](/installation/installation.md).

{% hint style="danger" %}
The numbers are not hypothetical: a single retired task has been seen holding over 140,000 rows for one identical `NEXT_SCHEDULED_RUN`, taking every other task on three ESB nodes down with it. The exposure is worst on Microsoft SQL Server, because its 2100-parameter cap is low and is a server limit rather than a configurable one. Other databases have their own limits on the length of such a list; the failure mode is the same, only the threshold differs.
{% endhint %}
{% endstep %}

{% step %}

### Disable that task

Set **Is Active** off on it. That stops the 5-minute sweep from materialising anything further for it, which is what keeps the count from growing while you work.
{% endstep %}

{% step %}

### Delete its pending schedule rows

Delete them one at a time from the task's **Scheduled Tasks** grid. That is the supported route, and for a handful of rows it is the whole fix.

{% hint style="warning" %}
For thousands of rows the grid is not a realistic instrument, and a bulk clear-out of one task's schedule rows is database work — it has to be scoped to the single task, taken after a backup, and it has to remove the schedule rows before any attempt to remove the task row itself. Hand it to whoever owns the database with the task ID you identified; see the [Installation Guide](/installation/installation.md).
{% endhint %}
{% endstep %}

{% step %}

### Confirm the scheduler recovered

Within one `org.openiam.batch.task.execute.time` interval (20 s) the `Exception occurred during batch tasks execution` line stops appearing. Within one `org.openiam.batch.task.sweep.time` interval (shipped `300000`, 5 minutes) the other tasks' `LAST_EXEC_TIME` values start advancing again. Check both before you touch anything else — the recovery is the evidence that this was the cause.
{% endstep %}

{% step %}

### Decide whether the task should exist

A task that accumulated a backlog was enabled and failing, not disabled and forgotten. Before re-enabling it, establish what it was for and whether a later task now does the same job — running two tasks over one population is its own class of problem, covered under [Cron and overlap](/administration-guide/automation/batch-tasks.md#cron-and-overlap). If the task is superseded, delete it rather than leaving it disabled: a disabled row is one an upgrade or an operator can switch back on.
{% endstep %}
{% endstepper %}

## How the schedule table works

`BATCH_SCHEDULE` is the queue between two independent timers in the batch-task-manager service, each holding a lock so that exactly one node acts:

| Property                              | Shipped default  | What it does                                                                                                         |
| ------------------------------------- | ---------------- | -------------------------------------------------------------------------------------------------------------------- |
| `org.openiam.batch.task.sweep.time`   | `300000` (5 min) | Re-reads **enabled tasks only** and materialises the next `BATCH_SCHEDULE` rows from the cron expression or `RUN_ON` |
| `org.openiam.batch.task.execute.time` | `20000` (20 s)   | Runs every incomplete schedule row whose time has passed                                                             |

Each row carries the task it belongs to, its `NEXT_SCHEDULED_RUN`, and the two flags `IS_RUNNING` and `COMPLETED`. You can see one task's rows in its own **Scheduled Tasks** grid; there is no screen that shows the whole table across tasks.

* The 5-minute sweep materialises the **next ten** occurrences of each enabled task's cron expression on every pass, not just the next one.
* It skips a date only when an **incomplete** row for the same task already exists within 30 seconds of it.
* The 20-second timer collects every row that is not completed, not already running and due, groups the rows by task, marks the group running, runs the task once, and marks the group completed.

So a healthy table holds roughly ten future rows per enabled task, and rows leave it by being completed. Two consequences follow. A **cron change can take up to five minutes to take effect**, and a **disabled task is never swept** — so disabling removes future scheduling but leaves any already-materialised row, which the 20-second timer will still run. Disabling is not an emergency stop.

## Verification

| What to check                   | Where                                                                                                                             |
| ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- |
| The cycle is no longer aborting | `Exception occurred during batch tasks execution` stops appearing in the batch-task-manager log within 20 seconds                 |
| Tasks are running again         | `LAST_EXEC_TIME` advancing on the enabled tasks, within one 5-minute sweep                                                        |
| A specific task ran             | Its `BATCH_TASK_EXECUTE` audit record, whose description is the task name and whose requestor is `org.openiam.idm.system.user.id` |
| The backlog is gone             | The offending task's **Scheduled Tasks** grid holds no pending rows                                                               |

## When it goes wrong

| Symptom                                                                                   | Cause to check                                                                                                                                       | Fix                                                                                       |
| ----------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- |
| Log shows `Can't schedule task:`                                                          | The sweep failed for that one task while materialising its next runs                                                                                 | Check the cron expression and the `RUN_ON` date on that task                              |
| Log shows `Trigger returned null next execution time for task`                            | The cron expression has no future fire time                                                                                                          | Correct the expression; record the first three expected fire times                        |
| An enabled task has no `BATCH_SCHEDULE` row at all                                        | The sweep never materialised one for it                                                                                                              | Check that task's cron and run-on, and the two log lines above                            |
| Rows stuck at `IS_RUNNING='Y'`, `COMPLETED='N'`, older than the task's worst-case runtime | A run died between being marked running and being marked completed; they will never execute again and they still suppress new rows for the same time | Delete them from the task's **Scheduled Tasks** grid                                      |
| Disabled the task, it ran anyway                                                          | A `BATCH_SCHEDULE` row was already materialised; the 20-second timer runs it regardless of the Enabled flag                                          | Delete the pending schedule rows from the task's grid                                     |
| Only one task is stalled, not all of them                                                 | This is not the parameter-cap failure                                                                                                                | [Create a batch task](/administration-guide/automation/batch-tasks.md#when-it-goes-wrong) |

## Related

* [Create a batch task](/administration-guide/automation/batch-tasks.md) — the task row, the two execution routes and the validation errors
* [Shipped batch tasks](/administration-guide/reference/shipped-batch-tasks.md) — which tasks are enabled out of the box, and which retired IDs may still be sitting in your table
* [Diagnose a failed operation](/administration-guide/operations/diagnose-a-failed-operation.md) — reading the batch-task-manager log alongside the audit trail
* [Installation Guide](/installation/installation.md) — bulk database cleanup of schedule rows, and the service's time zone

*Applies to OpenIAM 2026.8.1. Last reviewed 7 August 2026.*


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.openiam.com/administration-guide/automation/batch-tasks/diagnose-stalled-batch-tasks.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.