> For the complete documentation index, see [llms.txt](https://docs.openiam.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.openiam.com/administration-guide/lifecycle/synchronize-and-import-data/incremental-synchronization.md). # Schedule an incremental synchronization Move a proven synchronization onto a cron, switch it to INCREMENTAL, and confirm the batch task it creates. ## What this is, and when you need it A full run reads every record in the source every time. An incremental run reads only what changed since the last one, using a source column that records when each record was last updated. Scheduling and incremental are separate settings almost always turned on together, so they are one task here. Do this only after the feed has run manually, twice, without surprises. A schedule turns a configuration you are still testing into one that runs unattended at machine speed. ## At a glance | | | | ------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **You will end up with** | The `SYNCH_CONFIG` row set to `INCREMENTAL` and scheduled, and one `BATCH_CONFIG` task named `Synchronization=` whose **Last Execution time** moves on its own | | **Where** | **Provisioning → Synchronization**, `/webconsole/synchronization?id=`; the task appears under **Administration → Batch Tasks** | | **You need first** | Two clean manual runs, one no-change and one controlled-update, and a source column carrying the last-update time | | **Takes** | 15–30 minutes, plus the wait for the first scheduled fire | | **Reversible** | Yes — unticking **Active**, or clearing both the cron and the run-once date, deletes the batch task; not undone is a record an incremental run passed over while the marker moved past it | ## Before you begin Schedule only after two repeatable manual runs: one no-change, one controlled-update. Record the cron expression, its time zone, the next three expected fire times, the worst-case duration, and who monitors failures. Know which source column carries the last-update time, and what a deletion looks like in that source — an incremental read usually cannot see one. ## Put the feed on a schedule {% stepper %} {% step %} ### Prove the full run twice Open the config at `/webconsole/synchronization?id=` and run it twice with **Sync now**: once against unchanged data, once against a controlled update. Neither run may fail a record — a config that is not clean manually will not become clean on a timer. {% endstep %} {% step %} ### Switch Sync Type to INCREMENTAL Set **Sync Type** to `INCREMENTAL` and fill **Last Update Field** with the source column holding the last-update value. The form requires that field as soon as the type is incremental. Leave the type as `FULL` if the source has no such column — there is no way to make an incremental run work without one. {% endstep %} {% step %} ### Set the frequency **Sync Frequency** offers *Cron Job* or *Execute once on date*, and they are mutually exclusive — setting one clears the other. For the first scheduled run choose *Execute once on date*, a few minutes ahead, and watch it fire. {% endstep %} {% step %} ### Tick Active and save {% hint style="warning" %} From this save the feed runs unattended, at machine speed, with nobody reading the report. A non-zero `FAIL_COUNT` on an incremental run means records were passed over *and* the marker moved past them — silent data loss rather than a stopped feed. {% endhint %} Saving a config with **either** a cron expression or a run-once date, **and** `ACTIVE = 'Y'`, creates or updates a `BATCH_CONFIG` row for you: * `TASK_NAME` = `Synchronization=` * `PARAM1` = the `SYNCH_CONFIG_ID` * `TASK_SCRIPT_ID` = the value of `org.openiam.batch.synchronization.script`, shipped `batch/synchronization.groovy` * `ENABLED` = `true` * Either `CRON_EXPRESSION` or `RUN_ON`, with the other nulled Clearing both the cron and the date, **or** unticking Active, deletes the task. The task is found by its `PARAM1`, so each config owns at most one task — unlike reconciliation, whose task name is keyed on the managed system and can collide. Confirm the task exists after you schedule it: **Administration → Batch Tasks**, looking for `Synchronization=`. {% endstep %} {% step %} ### Watch the first two fires Confirm the run started on its own, then compare the record count against the source. After the second fire, confirm that records changed between the fires were picked up and unchanged records were not. {% endstep %} {% endstepper %} ## Verification * **The task exists.** **Administration → Batch Tasks** lists `Synchronization=`, `ENABLED` true, with the cron or run-once date you set and nothing in the other field. * **The run happened without you.** Reopen the config: **Last Execution time** has moved. The `SYNCHRONIZATION` audit tree carries a parent entry for each fire. * **The failure count is zero.** Part-level audit entries carry `SUCCESS_COUNT` and `FAIL_COUNT`. On an incremental feed this is not a nicety: a non-zero `FAIL_COUNT` means records were passed over *and* the marker moved past them. ## When it goes wrong | Symptom | Cause | Fix | | ----------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- | | Incremental run skips records after a partial failure | `LAST_REC_PROCESSED` is advanced from the newest record seen, not the newest successfully processed | Design the query with an overlapping look-back window and make processing idempotent. | | Run completes but no batch task was created | The cron was blank, the run-once date was in the past, or `ACTIVE` was `N` — the task is deleted in all three cases | Set one of cron or date, tick Active, and confirm `Synchronization=` under **Administration → Batch Tasks**. | ## Rollback Untick **Active**, or clear both the cron and the run-once date. Either deletes the batch task, and the config stays where it is for manual runs. To go back to reading everything, set **Sync Type** to `FULL`; the stored `LAST_REC_PROCESSED` marker is then no longer consulted. ## Design the look-back window The incremental marker advances from the newest record the run *saw*, not the newest record it successfully processed, so a partially failed run skips records permanently. Design for that before you schedule. For `INCREMENTAL`, document the marker column (`QUERY_TIME_FIELD`), the look-back window, the deletion signal, and what happens if the source read succeeds but later processing fails. `LAST_REC_PROCESSED` is set from the most recent record seen across all parts, not the most recent one successfully processed — so **a partially failed run advances the marker past records it never handled**, and the next run will not see them again. A safe design replays an overlapping window without creating duplicates. There is no screen that warns you this happened: compare `FAIL_COUNT` against zero after every incremental run. Overlap and idempotence are both needed: a query that re-reads a window before the marker, and a transformation that changes nothing when it meets a record it has already handled. Overlap alone duplicates work; idempotence alone still loses whatever the marker skipped. ## Related * [Set up a synchronization](/administration-guide/lifecycle/synchronize-and-import-data.md) — the configuration and the run evidence. * [Create a batch task](/administration-guide/automation/batch-tasks.md) — the scheduler this creates a task in. * [Synchronization configuration fields](/administration-guide/reference/synchronization-configuration-fields.md) — **Sync Type**, **Last Update Field**, **Sync Frequency** and the columns behind them. * [Shipped batch tasks](/administration-guide/reference/shipped-batch-tasks.md) — what already runs on this deployment. *Applies to OpenIAM 2026.8.1. Last reviewed 7 August 2026.* --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://docs.openiam.com/administration-guide/lifecycle/synchronize-and-import-data/incremental-synchronization.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.