> For the complete documentation index, see [llms.txt](https://docs.openiam.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.openiam.com/solution-architecture/deployment/where-load-lands.md). # Where load lands Which components carry each workload, which work partitions across replicas and which is cluster-serialised by design, what the shipped scheduled work actually is, and what saturation looks like when ## What this is, and when you need it [Sizing](/solution-architecture/deployment/sizing.md) answers how big. This page answers where the load lands: which component carries each workload, which ones benefit from another replica and which do not, and what saturation looks like when it arrives. You need it when a design has to justify headroom rather than total capacity, and when someone proposes to fix a slow overnight run by adding nodes. It has its own page because of one fact the topology catalogue does not show: **the heaviest full-population work in OpenIAM is cluster-serialised by design, and no number of replicas makes it faster.** ## Two kinds of work, and only one of them scales out **Serialised sweeps.** Segregation-of-duties detection across the population, the birthright recalculation sweep, the entitlement graph build and rebuild, and batch task execution each take a cluster-wide lock in the cache. Every replica runs the scheduled method; the one that wins the lock does the work and the rest stand down, and within that node the sweep is a single-threaded loop over pages of identities. Adding replicas buys availability of the sweep and nothing else. **Queued work.** Every call across a service boundary travels over the broker, and every work queue is a durable shared queue on a direct exchange, so every replica of the owning service is a competing consumer on it. Reconciliation parts, synchronization parts, provisioning operations, workflow initiation and reviewer decisions all sit here, and they genuinely fan out. Cache invalidation is the exception that is neither: it is **broadcast**, with a private queue per instance, so it grows with replica count rather than dividing across it. So *will another node help?* resolves to one question back — queued, or sweep — and that is knowable in advance. ## The decision Not how much headroom. Where to put it. | Where headroom goes | What it buys | What it does not buy | | --------------------------------------------------- | ------------------------------------------------------ | ---------------------------------------------------------------------- | | More replicas | Availability; throughput on queued work | Speed on a sweep; it also raises the database's connection requirement | | More memory on one service | Processing a larger source or target at all | Anything on services that never materialise a large set | | A separated console tier | Sign-in that does not compete with administrative work | Sweep or connector throughput | | Connector capacity, or connectors nearer the target | Reconciliation and bulk provisioning completion time | Anything upstream of the connector | | Narrower sweep scope, or a different schedule | The only real lever on serialised work | Nothing else — these sweeps have no batch-size setting | | A larger database tier | Headroom for database-bound work | Relief for memory- or round-trip-bound work | ## Which components carry which workload | Workload | Carried by | Partitions? | First constraint | | ------------------------------------------------------ | ------------------------------------------------- | ------------------------------------ | ----------------------------------------------------------------------------------------------------------- | | Large reconciliation against a managed system | Reconciliation service, connector, core service | Yes — parts fan out | The connector and the target, which serve the whole account set in one exchange; then reconciliation memory | | Large synchronization from a source | Synchronization service, core service, connectors | Parts yes, **the pull no** | Memory on the coordinating replica, then the core service | | Full-population birthright recalculation | Business rule service, then bulk provisioning | **No** — serialised, single-threaded | The serial loop; hardware does not shorten it | | Certification campaign generation | Certification service, then workflow service | Generation serial, **items fan out** | The workflow service's database connections | | Reviewer browsing and sign-off | Certification service, workflow service | Yes | Decision-path consumers; each decision holds one across blocking round trips | | Segregation-of-duties evaluation across the population | Core service, end to end | **No** — serialised, one transaction | Per-identity cost multiplied by the population | | Interactive sign-in at peak | Core service, authorization service, graph, cache | Yes | The authorization domain's shared consumer pool, then entitlement evaluation, then the graph | | Bulk provisioning after a cut-over | Provisioning service, core service, connectors | Yes, **throttled by design** | The identity save and the database, then the connector | | Report generation | Core service, report viewer | No — one synchronous exchange | Reply timeout, message size, core service memory | | Audit search | Audit service, search tier | Yes | The search tier; there is no relational fallback | ### Memory tracks the largest system, not the batch size **Synchronization and reconciliation both materialise the whole result set before they partition it.** A synchronization reads the entire source into memory on one replica before splitting it into parts; a reconciliation hands the entire target account set to the matching step in one list. Partition size controls message size and per-part working set; it does not bound peak memory on the service that did the pull. That is the concrete justification for [Sizing](/solution-architecture/deployment/sizing.md)'s claim that bulk cost tracks the source or target rather than your population, and it is why adding reconciliation replicas for a very large directory can still fail at the first step. Campaign generation is milder but the same shape. ### The platform throttles its own bulk work **The shipped configuration limits concurrency on bulk provisioning and connector dispatch, and applies a dedicated reduction factor to the bulk synchronization and reconciliation consumer pools, explicitly to protect the core service from being overloaded by its own bulk work.** Bulk traffic also runs on separate broker virtual hosts from interactive traffic, so the isolation is structural rather than tuned. That is a shipped acknowledgement of where the ceiling is: the core service and the database behind it carry every bulk workload eventually. It also means more bulk concurrency is not a free win — the configuration records a directional observation from OpenIAM's own testing that *fewer* concurrent provisioning operations completed the work faster overall. ### Segregation of duties is the most constrained workload The population-wide sweep runs in the core service, holds a cluster-wide lock, runs single-threaded, and holds one database connection and one open transaction throughout. Each identity costs several database reads plus a synchronous round trip to the authorization service, in series. **Adding replicas of anything does not help**, and its page size is not configurable: the levers are scope and schedule. The second consequence is usually found late. **The same evaluation runs inline on every provisioning save for an existing identity**, so enforcement is a per-operation tax on the provisioning path rather than only a nightly cost, and it is paid on every bulk run once policies are active. Part 4, *Designing a segregation-of-duties programme* covers what that means. ### Sign-in competes with administrative work, and pushes load onto the graph Authentication is served by the core service on the authorization domain's shared consumer pool — the same pool serving role, group, resource, organization and certification lookups, so **a busy administrative session and a sign-in peak compete for the same consumers.** That is a more specific argument for the separated console tier in [Choosing a topology](/solution-architecture/deployment/choosing-a-topology.md) than sign-in volume alone. Behind it is a second, non-obvious coupling. Every authorization check asks the authorization service for the identity's entitlements; that evaluation runs on a bounded executor with a bounded wait, and **on timeout the service falls back to querying the graph directly.** Graph query volume therefore rises exactly when the entitlement cache is under pressure — in a login storm, when the graph can least absorb it. ## Contention that looks like backlog but is not Saving an identity takes a per-identity lock. When a save arrives for an identity already being saved on another thread — which happens as soon as two bulk paths touch the same population — **the message is not failed. It is re-published to its own queue with an incremented retry count**, up to a bounded maximum, after which the operation is abandoned. The symptom is distinctive: **queue depth rises from re-queued messages rather than from unprocessed work, and throughput collapses without the queue ever draining.** Adding consumers makes it worse. The evidence is in the service logs — repeated save-in-progress messages for the same identities — not in the depth, which looks like ordinary backlog. The response is scheduling, not tuning. ## The shipped scheduled work, in one place Twenty-four task rows ship, and **four of them are enabled**. | Enabled as shipped | What it does | Schedule | | -------------------------- | -------------------------------------------------- | ------------------ | | Disable by last date | The leaver process | Daily, at 17:00 | | Activate by Start Date | Provisions identities whose start date has arrived | Daily, at midnight | | Clean Auth State table | Housekeeping of authentication state | Daily, at midnight | | Risk Manager nightly sweep | Recomputes risk across the population | Nightly, at 02:00 | The risk sweep's row is now the single owner of that cadence — the risk service no longer schedules it internally. Two of the four are overnight sweeps with real cost. The tasks that ship **disabled** include most of those a control description assumes are running. | Disabled as shipped | Inert until enabled | Schedule | | -------------------------------- | ---------------------------------------- | ------------------ | | Business rules recalculation | Population-wide birthright re-evaluation | Daily, at midnight | | Revocation of expiring access | Membership end dates | Daily, at 20:00 | | Escalation of expired requests | Approval escalation | Daily, at midnight | | Reminders for approvers | Reviewer and approver reminders | Daily, at midnight | | Activation reminder | Pre-start notification | Daily, at midnight | | Prune audit logs | Audit retention | Daily, at midnight | | Failed provision requests report | Notice of failed provisioning | Daily, at midnight | The remainder — date-driven deactivation and deletion, provision and deprovision on a date, file housekeeping, task email notification, and legacy notification scripts covering lockout, inactivity and password expiry — also ship disabled. Enabling any adds a recurring population-wide read to the overnight profile, so each is a load decision as well as a control decision. Two consequences follow. **Batch tasks execute serially, inline, one at a time**: the scheduler fires due tasks on a single thread inside the cluster-wide lock, each running to completion before the next begins and holding a database connection and an open transaction throughout. A long task delays every other due task and is a long-running transaction while it does so, so plan the window as a sequence rather than a set of parallel jobs. And **adding scheduler replicas buys availability of the scheduler, not batch throughput.** The shipped set has accumulated across many releases, and not every available sweep has a seeded row — the segregation-of-duties violation digest, for instance, has an implementation but no shipped schedule. **Confirm the enabled set in your own deployment before relying on it**, which is sound practice for any scheduled work carrying a control. ## Where saturation shows up | Bottleneck | Symptom | Where | | ------------------------------------------------- | ----------------------------------------------------------------------- | ------------------------------------------------------------- | | A cross-service call saturated, or a service down | No response within the reply timeout, surfaced to the caller | Console error text; the caller's log | | Connector or target slow | A connector reply-timeout warning | The provisioning service's log | | Backlog on a work queue | Queue depth, per queue and per virtual host | The broker's management interface, proxied inside the console | | Contention on the same identities | Depth rises, throughput falls; save-in-progress then retry exhaustion | Service logs. **Queue depth misleads here** | | A batch task overrunning its window | Schedule rows accumulating marked incomplete | Batch schedule data; the scheduler's log | | Database connection pool exhausted | Connection timeouts, **first in workflow initiation** | Service logs; the health endpoint | | Entitlement evaluation saturated | Fallback-to-graph messages; graph query volume rising | The authorization service's log | | Audit indexing failing | **The event is dropped; one error line is the only signal** | The audit service's log | | Audit backlog building | Memory growth **with no queue-depth signal** — the buffer is in-process | JVM metrics | The last two are worse than they look. A failed audit index write loses the event, with no retry queue and no second destination, so the log line is the control; and audit backlog is invisible to queue monitoring, because the buffer sits inside the service rather than on the broker. ### The instrumentation gap Every service exposes health and metrics endpoints covering the JVM, the connection pools and the HTTP surface. **There is no application-level instrumentation** — no counters or timers for workload progress. Nothing publishes how far a synchronization has got, how many review items a campaign has created, or how many identities a sweep has evaluated. An operator sees infrastructure and JVM health from metrics but must read logs or the audit trail to see whether work is progressing. Design for that: aggregate logs somewhere queryable, and treat the per-run audit records bulk workloads write as the progress signal. ## The tuning surface, named and unvalued | Lever | Applies to | Becomes the lever when | | ------------------------------------------------------------------------------------------------ | ------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `org.openiam.jdbc.connection.pool.size.max` | Every service | Replicas times pool size approach the database's connection cap | | `org.openiam.idm.activiti.jdbc.max.active.connections` | Workflow only | Approval or campaign volume. **Additional to** the shared pool, so the arithmetic in [Sizing](/solution-architecture/deployment/sizing.md) understates workflow | | `org.openiam.full.entitlement.timeout.ms`, executor and graph pool sizing | Authorization service | Login storms; the timeout trades console latency against graph load | | Heap | Synchronization, reconciliation, certification, audit, core | Largest source or target; broadest campaign; audit rate; large reports | | Synchronization partition size | **A setting on each synchronization, not a system property** | Trading message size and per-part memory against parallelism | | Consumer concurrency and prefetch, and `org.openiam.rabbitmq.sync.reducing.factor` for bulk work | Every queue | Throughput changes — but these are **per replica**, and the reduction factor is the explicit dial protecting the core service | | `org.openiam.mq.broker.reply.timeout` | All synchronous calls | Large reports, large searches, slow connectors | | `org.openiam.rabbitmq.message.max.retry.count` | The contention requeue path | How long a contended save bounces before it is abandoned | | `org.openiam.groovy.cache.ttl.ms` | Every service that runs scripts | Bulk work with per-record scripting; expiry costs a fetch per script | | Cache lifetimes and sweep intervals | Authorization, core, provisioning | Freshness against rebuild cost, once caches are large | | Scheduler intervals | Batch task service | How promptly a due task fires — **not** how fast it runs | **What is not tunable.** Several sweeps page through the population at a fixed size no property changes: segregation-of-duties detection, its reporting sweep, business rule recalculation, and the nightly index-consistency sweep. So the answer to *can we tune the segregation-of-duties sweep* is **scope and schedule, not batch size**. The Installation Guide holds the values for everything named above, once the deployment exists. ## What we recommend **Classify every workload as queued or serialised before proposing a node count.** It uses the table above, and it is the difference between headroom that helps and headroom that does not. **Put memory where whole result sets are materialised, and replicas where queues are.** Synchronization, reconciliation, certification and the audit service are memory-driven; the provisioning path, the workflow service and the part consumers benefit from another replica. **Treat the segregation-of-duties sweep as a scheduling problem from the first design conversation**, and do not let a control description imply a frequency the serialised sweep cannot support at this population. **Schedule bulk work so two paths never touch the same population at once**, and make sure whoever operates the platform knows what re-queue contention looks like — it is the failure mode most often misdiagnosed as insufficient capacity. **Decide the batch task set deliberately, verify it in the estate, and plan external observability.** Four tasks ship enabled, most of those a control description assumes are running ship disabled, and with no application-level metrics a platform whose logs are not aggregated cannot answer whether last night's run finished. What changes this is an estate whose heavy work is genuinely all queued — many managed systems, frequent provisioning, a modest population, no wide sweeps. There replicas do buy throughput and the constraint moves outward to the connectors and the targets: see [Connector placement](/solution-architecture/integration/connector-placement.md). ## What OpenIAM does not do here * **OpenIAM publishes no throughput figures.** There is no records-per-second, sign-ins-per-hour or items-per-minute figure in the shipped material. Any such number in a proposal was invented. * **Sweeps do not distribute across the cluster**, and batch tasks do not run in parallel. There is no work sharing and no per-task concurrency setting. * **There is no application-level instrumentation.** Health and JVM metrics exist everywhere; workload progress does not. * **A failed audit index write is not retried and is not surfaced.** The event is lost and a log line is the only evidence. * **Several sweep page sizes are not configurable.** Scope and schedule are the only levers on them. * **Nothing throttles a sweep against interactive load.** Serialised sweeps run when their schedule says so, whatever else is happening. ## Related * [Sizing](/solution-architecture/deployment/sizing.md) — how big each node has to be, and how to read the published reference figures. * [Choosing a topology](/solution-architecture/deployment/choosing-a-topology.md) — the separations this page's contention findings argue for. * *Shipped batch tasks* in the Administration Guide — the full shipped inventory, task by task. *Applies to OpenIAM 2026.8.1. Last reviewed 13 August 2026.* --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://docs.openiam.com/solution-architecture/deployment/where-load-lands.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.