> For the complete documentation index, see [llms.txt](https://docs.openiam.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.openiam.com/installation/rpm/rpm-upgrade/major-upgrade.md).

# Major upgrade

Crossing from the 4.2.1 line onto the 2026 line — a supported upgrade with a database migration script, run against a single node or a three-node cluster — with the pre-checks, the go decision and the

## What this is, and when you need it

A move from any **4.2.1.x** release onto the **2026** line. It applies to both **single-node** and **three-node** RPM deployments; where the two differ, the cluster form is called out.

The worked target below is **a generic 2026 line release**. Support confirms the correct target for **your** starting version — it is not always the newest release.

This is a supported upgrade rather than a self-service one — the database migration script is issued to you, so the support case is part of the procedure and <mark style="color:$primary;">**not a formality**</mark>.

{% hint style="warning" %}
**Open a support case before you begin.** OpenIAM supplies the Groovy database initialization script for your database type in the upgrade ticket, and confirms the path from your specific starting version.
{% endhint %}

This page is the whole procedure, in order: who you need, what to check and capture before you start, the go decision, the run itself, the checks afterwards, and what to sign. Work through it top to bottom.

### At a glance

<table><thead><tr><th width="185"></th><th></th></tr></thead><tbody><tr><td><strong>What this is for</strong></td><td>Running a 4.2.1.x → 2026 upgrade as an approved change. This page is written for one host; <a href="#on-a-three-node-cluster">On a three-node cluster</a> adds what a cluster needs on top</td></tr><tr><td><strong>You need first</strong></td><td>An agreed window, an open support case, a rehearsal in a lower environment, and a named DBA on call during the run</td></tr><tr><td><strong>Reversible</strong></td><td>Yes, until you accept the upgrade — the rollback position is the VM snapshot plus the database backup, and both have to be taken before the window</td></tr><tr><td><strong>The outage</strong></td><td>Starts when the script stops the platform and lasts the whole run. All sessions end regardless, because the cache is flushed</td></tr><tr><td><strong>Who is involved</strong></td><td>Five roles, below. The DBA and the change manager both have to be reachable <em>during</em> the window, not just before it</td></tr></tbody></table>

### Choose your path

1. Complete [Before you begin](#before-you-begin), then record the [go decision](#the-go-decision).
2. For one host, continue with [Run the upgrade](#run-the-upgrade).
3. For three hosts, use [On a three-node cluster](#on-a-three-node-cluster) alongside every shared step.
4. Familiarize yourself with the background changes below to know what changed in every 2026 line version and what you gain after the upgrade.

<details>

<summary>Background: platform changes and new capabilities</summary>

## What changes, coming from 4.2.1.x

Name releases may sit between 4.2.1.x and 2026.x.x. Several of changes alter how the deployment is operated, and are worth reading before you plan the window.

### Your deployment goes from two databases to more

4.2.1.x ran on `openiam` and `activiti`. Across this span the platform broke multiple subsystems out into their own services, each owning its own database:

<table><thead><tr><th width="127.99993896484375">Arrived in</th><th width="148.199951171875">Database</th><th>What it holds</th></tr></thead><tbody><tr><td>4.2.2</td><td><code>groovy</code></td><td>Groovy scripts, which stop being files on disk and become rows</td></tr><tr><td>2026.3.1</td><td><code>auditlog</code></td><td>The audit log</td></tr><tr><td>2026.5.2</td><td><code>batchtasks</code></td><td>Batch task configuration and scheduling</td></tr><tr><td>2026.6.1</td><td><code>certification</code></td><td>Certifications, campaigns and review items</td></tr><tr><td>2026.7.1</td><td><code>pbac</code></td><td>Policy-based access control policies and decisions</td></tr><tr><td>2026.8.1</td><td><code>privacy</code></td><td>Consent ledger, purpose registry and privacy events</td></tr><tr><td>2026.10.1</td><td><code>reporting</code></td><td>Report definitions, schedules and run history. On Oracle this one also needs a read grant you make by hand</td></tr></tbody></table>

**Each has to exist&#x20;**<mark style="color:$primary;">**before**</mark>**&#x20;you upgrade, and each service has to be deployed after.** This upgrade crosses all of them — take every set of steps in [Release-specific steps](https://docs.openiam.com/installation/rpm/rpm-upgrade/release-specific-steps.md).

{% hint style="danger" %}
**This changes what a backup has to cover.** A backup of `openiam` and `activiti` was a complete rollback position on 4.2.1.x. Upgrading to 2026.10.1 it captures two databases out of nine — it will still complete, still report success, and leave the other seven on the post-upgrade schema in a state no version expects.
{% endhint %}

### Custom Groovy scripts that extend certification or batch tasks must be updated

Base classes moved package as each subsystem was extracted — twice, in the case of the certification helper. The shipped scripts under `conf/iamscripts/` were updated for you. Anything you have customized was not: find those imports before you upgrade, not after a campaign fails to initialize.

The full table of moves is in [Release-specific steps](https://docs.openiam.com/installation/rpm/rpm-upgrade/release-specific-steps.md#custom-groovy-scripts-must-be-re-pointed).

### Two infrastructure components were replaced

Moving onto the 2026 line replaces two components: **Elasticsearch with OpenSearch**, and **ETCD with Consul**. Both are handled by the upgrade, but they change what you are monitoring and what appears in the port list.

### Custom Groovy scripts that reach outside their sandbox will stop working

Newer OpenIAM versions in 2026 line run Groovy scripts inside a sandboxed engine instead of an unrestricted shell. A script that touches system properties, arbitrary classes, the filesystem or the network directly will fail once this upgrade completes. Review your customized scripts for these before you upgrade — a script that only reads its own parameters and calls approved APIs is unaffected.

### What you gain

<table><thead><tr><th width="165.60003662109375">Area</th><th>What arrives in the 2026 line</th></tr></thead><tbody><tr><td><strong>Segregation of duties</strong></td><td>The whole subsystem — policy management, enforcement during access requests, continuous violation monitoring, mitigating controls and a violations report. Then a <strong>Rule Set Catalog</strong> of pre-built libraries for Manufacturing, HR and Payroll, Financial Services, Plant Maintenance and SAP Basis in US and EU variants, with drift detection, re-sync, rule set comparison, and scheduled detection runs that email a digest</td></tr><tr><td><strong>Access certification</strong></td><td>Rewritten, then rebuilt again. Event-driven campaigns, <strong>campaign preview</strong> — the full scope and reviewer calculation against real data, with every exclusion and its reason, promoted to launch from the snapshot — <strong>extended user search</strong> for building campaign populations without SQL, launch history, bulk approval, side-by-side entitlement comparison against a peer or a previous campaign, entitlement descriptions in the reviewer's view, and a pending-review-only filter. Campaign History, preview and report downloads were substantially reworked for large campaigns</td></tr><tr><td><strong>Identity risk</strong></td><td>Risk-driven campaign scope with a minimum-risk threshold, live risk that updates mid-campaign with drift indicators and a factor breakdown, a <strong>User Risk</strong> page showing the grant path for every access row, and a Risk Score Configuration screen</td></tr><tr><td><strong>Privacy management</strong></td><td>An event-sourced <strong>consent ledger</strong> with a live current-state projection, a registry of processing purposes, activities and lawful authorities with governed versioning, a durable privacy event backbone with reconciliation, evidence records for every state change, and a Privacy Administration Console. <strong>Notice authoring</strong> is added — versioned per language, with a draft-to-publish approval chain — and a self-service <strong>Privacy Center</strong> where users manage their own consent and preferences</td></tr><tr><td><strong>Access control</strong></td><td>A <strong>PBAC console</strong> with shadow, percentage and full rollout modes, a simulation runner that shows what a policy would decide without changing anything, and a decision log recording which policies applied. Delegated access requests can be disabled per application</td></tr><tr><td><strong>Authentication</strong></td><td>Compromised-password detection against Have I Been Pwned, passwordless sign-in with FIDO2 and the OpenIAM Auth App, and risk-based authentication with device intelligence</td></tr><tr><td><strong>Administration scope</strong></td><td>Virtual tenants — organization administrators can be scoped so they see only their own organizations, in both the Webconsole and SelfService</td></tr><tr><td><strong>Operations</strong></td><td>Prometheus monitoring, a descriptive analytics dashboard, a redesigned Groovy Manager with bulk import and export, RequestId enrichment across logs, a redesigned SelfService portal, and CJIS and STIG deployment profiles on RHEL</td></tr><tr><td><strong>Connectors</strong></td><td>Multi-instance Windows installer for .NET v6, PowerShell connectors shareable between production and non-production, and connector-level query-injection protection</td></tr><tr><td><strong>Security</strong></td><td>2026 line versions close a reflected XSS path on password reset, sandboxes Groovy script execution, parameterizes Oracle and PostgreSQL DDL against injection, and restores proper TLS verification in the Tableau, SAP UME, Thales and Salesforce connectors — see the Groovy sandbox note above before you upgrade</td></tr></tbody></table>

</details>

## Who you need

Name a person against each before the window opens. Two of these are on the critical path during the run: the script pauses for the DBA, and only the change manager can call the rollback.

<table><thead><tr><th width="236">Role</th><th>Responsibility</th></tr></thead><tbody><tr><td>Upgrade engineer</td><td>Runs the pre-checks, executes the upgrade, runs the infrastructure post-checks</td></tr><tr><td>DBA</td><td>Backups, executes the script's generated SQL <strong>during</strong> the run, restores on rollback</td></tr><tr><td>Application owner or QA</td><td>Functional post-checks and acceptance</td></tr><tr><td>OpenIAM support</td><td>The open case, and escalation during the window</td></tr><tr><td>Change manager</td><td>The go decision, and the rollback decision</td></tr></tbody></table>

## Before you begin

Take the backups and checks in [Upgrading OpenIAM](https://docs.openiam.com/installation/rpm/rpm-upgrade.md#before-any-upgrade) first, then work through the following. All of it can be completed days ahead of the window — nothing here changes the system.

{% hint style="info" %}
**Record OK / Not OK / N/A against every row, and keep the command output for anything that is not OK.** The output is the evidence; a ticked box on its own proves nothing a week later. On a production deployment this is an approved change and the evidence is the point — for a lab or a rehearsal you can leave the status columns empty, but the steps are the same ones.
{% endhint %}

Load the environment once in the shell you run checks from:

{% code overflow="wrap" %}

```bash
source /usr/local/openiam/env.conf
export VAULT_HOME=/usr/local/openiam/utils/vault/
```

{% endcode %}

### Planning and approvals

<table><thead><tr><th width="68">ID</th><th width="170">Check</th><th width="199">How</th><th width="249">Expected</th><th width="71" align="center">Status</th></tr></thead><tbody><tr><td>PA1</td><td>Change approved, window agreed</td><td>Change reference, window start and end, freeze on administrative changes</td><td>A window long enough for the index migration <em>and</em> the database migrations — the rehearsal tells you how long</td><td align="center">☐</td></tr><tr><td>PA2</td><td>Support case open</td><td>Raise it with OpenIAM support referencing this upgrade</td><td>Case number recorded. This is a prerequisite of the upgrade, not a courtesy</td><td align="center">☐</td></tr><tr><td>PA3</td><td>Rehearsed in a lower environment</td><td>Same starting build, same database type, comparable data volume</td><td>Rehearsal completed; run time and problems recorded</td><td align="center">☐</td></tr><tr><td>PA4</td><td>Target release and its own <code>upgrade.sh</code> confirmed</td><td>Each release's script accepts only its own target</td><td>Target agreed, and the <code>upgrade.sh</code> published <em>for that release</em> downloaded</td><td align="center">☐</td></tr><tr><td>PA5</td><td>Stakeholders told what users will see</td><td>All sessions end; some indices are not carried across; interface customizations are removed</td><td>Application owners and the helpdesk informed</td><td align="center">☐</td></tr><tr><td>PA6</td><td>Custom Groovy scripts reviewed</td><td><code>grep -rl "org.openiam.esb.core.batch.tasks" &#x3C;your-groovy-dir></code></td><td>Scripts naming the old packages found and corrected <strong>in advance</strong> — see <a href="https://docs.openiam.com/installation/rpm/rpm-upgrade/release-specific-steps.md#custom-groovy-scripts-must-be-re-pointed">Release-specific steps</a></td><td align="center">☐</td></tr><tr><td>PA7</td><td>Connector versions reviewed</td><td>List what is deployed and at what version</td><td>A compatible version planned for each</td><td align="center">☐</td></tr><tr><td>PA8</td><td>Rollback criteria and owner agreed</td><td>See <a href="#rollback">Rollback</a></td><td>The owner named and the rollback deadline fixed</td><td align="center">☐</td></tr></tbody></table>

### Platform readiness

The upgrade is supported on **Enterprise Linux 8, 9 and 10**. Only the dependency archive differs between them; the RPM and both binaries are the same file on every platform.

You need roughly **50 GB free**, verified against current utilization of the `/root` partition rather than the disk as a whole.

<table><thead><tr><th width="66">ID</th><th width="164">Check</th><th width="220">How</th><th width="226">Expected</th><th width="77" align="center" valign="middle">Status</th></tr></thead><tbody><tr><td>PB1</td><td>Supported operating system</td><td><code>cat /etc/os-release | grep -E "^(NAME|VERSION_ID|PLATFORM_ID)"</code></td><td>Enterprise Linux only. The upgrade is not supported elsewhere</td><td align="center" valign="middle">☐</td></tr><tr><td>PB2</td><td>Starting version is 4.2.1.x</td><td><code>cat /usr/local/openiam/version</code></td><td><code>4.2.1.x</code></td><td align="center" valign="middle">☐</td></tr><tr><td>PB3</td><td>Administrative access to host and database</td><td><code>sudo -l</code>; and a database account that can create databases and users</td><td>Confirmed for both</td><td align="center" valign="middle">☐</td></tr><tr><td>PB4</td><td>Disk headroom</td><td><code>df -h / /usr/local /var/lib /usr/src /tmp</code></td><td>Judge against current usage, not a fixed figure</td><td align="center" valign="middle">☐</td></tr><tr><td>PB5</td><td><strong>The script's own space checks</strong></td><td>Below</td><td>Free space at least twice the platform directory, and twice the search data</td><td align="center" valign="middle">☐</td></tr><tr><td>PB6</td><td>CPU and memory</td><td><code>nproc; free -h</code></td><td>Meets sizing for the target release</td><td align="center" valign="middle">☐</td></tr><tr><td>PB7</td><td>Clock synchronized</td><td><code>timedatectl | grep -i synchronized</code></td><td><code>yes</code></td><td align="center" valign="middle">☐</td></tr><tr><td>PB8</td><td>Utilities the script uses</td><td>Below</td><td>No output</td><td align="center" valign="middle">☐</td></tr><tr><td>PB9</td><td>Package repositories reachable</td><td><code>dnf repolist</code></td><td>Repositories available — the script installs packages of its own</td><td align="center" valign="middle">☐</td></tr></tbody></table>

{% hint style="danger" %}
**PB5 is the catchy one.** The script checks free space itself and, if it is short, prints a message and exits without changing anything. That is the good case — but in a long window it reads as the upgrade having silently refused to start. Check it yourself first:

{% code overflow="wrap" expandable="true" %}

```bash
OH=$(du -s /usr/local/openiam   | awk '{print $1}')
FREE=$(df   /usr/local/openiam  | awk 'NR==2{print $4}')
echo "platform: need $((2*OH)) have $FREE"

ES=$(du -s /var/lib/elasticsearch | awk '{print $1}')
FV=$(df   /var/lib                | awk 'NR==2{print $4}')
echo "search:   need $((2*ES)) have $FV"
```

{% endcode %}
{% endhint %}

**PB8** is the other check the table sends here — the utilities the script shells out to. Anything this prints is missing, and installing it afterwards means restarting the run:

{% code overflow="wrap" %}

```bash
for c in curl jq m4 rsync unzip rpm2cpio cpio tar; do
  command -v "$c" >/dev/null || echo "missing $c"
done
```

{% endcode %}

### Access

* Admin-privileged credentials for every Linux host taking part.
* Admin-privileged credentials for the external database instance.

### Network, configuration and media

The following must be open between cluster nodes and services:

| Host                  | Port                                                                           | What uses it                                                   |
| --------------------- | ------------------------------------------------------------------------------ | -------------------------------------------------------------- |
| OpenIAM cluster nodes | 8500/TCP, 8503/TCP, 8301/TCP, 8301/UDP, 8302/UDP, 8300/TCP, 8600/TCP, 8600/UDP | Between Consul agents — server and client — and other services |
|                       | 8081                                                                           | IdP                                                            |
|                       | 8082                                                                           | `WEBCONSOLE_PORT`                                              |
|                       | 8083                                                                           | `SELFSERVICE_PORT`                                             |
|                       | 8084                                                                           | `SELFSERVICE_EXT_PORT`                                         |
|                       | 8085                                                                           | `REPORTVIEWER_PORT`                                            |
|                       | 8086                                                                           | `UI_STATIC_PORT`                                               |
|                       | 8087                                                                           | HAProxy                                                        |

The complete list for a three-node cluster is in [Ports](https://docs.openiam.com/installation/rpm/rpm-ha/ports.md).

<table><thead><tr><th width="64">ID</th><th>Check</th><th>How</th><th>Expected</th><th width="70" align="center">Status</th></tr></thead><tbody><tr><td>PF1</td><td>Download site reachable (online run)</td><td><code>curl -sI https://download.openiam.com/ | head -1</code></td><td>A response. Otherwise plan an air-gapped run</td><td align="center">☐</td></tr><tr><td>PF2</td><td>Firewall as the guide requires</td><td><code>firewall-cmd --list-all</code>; <code>ss -tlnp | grep -E ':(80|443)\b'</code></td><td>Clients reach the web tier; the host reaches the database and, online, the download site</td><td align="center">☐</td></tr><tr><td>PF3</td><td>Upgrade variables set</td><td><code>grep -E "UPGRADE_TO_VERSION|INTERNET_AVAILABLE" /usr/local/openiam/env.conf</code></td><td>The target release, and internet availability set truthfully</td><td align="center">☐</td></tr><tr><td>PF4</td><td>Air-gapped artifacts staged</td><td><code>ls -lh /usr/src/</code></td><td>Every file present and non-zero, and the dependency archive matching your EL version</td><td align="center">☐</td></tr><tr><td>PF5</td><td><strong>A session that survives a dropped connection</strong></td><td><code>dnf install -y tmux</code></td><td>Installed. The run is long and an SSH drop during it is not recoverable</td><td align="center">☐</td></tr></tbody></table>

### Health baseline

Capture this **before** the window. Its only purpose is comparison afterwards, and it cannot be taken retrospectively.

<table><thead><tr><th width="64">ID</th><th width="160">Check</th><th width="218">How</th><th width="240">Expected</th><th width="73" align="center">Status</th></tr></thead><tbody><tr><td>PC1</td><td>Services healthy</td><td><code>openiam-cli status</code>; <code>systemctl --failed</code></td><td>Everything up, no failed units</td><td align="center">☐</td></tr><tr><td>PC2</td><td>Search tier healthy, <strong>index list saved</strong></td><td>Below</td><td><code>green</code> or <code>yellow</code>; index names and document counts saved for comparison</td><td align="center">☐</td></tr><tr><td>PC3</td><td>Secret store and its backend healthy</td><td><code>systemctl status openiam-vault etcd --no-pager</code>, and read one property back</td><td>Both active, secrets readable</td><td align="center">☐</td></tr><tr><td>PC4</td><td><strong>Broker queues drained</strong></td><td><code>rabbitmqctl list_queues -p &#x3C;vhost> name messages | awk '$2>0'</code></td><td>Nothing pending. The broker is reinstalled and in-flight messages are lost</td><td align="center">☐</td></tr><tr><td>PC5</td><td>No jobs running</td><td>Console: synchronizations, reconciliations, batch tasks, campaign generation</td><td>Nothing running, and scheduled work paused for the window</td><td align="center">☐</td></tr><tr><td>PC6</td><td>Database baseline</td><td>Row counts of the key tables — <code>USERS</code>, <code>LOGIN</code>, <code>ROLE</code>, <code>GRP</code>, <code>RES</code>, <code>MANAGED_SYS</code> — and <code>SELECT version, success FROM flyway_schema_history WHERE success = 0;</code></td><td>Counts saved; <strong>no failed migration rows before you start</strong></td><td align="center">☐</td></tr><tr><td>PC7</td><td><strong>Trusted certificate count</strong></td><td><code>keytool -list -keystore &#x3C;jdk>/lib/security/cacerts -storepass changeit | grep -c trustedCertEntry</code></td><td>Count recorded. The JDK is replaced during the run, and custom CAs have to survive it</td><td align="center">☐</td></tr><tr><td>PC8</td><td>Integration inventory</td><td>SSO applications, managed systems, MFA providers, mail, reports in use</td><td>Written down, because it drives the functional post-checks</td><td align="center">☐</td></tr></tbody></table>

**PC2** — the index inventory the post-check compares against. Keep the file it writes:

{% code overflow="wrap" expandable="true" %}

```bash
ES_U=$(. ${VAULT_HOME}vault.fetch.property.sh vault.secret.elasticsearch.username)
ES_P=$(. ${VAULT_HOME}vault.fetch.property.sh vault.secret.elasticsearch.password)

curl -s -u "$ES_U:$ES_P" localhost:9200/_cluster/health?pretty
curl -s -u "$ES_U:$ES_P" "localhost:9200/_cat/indices?v&s=index" > /root/pre_indices.txt
```

{% endcode %}

### Backups

**The script backs up files, not databases.** Nothing here is optional.

<table><thead><tr><th width="69">ID</th><th>Artifact</th><th>How</th><th>Expected</th><th width="80" align="center">Status</th></tr></thead><tbody><tr><td>PD1</td><td>VM snapshot</td><td>Hypervisor or cloud console</td><td>A full snapshot of <strong>every</strong> Linux host in the deployment — all three, for a cluster — taken with services stopped if you can</td><td align="center">☐</td></tr><tr><td>PD2</td><td><strong>Database backup</strong></td><td>Native backup of every database the deployment uses, from the external database instance — <code>openiam</code> and <code>activiti</code> <strong>and every other database your release uses</strong></td><td>Completed <strong>and a restore tested</strong>. The script does not do this</td><td align="center">☐</td></tr><tr><td>PD3</td><td>Configuration</td><td><code>tar -czf /root/pre-upgrade-conf-$(date +%F).tgz /usr/local/openiam/env.conf /usr/local/openiam/conf /usr/local/openiam/vault/certs /etc/httpd/conf.d /etc/systemd/system/openiam*</code></td><td>Archive created <strong>and copied off the host</strong></td><td align="center">☐</td></tr><tr><td>PD4</td><td>Secret store data</td><td><code>ETCDCTL_API=3 etcdctl snapshot save /root/etcd-pre-upgrade.db</code></td><td>Saved. Secrets move to a different backend during the upgrade</td><td align="center">☐</td></tr><tr><td>PD5</td><td><strong>Interface customizations</strong></td><td><code>tar -czf /root/ui-custom-$(date +%F).tgz /usr/local/openiam/ui/webapps</code></td><td>Saved — the script deletes this directory</td><td align="center">☐</td></tr><tr><td>PD6</td><td>TLS material</td><td>Web server certificate, chain and key; any custom keystores</td><td>Copies stored securely</td><td align="center">☐</td></tr></tbody></table>

### Database preparation

<table><thead><tr><th width="61">ID</th><th>Check</th><th>How</th><th>Expected</th><th width="78" align="center">Status</th></tr></thead><tbody><tr><td>PE1</td><td><strong>DBA on call for the window</strong></td><td>The script pauses and asks for generated SQL to be run</td><td>A named person, reachable, for the duration</td><td align="center">☐</td></tr><tr><td>PE2</td><td>Permission to create databases and users</td><td>For every new database the target release adds</td><td>Confirmed — on Oracle these are users and schemas</td><td align="center">☐</td></tr><tr><td>PE3</td><td>Credentials chosen for the new databases and the search user</td><td>Decide them now and store them in your password manager</td><td><strong>Do not accept defaults</strong> such as <code>groovy</code>/<code>groovy</code></td><td align="center">☐</td></tr><tr><td>PE4</td><td>External database parameters correct</td><td><code>env.conf</code> and <code>conf/properties/datasource.properties</code></td><td>Right type, host and port</td><td align="center">☐</td></tr><tr><td>PE5</td><td>Host reaches the database</td><td><code>nc -zv &#x3C;db_host> &#x3C;db_port></code></td><td>Connects</td><td align="center">☐</td></tr></tbody></table>

{% hint style="warning" %}
**Confirm the exact set of new databases for your target release before PE2.** The count has changed release by release as subsystems were extracted, and the generated SQL is produced for the release you are going to — not the one this page was written against. [Major upgrade](#your-deployment-goes-from-two-databases-to-more) has the set and when each arrived, and [Release-specific steps](https://docs.openiam.com/installation/rpm/rpm-upgrade/release-specific-steps.md) has the per-release detail. Ask on the support case if the two disagree with what the script prompts for.
{% endhint %}

### Export the audit log if it is large

{% hint style="info" %}
**If your audit log is larger than 3 GB, export it before upgrading.** The upgrade script carries procedures for both online and offline export. Skipping this on a large log lengthens the maintenance window considerably.
{% endhint %}

### Get the script and the artifacts

The script and the artifacts are fetched before the window opens, not during it. Download them from the release download area for the version you are upgrading to, or use the copies your OpenIAM account team supplied. Which of the two paths below you take depends on whether the host can reach the download site; the run itself is identical either way.

#### With internet access

Place the upgrade script — `upgrade.sh` for a **single node** in `/usr/src/`, or `ha_upgrade.sh` for a **three-node cluster**. In an online run the script downloads the release artifacts itself.

#### Without internet access

Stage the script and every artifact under `/usr/src/` first.

{% stepper %}
{% step %}
**Prepare the upgrade script**

Single node — place `upgrade.sh` in `/usr/src/`, then:

{% code overflow="wrap" %}

```bash
chmod +x /usr/src/upgrade.sh
cp /usr/src/upgrade.sh /usr/local/openiam/utils/
```

{% endcode %}

Three-node cluster — copy `ha_upgrade.sh` into the `openiam-cluster-rpm` folder, fill in its **CONFIGURATION** section, then make it executable:

{% code overflow="wrap" %}

```bash
chmod +x ha_upgrade.sh
```

{% endcode %}
{% endstep %}

{% step %}
**Stage the shared artifact**

Place `elasticdump-offlines.tar.gz` in `/usr/src/`.
{% endstep %}

{% step %}
**Stage the release artifacts for your platform**

<table><thead><tr><th width="300">File</th><th>Place it at</th></tr></thead><tbody><tr><td><code>openiam-&#x3C;version>.noarch.x86_64.rpm</code></td><td><code>/usr/src/openiam-&#x3C;version>.noarch.x86_64.rpm</code></td></tr><tr><td><code>openiamrepo.tar.gz</code>, for your Enterprise Linux version — <code>el8</code>, <code>el9</code> or <code>el10</code></td><td><code>/usr/src/openiamrepo.tar.gz</code></td></tr><tr><td><code>frontend.tar.gz</code></td><td><code>/usr/src/frontend.tar.gz</code></td></tr><tr><td><code>backend.tar.gz</code></td><td><code>/usr/src/backend.tar.gz</code></td></tr></tbody></table>

**Match the dependency repository to the host's Enterprise Linux version**, from `grep PLATFORM_ID /etc/os-release`. It is the one artifact that differs between versions.
{% endstep %}
{% endstepper %}

## The go decision

One row, and it belongs to the change manager.

<table><thead><tr><th width="61">ID</th><th width="135">Gate</th><th>Expected</th><th width="81" align="center">Status</th></tr></thead><tbody><tr><td>S1</td><td>Go / no-go</td><td>Every mandatory pre-check OK, the DBA confirmed on call, the rollback deadline fixed. <strong>Decision and time recorded</strong></td><td align="center">☐</td></tr></tbody></table>

A *no-go* is a **successful** outcome of this gate. The window closes, nothing was touched, and the pre-checks you have already completed stay valid for the next attempt.

## Run the upgrade

{% hint style="danger" %}
**Run this in a session that is not shared and not being recorded.** The script asks for credentials with the input hidden, then prints a confirmation summary **in clear text**. Anyone watching the session, and any terminal recording, captures them.
{% endhint %}

{% stepper %}
{% step %}
**Announce the start** and put up whatever maintenance notice your change process expects. The outage begins at the next step, not here.
{% endstep %}

{% step %}
**Open a protected session**, then start the upgrade inside it:

{% code overflow="wrap" %}

```bash
tmux new -s oiam-upgrade
openiam-cli upgrade
```

{% endcode %}

Watch the log from a second pane rather than the same one:

{% code overflow="wrap" %}

```bash
tail -f /usr/local/openiam/upgrade.log
```

{% endcode %}
{% endstep %}

{% step %}
**Answer the credential prompts** with the values you chose at PE3 — the search user first, then a username and password for each new database. Confirm when it asks.
{% endstep %}

{% step %}
**Hand the generated SQL to the DBA** when the script stops and names the file. Copy it to them securely, have it run against the database, and only then press Enter to let the script continue.

This is a **genuine pause**: the script is waiting, and it will wait.
{% endstep %}

{% step %}
**Let it finish.** It prints `Done` and the path of the backup it took. Record the end time and go straight to the post-checks.
{% endstep %}
{% endstepper %}

#### The script waits for a person

Two phases, and both block until answered — **phase 8** for the credentials, **phase 13** for the DBA. Plan for someone to be at the terminal for both. The explanation of phases and what to expect as a result for every phase is in the reference below.

<details>

<summary>Reference: script phases and their effects</summary>

### What the script is doing, phase by phase

Follow this in the log, where each phase announces itself by the function name shown under its label. Its real value is when a run stops: the phase tells you what state the system is in, which decides whether you can continue or have to roll back.

<table><thead><tr><th width="46">#</th><th width="168">Phase</th><th>What happens</th><th>What it means for you</th></tr></thead><tbody><tr><td>1</td><td>Space check<br><code>free_space</code></td><td>Checks free space for the backup and the index migration</td><td>Short on space, it prints and exits. <strong>Nothing has changed</strong></td></tr><tr><td>2</td><td>Stop<br><code>stop_app</code></td><td>Stops the platform, waits for the bus to exit</td><td><strong>The outage starts here</strong></td></tr><tr><td>3</td><td>Backup<br><code>backup</code></td><td>Tars the platform directory</td><td>Files only. <strong>Not the databases</strong></td></tr><tr><td>4</td><td>Clean<br><code>clean_folders</code></td><td>Removes service binaries, the interface web applications, the package repository, cache and broker utilities, and the JDK</td><td><strong>Interface customizations are deleted here.</strong> The old truststore is kept aside</td></tr><tr><td>5</td><td>Download<br><code>download</code></td><td>Fetches and unpacks the target release</td><td>Online runs need the download site</td></tr><tr><td>6</td><td>Replace<br><code>replace</code></td><td>Installs the new JDK, utilities, backend and frontend, and <strong>restores the old truststore</strong></td><td>Your custom CAs come back here</td></tr><tr><td>7</td><td>Cache<br><code>upgrade_redis</code></td><td>Flushes and reinstalls the cache</td><td><strong>Every session is invalidated</strong></td></tr><tr><td>8</td><td>Secret bootstrap<br><code>bootstrap_vault</code></td><td>Collects the new credentials and writes them to the store</td><td><strong>Interactive</strong> — the prompts above</td></tr><tr><td>9</td><td>Search migration<br><code>migrate_elasticsearch_to_opensearch</code></td><td>Dumps the indices, replaces the search tier, loads them back, sets up curation</td><td><strong>The longest phase</strong>, and it grows with index size. Some indices are deliberately not carried across</td></tr><tr><td>10</td><td>Secret backend<br><code>migrate_etcd_to_consul</code></td><td>Exports the secrets, swaps the backend, imports them again</td><td>Writes an export file <strong>containing every secret in clear text</strong></td></tr><tr><td>11</td><td>Graph store<br><code>upgrade_cassandra_janusgraph</code></td><td>Removes the old store and reinstalls the graph engine</td><td>—</td></tr><tr><td>12</td><td>Broker<br><code>upgrade_rabbitmq</code></td><td>Removes and reinstalls the broker</td><td><strong>Queues come back empty</strong> — which is why PC4 drained them</td></tr><tr><td>13</td><td>Migrations<br><code>apply_migrations</code></td><td>Regenerates the datasource configuration, waits for the DBA, runs the schema migrations</td><td><strong>Interactive</strong> — the pause above</td></tr><tr><td>14</td><td>Start<br><code>start_app</code></td><td>Installs the new unit files and <code>env.conf</code>, starts the platform</td><td><strong><code>env.conf</code> is replaced by the package copy</strong></td></tr><tr><td>15</td><td>Web tier<br><code>nginx init / update_module</code></td><td>Initializes the interface router, installs the new web server module, restarts it</td><td>The script prints <code>Done</code></td></tr></tbody></table>

</details>

### If the run stops

{% hint style="danger" %}
**Do not simply run it again.** The script appends to `env.conf` every time it starts, and later phases assume the earlier ones completed. A second run against a half-upgraded host is **not a retry**.
{% endhint %}

**Note the last phase reached**, capture `/usr/local/openiam/upgrade.log`, and take it to OpenIAM support on the case you opened at PA2. If it cannot be resolved inside the agreed window, call the [rollback](#rollback) — that is what the deadline at PA8 is for.

## After the upgrade

Infrastructure first, because a functional check that fails for an infrastructure reason wastes the application owner's time. Record the output, not just the tick: these are the rows a reviewer asks about afterwards.

### Completion and version

<table><thead><tr><th width="62">ID</th><th width="168">Check</th><th width="232">How</th><th>Expected</th><th width="74" align="center">Status</th></tr></thead><tbody><tr><td>QA1</td><td>Version file updated</td><td><code>cat /usr/local/openiam/version</code></td><td>2026.x.x</td><td align="center">☐</td></tr><tr><td>QA2</td><td>Upgrade finished</td><td><code>tail -50 /usr/local/openiam/upgrade.log</code></td><td>Ends with "Done" and the backup path</td><td align="center">☐</td></tr><tr><td>QA3</td><td>No unexplained errors (script runs without set -e)</td><td><code>grep -inE "error|fail|exception|cannot|can't|denied|refused" /usr/local/openiam/upgrade.log</code></td><td>Every hit reviewed and explained</td><td align="center">☐</td></tr><tr><td>QA4</td><td>File backup exists</td><td><code>ls -lh /usr/local/openiam/backup/openiam-*-files.tar</code><br><code>tar -tf /usr/local/openiam/backup/openiam-*-files.tar | head</code></td><td>Archive present and readable</td><td align="center">☐</td></tr><tr><td>QA5</td><td>Disk headroom</td><td><code>df -h /usr/local/openiam /var/lib /tmp /usr/src</code></td><td>Recommend ≥ 20% free per filesystem</td><td align="center">☐</td></tr></tbody></table>

### Landing on 2026.11.2 or later: bring Vault up to the release

From 2026.11.2 the release ships Vault 2.2.1 and Consul 2.1.0. Phase 10 starts the release's Consul on the data it imports, but this script changes the Vault binary only when `VAULT_VERSION_UPGRADE` is set, and the `secret/` mount stays key-value (KV) version 1 either way. Check which Vault the host runs:

{% code overflow="wrap" %}

```bash
vault version
```

{% endcode %}

**On a three-node cluster, skip this step.** The step is written for a single host; see [Three-node clusters](https://docs.openiam.com/installation/rpm/rpm-upgrade/release-specific-steps.md#three-node-clusters-are-not-moved-by-this-release) in Release-specific steps, under 2026.11.2.

**If it is older than 2.2.1**, run the release's Vault and Consul step once, with the OpenIAM services stopped. The upgrade script empties `/usr/src/` when it finishes, so unpack the release's RPM there again first. Without internet access, copy in the RPM you staged and skip the `curl`:

{% code overflow="wrap" %}

```bash
export UPGRADE_TO_VERSION=<the release you upgraded to>
cd /usr/src
curl -fO "https://download.openiam.com/release/enterprise/${UPGRADE_TO_VERSION}/rpm/openiam-${UPGRADE_TO_VERSION}.noarch.x86_64.rpm"
rpm2cpio "openiam-${UPGRADE_TO_VERSION}.noarch.x86_64.rpm" | cpio -idm
tar -xzf tmp/openiam-tmproot/openiam.tar.gz
openiam-cli stop
/usr/src/openiam/utils/vault/upgrade_vault_consul.sh /usr/src/openiam
openiam-cli start
```

{% endcode %}

What the step does, how to check it, and the later switch to KV version 2 are in [Release-specific steps](https://docs.openiam.com/installation/rpm/rpm-upgrade/release-specific-steps.md#vault-and-consul-move-to-new-major-versions), under 2026.11.2. Confirm this follow-on step on your support case before the window, alongside the path from your starting version.

<details>

<summary>Detailed infrastructure verification</summary>

### Configuration files

<table><thead><tr><th width="62">ID</th><th width="168">Check</th><th width="232">How</th><th>Expected</th><th width="74" align="center">Status</th></tr></thead><tbody><tr><td>QB1</td><td>env.conf site values preserved (replaced by package copy)</td><td><code>tar -xOf /usr/local/openiam/backup/openiam-*-files.tar env.conf > /tmp/env.conf.old</code><br><code>diff /tmp/env.conf.old /usr/local/openiam/env.conf</code></td><td>Customer values (VAULT_URL, ports, DB host, JVM memory, hostnames) re-applied where needed</td><td align="center">☐</td></tr><tr><td>QB2</td><td>Flyway / OS CURATOR entries present, no duplicates</td><td><code>grep -cE "FLYWAY_(GROOVY|AUDITLOG|BATCHTASKS|CERTIFICATION|PBAC)_" /usr/local/openiam/env.conf</code><br><code>grep -c "OS CURATOR" /usr/local/openiam/env.conf</code></td><td>Each variable defined once</td><td align="center">☐</td></tr><tr><td>QB3</td><td>Log level suitable for production</td><td><code>grep -E "OPENIAM_LOG_LEVEL|OPENIAM_DEBUG_SLEEP" /usr/local/openiam/env.conf</code></td><td>Script sets trace; change to agreed level and restart</td><td align="center">☐</td></tr><tr><td>QB4</td><td>datasource.properties correct</td><td>Below</td><td>Correct type, host, port and all nine database names</td><td align="center">☐</td></tr><tr><td>QB5</td><td>OpenSearch properties</td><td><code>ls -l /usr/local/openiam/conf/properties/ | grep -E "elastic|opensearch"</code><br><code>grep -n "http" /usr/local/openiam/conf/properties/opensearch.properties</code></td><td>elasticsearch.properties gone; URLs use https; no "httpss"</td><td align="center">☐</td></tr><tr><td>QB6</td><td>Refreshed property files</td><td><code>ls -l /usr/local/openiam/conf/properties/rabbitmq.properties /usr/local/openiam/conf/properties/redisson.yaml</code></td><td>Present with upgrade-date timestamp</td><td align="center">☐</td></tr></tbody></table>

**QB4** — the datasource configuration the script regenerates. Every database the release now uses has to appear in it:

{% code overflow="wrap" %}

```bash
grep -E "dbType|jdbc.host|jdbc.port|database.name|activiti.name|groovy|auditlog|batchtasks|certification|pbac" /usr/local/openiam/conf/properties/datasource.properties
```

{% endcode %}

### Secret store and its backend

<table><thead><tr><th width="62">ID</th><th width="168">Check</th><th width="232">How</th><th>Expected</th><th width="74" align="center">Status</th></tr></thead><tbody><tr><td>QC1</td><td>etcd stopped and disabled</td><td><code>systemctl is-active etcd; systemctl is-enabled etcd</code></td><td>inactive / disabled</td><td align="center">☐</td></tr><tr><td>QC2</td><td>Consul running</td><td><code>systemctl status consul --no-pager</code><br><code>consul members</code></td><td>active, enabled, node alive</td><td align="center">☐</td></tr><tr><td>QC3</td><td>Vault running (single process)</td><td><code>systemctl status openiam-vault --no-pager</code><br><code>pgrep -a vault</code></td><td>active; one vault server process</td><td align="center">☐</td></tr><tr><td>QC4</td><td>Certificate login to Vault</td><td>Below</td><td>Non-empty token, or on 2026.11.2 and later a <code>PASS</code> line for the role <code>services</code></td><td align="center">☐</td></tr><tr><td>QC5</td><td>Old secrets migrated</td><td>Below</td><td>Each returns a value (length > 1)</td><td align="center">☐</td></tr><tr><td>QC6</td><td>New secrets created</td><td>Below</td><td>All return a value</td><td align="center">☐</td></tr><tr><td>QC7</td><td>Helper symlinks</td><td><code>ls -l /usr/bin/consul /usr/bin/medusa</code></td><td>Point into /usr/local/openiam/vault/</td><td align="center">☐</td></tr></tbody></table>

**QC4** — the certificate login the services themselves use. On a host that landed on 2026.11.2 or later and ran the Vault and Consul step, the services sign in with the role `services`, and `web` stays pinned to the previous certificate, kept as `vault-web.crt`. Check the roles instead of the command below. The script is safe to run at any time:

{% code overflow="wrap" %}

```bash
/usr/local/openiam/utils/vault/client-identities.sh
```

{% endcode %}

It prints `PASS: the role services logs in with the policy openiam-services`, and a line for each other role. See [Vault client identities](https://docs.openiam.com/administration-guide/security/vault-client-identities.md).

On an earlier release, a token comes back only if certificate authentication still works:

{% code overflow="wrap" %}

```bash
curl -sk -X POST --cert /usr/local/openiam/vault/certs/vault.crt \
--key /usr/local/openiam/vault/certs/vault.key --data '{"name":"web"}' \
https://${VAULT_URL}:${VAULT_PORT}/v1/auth/cert/login | jq -r .auth.client_token
```

{% endcode %}

**QC5** — the secrets that existed before the upgrade. Each should report a length greater than 1; an empty value means the migration dropped it:

{% code overflow="wrap" %}

```bash
export VAULT_HOME=/usr/local/openiam/utils/vault/
for k in jdbc.username activiti.jdbc.username redis.password; do
echo -n "$k: "; . ${VAULT_HOME}vault.fetch.property.sh vault.secret.$k | wc -c; done
```

{% endcode %}

**QC6** — the secrets the upgrade created: a username and password for each new database, and the search tier's own. This reuses `VAULT_HOME` from QC5:

{% code overflow="wrap" %}

```bash
for s in groovy auditlog batchtasks certification pbac; do for f in username password; do
echo -n "$s.$f: "; . ${VAULT_HOME}vault.fetch.property.sh vault.secret.$s.jdbc.$f | wc -c; done; done
for k in opensearch.username opensearch.password opensearch.admin.username opensearch.admin.password; do
echo -n "$k: "; . ${VAULT_HOME}vault.fetch.property.sh vault.secret.$k | wc -c; done
```

{% endcode %}

### Databases and schema migrations

<table><thead><tr><th width="62">ID</th><th width="168">Check</th><th width="232">How</th><th>Expected</th><th width="74" align="center">Status</th></tr></thead><tbody><tr><td>QD1</td><td>New databases exist</td><td>Below</td><td>groovy, auditlog, batchtasks, certification, pbac present</td><td align="center">☐</td></tr><tr><td>QD2</td><td>New users can connect</td><td><code>e.g. mysql -h &#x3C;db_host> -u groovy -p groovy -e "SELECT 1;" (repeat for each)</code></td><td>All five connect</td><td align="center">☐</td></tr><tr><td>QD3</td><td>No failed Flyway migrations (all nine schemas)</td><td><code>SELECT version, description, success FROM flyway_schema_history WHERE success = 0;</code><br><code>PostgreSQL: WHERE success = false</code></td><td>0 rows everywhere</td><td align="center">☐</td></tr><tr><td>QD4</td><td>Latest migration applied</td><td>Below</td><td>Highest applied version = highest shipped file</td><td align="center">☐</td></tr><tr><td>QD5</td><td>Data integrity</td><td><code>Re-run PC6 row counts</code></td><td>Matches pre-upgrade baseline</td><td align="center">☐</td></tr><tr><td>QD6</td><td>Init SQL removed</td><td><code>ls -l /tmp/init.sql</code></td><td>Not present</td><td align="center">☐</td></tr></tbody></table>

**QD1** — list the databases, using the form for your engine:

{% code overflow="wrap" %}

```sql
-- MariaDB and MySQL
SHOW DATABASES;

-- PostgreSQL
\l

-- MSSQL
SELECT name FROM sys.databases;

-- Oracle, where they are users rather than databases
SELECT username FROM dba_users
 WHERE username IN ('GROOVY','AUDITLOG','BATCHTASKS','CERTIFICATION','PBAC');
```

{% endcode %}

**QD4** — the migration history against the schema files that produced it. The newest applied version should match the newest file on disk:

{% code overflow="wrap" %}

```sql
SELECT version, installed_on FROM flyway_schema_history ORDER BY installed_rank DESC;
```

{% endcode %}

{% code overflow="wrap" %}

```bash
ls /usr/local/openiam/conf/schema/<mysql|postgres|mssql|oracle>/openiam | sort -V | tail -3
```

{% endcode %}

### Search tier and index curation

<table><thead><tr><th width="62">ID</th><th width="168">Check</th><th width="232">How</th><th>Expected</th><th width="74" align="center">Status</th></tr></thead><tbody><tr><td>QE1</td><td>Elasticsearch stopped and disabled</td><td><code>systemctl is-active elasticsearch; systemctl is-enabled elasticsearch</code></td><td>inactive / disabled</td><td align="center">☐</td></tr><tr><td>QE2</td><td>OpenSearch running</td><td><code>systemctl list-units --type=service | grep -i opensearch</code><br><code>ss -tlnp | grep 9200</code></td><td>active; listening on 9200</td><td align="center">☐</td></tr><tr><td>QE3</td><td>Cluster health</td><td>Below</td><td>green (yellow acceptable on single node with replicas)</td><td align="center">☐</td></tr><tr><td>QE4</td><td>Indices migrated</td><td>Below</td><td>Only expected differences: provisionrequest-*, connectorreply-*, provisionconnectorrequest-*, .security-*</td><td align="center">☐</td></tr><tr><td>QE5</td><td>Document counts</td><td><code>curl -sk -u "$OS_USER:$OS_PASS" "https://localhost:9200/_cat/count/&#x3C;index>?v"</code></td><td>Equal to PC2 baseline per index</td><td align="center">☐</td></tr><tr><td>QE6</td><td>TLS enforced</td><td><code>curl -s http://localhost:9200 # should fail</code></td><td>Plain HTTP rejected</td><td align="center">☐</td></tr><tr><td>QE7</td><td>Curator scheduled</td><td><code>systemctl list-timers --all | grep -i curator; crontab -l -u openiam | grep -i curator</code></td><td>Job configured</td><td align="center">☐</td></tr><tr><td>QE8</td><td>Historical audit visible</td><td><code>Webconsole > Audit Log: filter a pre-upgrade date range</code></td><td>Old events returned</td><td align="center">☐</td></tr></tbody></table>

**QE3** — cluster health, read with the credentials from the secret store:

{% code overflow="wrap" %}

```bash
OS_USER=$(. ${VAULT_HOME}vault.fetch.property.sh vault.secret.opensearch.username)
OS_PASS=$(. ${VAULT_HOME}vault.fetch.property.sh vault.secret.opensearch.password)
curl -sk -u "$OS_USER:$OS_PASS" https://localhost:9200/_cluster/health?pretty
```

{% endcode %}

**QE4** — the index inventory, compared against the baseline you captured at PC2:

{% code overflow="wrap" %}

```bash
curl -sk -u "$OS_USER:$OS_PASS" "https://localhost:9200/_cat/indices?v&s=index"
diff <(awk '{print $3}' /root/pre_indices.txt | sort) <(curl -sk -u "$OS_USER:$OS_PASS" "https://localhost:9200/_cat/indices?h=index" | sort)
```

{% endcode %}

### Graph store

<table><thead><tr><th width="62">ID</th><th width="168">Check</th><th width="232">How</th><th>Expected</th><th width="74" align="center">Status</th></tr></thead><tbody><tr><td>QF1</td><td>Cassandra removed</td><td><code>rpm -qa | grep -i cassandra</code></td><td>No packages</td><td align="center">☐</td></tr><tr><td>QF2</td><td>JanusGraph running</td><td><code>systemctl status janusgraph --no-pager</code></td><td>active, enabled</td><td align="center">☐</td></tr><tr><td>QF3</td><td>Ownership</td><td><code>ls -ld /usr/local/openiam/janusgraph /usr/local/openiam/utils/graph</code></td><td>openiam:openiam</td><td align="center">☐</td></tr><tr><td>QF4</td><td>No startup errors</td><td><code>journalctl -u janusgraph --since "-2h" -p err --no-pager</code></td><td>None</td><td align="center">☐</td></tr></tbody></table>

### Broker

<table><thead><tr><th width="62">ID</th><th width="168">Check</th><th width="232">How</th><th>Expected</th><th width="74" align="center">Status</th></tr></thead><tbody><tr><td>QG1</td><td>RabbitMQ running</td><td><code>systemctl status rabbitmq-server --no-pager</code><br><code>rabbitmqctl status | grep -iE "rabbitmq|erlang"</code></td><td>active; new versions shown</td><td align="center">☐</td></tr><tr><td>QG2</td><td>Vhosts, users, permissions</td><td><code>rabbitmqctl list_vhosts; rabbitmqctl list_users</code><br><code>rabbitmqctl list_permissions -p &#x3C;openiam_vhost></code></td><td>As defined in rabbitmq.properties</td><td align="center">☐</td></tr><tr><td>QG3</td><td>Queues consumed</td><td><code>rabbitmqctl list_queues -p &#x3C;openiam_vhost> name messages consumers</code></td><td>Consumers > 0; counts not growing</td><td align="center">☐</td></tr></tbody></table>

### Cache

<table><thead><tr><th width="62">ID</th><th width="168">Check</th><th width="232">How</th><th>Expected</th><th width="74" align="center">Status</th></tr></thead><tbody><tr><td>QH1</td><td>Redis running</td><td><code>systemctl status redis --no-pager</code></td><td>active, enabled</td><td align="center">☐</td></tr><tr><td>QH2</td><td>Authenticated ping</td><td><code>export REDISCLI_AUTH=$(. ${VAULT_HOME}vault.fetch.property.sh vault.secret.redis.password)</code><br><code>redis-cli ping</code></td><td>PONG</td><td align="center">☐</td></tr></tbody></table>

### JDK and certificates

<table><thead><tr><th width="62">ID</th><th width="168">Check</th><th width="232">How</th><th>Expected</th><th width="74" align="center">Status</th></tr></thead><tbody><tr><td>QI1</td><td>New JDK</td><td><code>/usr/local/openiam/jdk/bin/java -version</code></td><td>JDK bundled with 2026.x.x</td><td align="center">☐</td></tr><tr><td>QI2</td><td>Custom CAs restored</td><td><code>/usr/local/openiam/jdk/bin/keytool -list -keystore /usr/local/openiam/jdk/lib/security/cacerts -storepass changeit | grep -c trustedCertEntry</code></td><td>Equals PC7 count; customer CAs present (note: new JDK default roots may be missing)</td><td align="center">☐</td></tr></tbody></table>

### Services

<table><thead><tr><th width="62">ID</th><th width="168">Check</th><th width="232">How</th><th>Expected</th><th width="74" align="center">Status</th></tr></thead><tbody><tr><td>QJ1</td><td>OpenIAM status</td><td><code>openiam-cli status</code></td><td>All components UP</td><td align="center">☐</td></tr><tr><td>QJ2</td><td>No failed units</td><td><code>systemctl --failed</code></td><td>None</td><td align="center">☐</td></tr><tr><td>QJ3</td><td>ESB running new build</td><td><code>pgrep -af openiam-esb.jar</code></td><td>Running from /usr/local/openiam/services/bin</td><td align="center">☐</td></tr><tr><td>QJ4</td><td>Service logs clean</td><td><code>journalctl -u "openiam-*" --since "-2h" -p err --no-pager</code></td><td>No critical errors</td><td align="center">☐</td></tr><tr><td>QJ5</td><td>Ownership</td><td><code>find /usr/local/openiam/services /usr/local/openiam/ui ! -user openiam | head</code></td><td>No output</td><td align="center">☐</td></tr><tr><td>QJ6</td><td>Survives reboot</td><td><code>Reboot in window; repeat J1, J2</code></td><td>Services start automatically</td><td align="center">☐</td></tr></tbody></table>

### Web tier

<table><thead><tr><th width="62">ID</th><th width="168">Check</th><th width="232">How</th><th>Expected</th><th width="74" align="center">Status</th></tr></thead><tbody><tr><td>QK1</td><td>mod_openiam upgraded</td><td><code>rpm -q mod_openiam</code></td><td>mod_openiam-2026.*</td><td align="center">☐</td></tr><tr><td>QK2</td><td>httpd and Nginx</td><td><code>httpd -t; nginx -t</code><br><code>systemctl is-active httpd nginx</code></td><td>Syntax OK; active</td><td align="center">☐</td></tr><tr><td>QK3</td><td>Server certificate</td><td><code>openssl s_client -connect &#x3C;fqdn>:443 -servername &#x3C;fqdn> &#x3C;/dev/null | openssl x509 -noout -subject -enddate</code></td><td>Customer certificate, not expired</td><td align="center">☐</td></tr><tr><td>QK4</td><td>Application URLs</td><td><code>curl -skI https://&#x3C;fqdn>/webconsole</code><br><code>curl -skI https://&#x3C;fqdn>/selfservice</code></td><td>200 or redirect to login</td><td align="center">☐</td></tr></tbody></table>

</details>

### Functional checks

Against the inventory captured at PC8.

<table><thead><tr><th width="62">ID</th><th width="168">Area</th><th width="232">Test steps</th><th>Expected</th><th width="74" align="center">Status</th></tr></thead><tbody><tr><td>QL1</td><td>Administrator login</td><td><code>Log in to Webconsole as super-admin</code></td><td>Dashboard loads</td><td align="center">☐</td></tr><tr><td>QL2</td><td>End-user login</td><td><code>Log in to Selfservice (all users must re-login)</code></td><td>Login succeeds</td><td align="center">☐</td></tr><tr><td>QL3</td><td>SSO / federation</td><td><code>Open each SAML / OIDC / OAuth application</code></td><td>SSO works</td><td align="center">☐</td></tr><tr><td>QL4</td><td>MFA</td><td><code>Log in with an MFA-enabled account</code></td><td>Second factor challenged and accepted</td><td align="center">☐</td></tr><tr><td>QL5</td><td>User lifecycle</td><td><code>Create, modify, disable, re-enable a test user</code></td><td>Succeeds and is audited</td><td align="center">☐</td></tr><tr><td>QL6</td><td>Password management</td><td><code>Change password; forgotten-password reset</code></td><td>Policy enforced; flow works</td><td align="center">☐</td></tr><tr><td>QL7</td><td>Provisioning</td><td><code>Provision test user to a managed system</code></td><td>Account created in target</td><td align="center">☐</td></tr><tr><td>QL8</td><td>Synchronization</td><td><code>Run a source sync manually</code></td><td>Completes; changes applied</td><td align="center">☐</td></tr><tr><td>QL9</td><td>Workflow (activiti)</td><td><code>Raise and approve an access request</code></td><td>Routed, approved, fulfilled</td><td align="center">☐</td></tr><tr><td>QL10</td><td>Groovy (groovy schema)</td><td><code>Execute a custom Groovy rule/script</code></td><td>Compiles and runs (check moved batch-task packages)</td><td align="center">☐</td></tr><tr><td>QL11</td><td>Audit (auditlog schema)</td><td><code>View audit for L5–L9</code></td><td>New events recorded</td><td align="center">☐</td></tr><tr><td>QL12</td><td>Batch tasks (batchtasks schema)</td><td><code>Check schedules; run one task manually</code></td><td>Schedules intact; task completes</td><td align="center">☐</td></tr><tr><td>QL13</td><td>Certification (certification schema)</td><td><code>Open existing campaigns; launch a small test</code></td><td>Visible; reviewer tasks generated</td><td align="center">☐</td></tr><tr><td>QL14</td><td>PBAC (pbac schema)</td><td><code>Evaluate a policy for a test user</code></td><td>Correct decision</td><td align="center">☐</td></tr><tr><td>QL15</td><td>Reports and notifications</td><td><code>Run common reports; trigger an email</code></td><td>Reports render; email received</td><td align="center">☐</td></tr><tr><td>QL16</td><td>UI customizations</td><td><code>Check branding / custom pages</code></td><td>Present or restored from PD5</td><td align="center">☐</td></tr><tr><td>QL17</td><td>Resume scheduled jobs</td><td><code>Re-enable jobs paused in PC5</code></td><td>Jobs run on next schedule</td><td align="center">☐</td></tr></tbody></table>

### Security clean-up

Mandatory, and easy to forget because by this point everything works.

<table><thead><tr><th width="62">ID</th><th width="168">Check</th><th width="232">How</th><th>Expected</th><th width="74" align="center">Status</th></tr></thead><tbody><tr><td>QM1</td><td>Remove exported Vault secrets</td><td><code>shred -u /tmp/export.yaml</code></td><td>CRITICAL — contains every Vault secret in clear text</td><td align="center">☐</td></tr><tr><td>QM2</td><td>Remove DB init script</td><td><code>shred -u /tmp/init.sql</code></td><td>Removed</td><td align="center">☐</td></tr><tr><td>QM3</td><td>Remove temporary copies</td><td><code>rm -f /tmp/cacerts /tmp/env.conf.old</code></td><td>Removed</td><td align="center">☐</td></tr><tr><td>QM4</td><td>Protect upgrade log / terminal history</td><td><code>chmod 600 /usr/local/openiam/upgrade.log</code><br><code>grep -inE "password" /usr/local/openiam/upgrade.log</code></td><td>Passwords echoed by the script are not exposed</td><td align="center">☐</td></tr><tr><td>QM5</td><td>Secure / remove index dump</td><td><code>du -sh /usr/local/openiam/elasticdump</code></td><td>Delete after sign-off</td><td align="center">☐</td></tr><tr><td>QM6</td><td>Clean installation artifacts</td><td><code>ls /usr/src</code></td><td>Cleaned after sign-off</td><td align="center">☐</td></tr><tr><td>QM7</td><td>Credentials recorded properly</td><td>—</td><td>The values you chose at PE3 in your password manager, not in a ticket or a chat message</td><td align="center">☐</td></tr></tbody></table>

## Rollback

Roll back on the change manager's decision, **against** the criteria agreed at PA8. Typical triggers: the run fails in a phase that cannot be resumed, the schema migrations fail, or the functional checks fail in a way that cannot be fixed inside the window.

**The rollback is a restore, not a reverse upgrade.** There is no downgrade path, so it is the snapshot and the database backup — which is why PD1 and PD2 are **not optional**.

{% stepper %}
{% step %}
**Decide and record it**, with the time and who made the call.
{% endstep %}

{% step %}
**Stop the platform** on the host.
{% endstep %}

{% step %}
**Restore the VM snapshot** from PD1.
{% endstep %}

{% step %}
**Restore the databases** from PD2 — every database, to the same point in time as the snapshot.

A snapshot restored against a database from a different moment is **not a working deployment.**
{% endstep %}

{% step %}
**Start up and run the PC checks again** as acceptance. The baseline you captured **before** the window is what tells you the restore is good.
{% endstep %}

{% step %}
**Tell OpenIAM support on the case**, with the log from the failed run.
{% endstep %}
{% endstepper %}

## Sign-off

| Role                    | Name | Decision                            | Date and time |
| ----------------------- | ---- | ----------------------------------- | ------------- |
| Upgrade engineer        |      | Infrastructure post-checks complete |               |
| DBA                     |      | Migrations applied, no failed rows  |               |
| Application owner or QA |      | Functional post-checks accepted     |               |
| Change manager          |      | Upgrade accepted, or rolled back    |               |

Record the rollback deadline passing **as its own event**. Until it does, keep the snapshot and the database backup; after it, the upgrade is the system of record and those are ordinary backups again.

## On a three-node cluster

Everything above still applies on a cluster — the roles, the baseline, the backups, the credential handling, the security clean-up, the rollback shape and the sign-off. Work through that section, and apply this one **on top of it**.

A cluster adds four things, and they are the reason a cluster is called out separately here:

* **A different script, driven by a configuration file** rather than by prompts. It will not ask you for anything it cannot find, and some of what it does not ask for changes the outcome.
* **Consistency** — three nodes have to agree before and after, and they can diverge during.
* **Traffic management**, because the upgrade stops the thing that moves the virtual address.
* **Failover**, which has to be proved again afterwards. A cluster that works but no longer fails over is a worse outcome than one that is plainly broken, because nobody finds out until a node dies.

{% hint style="info" %}
**Run everything from the first node**, inside the repository directory, in a session that survives a dropped connection. The `cluster` helper runs a command on all three — use it for every check that is meant to hold everywhere, so you see a disagreement instead of assuming consistency from one node.
{% endhint %}

### Before you begin on a cluster

#### All three nodes have to match

<table><thead><tr><th width="61">ID</th><th>Check</th><th>How</th><th>Expected</th><th width="72" align="center">Status</th></tr></thead><tbody><tr><td>HA1</td><td>Same operating system major version everywhere</td><td><code>cluster "cat /etc/os-release | grep -E 'PRETTY_NAME|PLATFORM_ID'"</code></td><td>Enterprise Linux, <strong>the same major version on all three</strong></td><td align="center">☐</td></tr><tr><td>HA2</td><td>Same starting release everywhere</td><td><code>cluster cat /usr/local/openiam/version</code></td><td><code>4.2.1.x</code>, and <strong>the same value on every node</strong></td><td align="center">☐</td></tr><tr><td>HA3</td><td>Clock synchronized everywhere</td><td><code>cluster "timedatectl | grep synchronized"</code></td><td><code>yes</code> on all three</td><td align="center">☐</td></tr><tr><td>HA4</td><td>Passwordless SSH from the first node</td><td><code>for ip in &#x3C;node1> &#x3C;node2> &#x3C;node3>; do ssh -p "$SSH_PORT" "$ip" hostname; done</code> then <code>cluster hostname</code></td><td>Three hostnames, no password prompt, and the helper working</td><td align="center">☐</td></tr><tr><td>HA5</td><td>Running as root on the first node</td><td><code>id</code></td><td>Root</td><td align="center">☐</td></tr></tbody></table>

#### The script and its configuration

This is where a cluster upgrade goes wrong before it starts. The script reads its settings from a file; nothing prompts you for a value you left out.

<table><thead><tr><th width="65">ID</th><th>Check</th><th>Expected</th><th width="68" align="center">Status</th></tr></thead><tbody><tr><td>HA6</td><td>The cluster package repository is in place, at the path the script expects</td><td>The repository directory present with a folder for each tier — the secret backend, cache, search, graph and broker. <strong>The script copies from an absolute path</strong>, so it has to be exactly where it expects</td><td align="center">☐</td></tr><tr><td>HA7</td><td>The script sits <strong>inside</strong> that directory</td><td>It sources its configuration from the current directory, so running it from elsewhere finds no configuration</td><td align="center">☐</td></tr><tr><td>HA8</td><td><strong>Unix line endings, and it parses</strong></td><td><code>file ha_upgrade.sh</code> shows no CRLF; <code>bash -n ha_upgrade.sh</code> is silent. A script edited on Windows fails in ways that do not name the cause</td><td align="center">☐</td></tr><tr><td>HA9</td><td>The configuration file is complete</td><td>Cluster addresses, SSH port, secret-store address and port, environment, deployment type, broker port, target release and internet availability all present</td><td align="center">☐</td></tr><tr><td>HA10</td><td><strong>Internet availability set explicitly</strong></td><td>See below</td><td align="center">☐</td></tr><tr><td>HA11</td><td>The target release matches what the script accepts</td><td>The script compares your target against a release string compiled into it and <strong>exits if they differ</strong>. Use the script published for your target</td><td align="center">☐</td></tr><tr><td>HA12</td><td>The configuration block filled in</td><td>Database host — <strong>the script exits if it is empty</strong> — database port, search host and port, the dump target directory, and the new database names agreed with the DBA</td><td align="center">☐</td></tr><tr><td>HA13</td><td>Known script defects decided</td><td>See <a href="#known-defects-ask-before-you-run">Known defects</a></td><td align="center">☐</td></tr><tr><td>HA14</td><td>Tools present on the first node</td><td><code>jq</code>, <code>rsync</code>, <code>m4</code>, <code>curl</code>, <code>cpio</code>, <code>rpm2cpio</code>, <code>tar</code>, <code>unzip</code>. The script installs the index-dump tooling itself, so it needs a reachable package repository</td><td align="center">☐</td></tr></tbody></table>

{% hint style="danger" %}
**HA10 is the trap.** The script **never asks** whether you have internet access. If the variable is unset it takes the **offline path silently** — then fails later looking for artifacts nobody staged, with an error that points at the missing files rather than at the unset variable. Set it explicitly either way, and set it truthfully.
{% endhint %}

#### Artifacts

<table><thead><tr><th width="69">ID</th><th>Check</th><th>Expected</th><th width="71" align="center">Status</th></tr></thead><tbody><tr><td>HA15</td><td>Online: the download site answers from every node that fetches</td><td>A response from each</td><td align="center">☐</td></tr><tr><td>HA16</td><td>Air-gapped: artifacts staged</td><td>The package, the dependency archive <strong>for your EL version</strong>, and both binary archives — present and non-zero</td><td align="center">☐</td></tr><tr><td>HA17</td><td><strong>Artifact integrity checked</strong></td><td><code>tar -tzf &#x3C;archive> >/dev/null</code> on each archive, and <code>rpm -K</code> on the package. Compare against the checksums support gave you, if they did</td><td align="center">☐</td></tr></tbody></table>

#### Baseline, clusterwide

[The health baseline above](#health-baseline) applies, with one change: take every measurement on **all three nodes**, and add the cluster-membership checks — because "healthy" for a cluster includes "all members present".

<table><thead><tr><th width="73">ID</th><th>Check</th><th>Expected</th><th width="70" align="center">Status</th></tr></thead><tbody><tr><td>HA18</td><td>Services up on every node</td><td><code>cluster openiam-cli status</code>. <strong>Do not upgrade a cluster that is already degraded</strong></td><td align="center">☐</td></tr><tr><td>HA19</td><td>Search tier healthy <strong>with all three members</strong></td><td><code>green</code>, three nodes</td><td align="center">☐</td></tr><tr><td>HA20</td><td>Secret store and its backend healthy</td><td>The store initialized and unsealed; every backend endpoint healthy</td><td align="center">☐</td></tr><tr><td>HA21</td><td>Cache, broker and graph store memberships</td><td>Cache: one primary and two replicas. Broker: three running nodes. Graph store: all nodes up and normal</td><td align="center">☐</td></tr><tr><td>HA22</td><td><strong>Configuration snapshot from every node</strong></td><td>Taken per node, plus the installed-package list, because the nodes are what you compare afterwards</td><td align="center">☐</td></tr><tr><td>HA23</td><td>Queues drained across <strong>every</strong> virtual host</td><td>No queue with pending messages on any of them. The broker is reinitialized and in-flight messages are lost</td><td align="center">☐</td></tr></tbody></table>

#### Backups, clusterwide

[The backup list above](#backups) applies, times three, with one warning that is specific to the cluster script.

{% hint style="danger" %}
**The script backs up the platform directory only, and it does so while the services are still running.** That is not a consistent backup of anything stateful, and it is not a rollback position. The database backup, the snapshots of all three nodes and the secret-store snapshot are what you actually roll back to — and all of them must exist, restore-tested, before the go decision.
{% endhint %}

<table><thead><tr><th width="78">ID</th><th>Check</th><th>Expected</th><th width="68" align="center">Status</th></tr></thead><tbody><tr><td>HA24</td><td>Database backup, restore-tested by the DBA</td><td>Every database the deployment uses</td><td align="center">☐</td></tr><tr><td>HA25</td><td>Snapshots of <strong>all three</strong> nodes</td><td>Taken with services stopped, or crash-consistent snapshots explicitly accepted</td><td align="center">☐</td></tr><tr><td>HA26</td><td>Secret-store backend snapshot</td><td>Saved</td><td align="center">☐</td></tr><tr><td>HA27</td><td>Search data</td><td>A restorable copy exists — a snapshot repository, or the script's dump plus the node snapshots</td><td align="center">☐</td></tr><tr><td>HA28</td><td><strong>Everything stored off the nodes</strong></td><td>Not only on the hosts being upgraded</td><td align="center">☐</td></tr></tbody></table>

#### Traffic, and the thing that moves the address

<table><thead><tr><th width="76">ID</th><th>Check</th><th>Expected</th><th width="70" align="center">Status</th></tr></thead><tbody><tr><td>HA29</td><td>Inter-node ports open</td><td>The search tier's client and transport ports, and the HA ports the guide lists. The script opens the secret backend's own ports itself</td><td align="center">☐</td></tr><tr><td>HA30</td><td><strong>A plan for the virtual address</strong></td><td>See below</td><td align="center">☐</td></tr><tr><td>HA31</td><td>Load balancer plan</td><td>A maintenance page, or the pool drained for the window</td><td align="center">☐</td></tr><tr><td>HA32</td><td>Credentials chosen for the new databases and the search user</td><td><strong>Not the defaults — they are the same as the user name</strong></td><td align="center">☐</td></tr></tbody></table>

{% hint style="warning" %}
**The script stops and disables the service that holds the virtual address.** It does not put it back for you. Decide before the window how traffic reaches the cluster during the upgrade and how it reaches it afterwards, and whose job each half is — otherwise the upgrade finishes, every check passes, and nobody can reach the platform.
{% endhint %}

### Known defects: ask before you run

The cluster script has carried defects that change the outcome rather than stopping the run. Two that have been seen in the field:

| Defect                                                                       | What it does                                                                                                                                                                  |
| ---------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| A line-continuation fault in the MariaDB path                                | Affects how the generated SQL is assembled                                                                                                                                    |
| A duplicated user definition in the PostgreSQL path, for the Groovy database | The user that gets created may not be the one the secret store was told about — so the service authenticates with a credential for a user that does not exist under that name |

{% hint style="warning" %}
**Ask OpenIAM support for the current defect list for your target release's script, on the case you opened.** Treat anything on it as a decision to make before the window — fixed, or accepted knowingly — rather than something to discover at two in the morning. The second defect above is why [the post-check below](#after-the-upgrade-cluster-checks) compares the Groovy database's user against what the secret store holds.
{% endhint %}

### The go decision: three more rows

Same gate as [above](#the-go-decision), with three more rows that only a cluster has.

<table><thead><tr><th width="652">Criterion</th><th width="105" align="center">Status</th></tr></thead><tbody><tr><td>Every pre-check above passed, or accepted knowingly</td><td align="center">☐</td></tr><tr><td>Backups complete and <strong>stored off the nodes</strong></td><td align="center">☐</td></tr><tr><td>DBA, infrastructure and load-balancer teams available <strong>for the window</strong>, not just informed of it</td><td align="center">☐</td></tr><tr><td>The virtual-address plan agreed, both halves</td><td align="center">☐</td></tr><tr><td>Rollback plan agreed, with its deadline</td><td align="center">☐</td></tr><tr><td><strong>Decision, approver and time recorded</strong></td><td align="center">☐</td></tr></tbody></table>

### Running it on a cluster

Run from the first node, inside the repository directory, in a protected session. The script **writes its own log**; follow that rather than the console.

The "**do not simply re-run it**" rule applies here too. The phases are the cluster script's own, and run in a different order from the single-node ones — the secret backend is migrated before the search tier rather than after:

<table><thead><tr><th width="52">#</th><th width="186">Phase</th><th width="250">Command</th><th>What happens</th></tr></thead><tbody><tr><td>S0</td><td>Prepare the session</td><td><code>screen -S oiam_upgrade</code><br><code>cd /usr/src/openiam-cluster-rpm</code><br><code>. /usr/local/openiam/env.conf ; . ./infra.conf</code></td><td>Go/No-Go call held: all Part A checks passed or accepted.<br>Downtime has not started yet.</td></tr><tr><td>S1</td><td>Start the script</td><td><code>./ha_upgrade.sh</code></td><td>Sources env.conf and infra.conf; all output goes to ha_upgrade.log.<br>Adds FLYWAY_* variables for groovy, auditlog, batchtasks, certification and pbac to env.conf (node 1).<br>Checks DB_HOST and UPGRADE_TO_VERSION, reads the current version and EL version, reads DB / Elasticsearch / Activiti credentials from Vault.</td></tr><tr><td>S2</td><td><code>backup_old_version</code></td><td><code>cluster tar -cf $HOME_DIR/backup/openiam-&#x3C;old>-files.tar $HOME_DIR</code></td><td>Tar of the full OpenIAM home on every node (services still running).<br>Can take long on big installations.</td></tr><tr><td>S3</td><td><code>download_artifacts</code></td><td>(online) download openiamrepo, backend, frontend and RPM<br>(offline) copy from /usr/src to all nodes<br><code>rpm2cpio … | cpio -idm --directory=/usr/src</code></td><td>Artifacts distributed to all nodes and the RPM extracted to /usr/src/openiam on each node.<br>Offline mode stops here if /usr/src/openiamrepo.tar.gz or the opensearch folder is missing.</td></tr><tr><td>S4</td><td><code>stop_openiam</code> <strong>(the outage starts here)</strong></td><td><code>cluster openiam-cli stop</code></td><td>All OpenIAM services stopped; the script waits until the ESB process exits.<br>Last safe point to abort without component changes.</td></tr><tr><td>S5</td><td><code>replace_old_version</code></td><td>(removes services/bin, *.war, health files, openiamrepo, jdk, curator)</td><td>New backend, frontend, openiamrepo and JDK installed on all nodes.<br>JDK cacerts copied to /tmp and restored into the new JDK.<br>Ownership set to openiam:openiam.</td></tr><tr><td>S6</td><td><code>upgrade_redis</code></td><td><code>cluster yum remove redis -y ; rm dump.rdb</code><br><code>redis/init.sh</code></td><td>Redis and Sentinel reinstalled. Cached data and sessions are lost (expected).</td></tr><tr><td>S7</td><td><code>migrate_etcd_to_consul</code></td><td><code>medusa export secret … -o /tmp/export.yaml</code><br><code>consul/init.sh</code><br><code>medusa import secret … /tmp/export.yaml</code></td><td>Opens 8300/8301 tcp+udp. Exports all Vault secrets, stops Vault and etcd (etcd disabled, data kept), builds the Consul cluster, starts Vault and imports the secrets.<br>Script exits with "Secrets were not exported" if the export fails.</td></tr><tr><td>S8</td><td><code>backup_elasticsearch</code></td><td><code>multielasticdump --direction=dump --output=${TARGET_DIRECTORY}</code></td><td>Installs npm/elasticdump if needed and dumps every index.<br>Deletes the provisionrequest-*, connectorreply-*, provisionconnectorrequest-* and .security-* dumps (not migrated).</td></tr><tr><td>S9</td><td><code>bootstrap_vault</code> <strong>(interactive)</strong></td><td>Prompts: OpenSearch user/password, then user/password for groovy, auditlog, batchtasks, certification, pbac</td><td>Typing is hidden; Enter accepts the default shown.<br>The script then prints ALL values in clear text for confirmation (also written to ha_upgrade.log). Type y to continue, n to re-enter.<br>Values are written to Vault under secret/openiam/…</td></tr><tr><td>S10</td><td><code>migrate_elasticsearch_to_opensearch</code></td><td><code>opensearch/init.sh</code><br><code>multielasticdump --direction=load</code><br><code>curator/init.sh</code></td><td>Removes Elasticsearch and its data, stops AND disables keepalived, installs OpenSearch, loads the dump, swaps elasticsearch.properties for opensearch.properties (https), initializes curator.</td></tr><tr><td>S11</td><td><code>upgrade_cassandra_janusgraph</code></td><td><code>graph/init.sh</code></td><td>Stops JanusGraph, removes Cassandra and /var/lib/cassandra, installs the new JanusGraph service and initializes the graph.</td></tr><tr><td>S12</td><td><code>rabbitmq_reinit</code></td><td><code>rabbitmqctl add_vhost … ; set_permissions …</code></td><td>Regenerates rabbitmq.properties (node 1) and creates the 17 openiam_* vhosts with permissions for user openiam.</td></tr><tr><td>S13</td><td>apply_migrations (MANUAL DBA STEP)</td><td>Script pauses: "Please perform script located at /tmp/init.sql …"</td><td>DBA runs /tmp/init.sql on the database server to create the groovy, auditlog, batchtasks, certification and pbac databases/users, then the operator presses Enter.<br>datasource.properties is regenerated (node 1) and Flyway repair/migrate runs for openiam, activiti, groovy, auditlog, batchtasks, certification and pbac.<br>/tmp/init.sql contains passwords: delete after use.</td></tr><tr><td>S14</td><td><code>upgrade_app</code></td><td><code>cluster openiam-cli start ; openiam-cli status</code></td><td>New systemd units and service files deployed; application port block (8080-8086) added to env.conf; services started on all nodes.</td></tr><tr><td>S15</td><td><code>httpd_nginx_conf</code></td><td><code>nginx/init.sh on each node</code><br><code>yum install mod_openiam-2026.* ; systemctl restart httpd</code></td><td>Nginx configured on all nodes; old mod_openiam removed and the 2026.x.x module installed; httpd restarted.</td></tr><tr><td>S16</td><td><code>prompt_monitoring</code> <strong>(interactive)</strong></td><td>Answer y / n</td><td>y installs the Prometheus + Grafana stack; n skips it.</td></tr><tr><td>S17</td><td>Finish</td><td><code>tail ha_upgrade.log</code></td><td>Version file written (node 1 only), "Done" printed with the backup path.<br>Downtime ends only after the post-checks in Part C pass.</td></tr></tbody></table>

{% hint style="warning" %}
**A cluster run stops for a person three times, not twice.** As well as the credentials and the DBA, the script asks near the end whether to install the Prometheus and Grafana monitoring stack, and waits for `y` or `n`. Decide that before the window so nobody answers it on the spot.
{% endhint %}

One more difference: the script adds migration variables for each new database to the configuration on **the first node only**, which is one more reason the nodes are compared afterwards rather than assumed identical.

### After the upgrade: cluster checks

Run [the post-checks above](#after-the-upgrade) first — they apply per node — then these.

<table><thead><tr><th width="71">ID</th><th>Check</th><th>Expected</th><th width="79" align="center">Status</th></tr></thead><tbody><tr><td>HB1</td><td>The run completed</td><td>The script's log ends cleanly, with no errors above it</td><td align="center">☐</td></tr><tr><td>HB2</td><td>Same release on all three nodes</td><td>Identical, and the target</td><td align="center">☐</td></tr><tr><td>HB3</td><td><strong>Configuration consistent across nodes</strong></td><td>Compare each node against the snapshot from HA22 and against each other. The script edits the first node; divergence here is the expected failure mode</td><td align="center">☐</td></tr><tr><td>HB4</td><td>Secret store and its new backend</td><td>Store unsealed, every backend member healthy, secrets readable from every node</td><td align="center">☐</td></tr><tr><td>HB5</td><td>Cache cluster re-formed</td><td>One primary, two replicas. Empty is expected</td><td align="center">☐</td></tr><tr><td>HB6</td><td>Search tier re-formed over HTTPS</td><td>Healthy, three members, and <strong>the index inventory compared against the baseline</strong></td><td align="center">☐</td></tr><tr><td>HB7</td><td>Graph store and broker memberships</td><td>All nodes present; queues recreated, bound and empty</td><td align="center">☐</td></tr><tr><td>HB8</td><td>Schema migrations</td><td>No failed rows in any database's migration history</td><td align="center">☐</td></tr><tr><td>HB9</td><td><strong>The Groovy database's user matches the secret store</strong></td><td>The user that exists is the one the store holds a credential for — see <a href="#known-defects-ask-before-you-run">Known defects</a></td><td align="center">☐</td></tr><tr><td>HB10</td><td>Web tier on every node</td><td>The console loads through each node's own path</td><td align="center">☐</td></tr><tr><td>HB11</td><td><strong>Trusted certificates on every node</strong></td><td>Alias lists match the baseline, per node. A custom CA lost on one node only produces failures that come and go with routing</td><td align="center">☐</td></tr><tr><td>HB12</td><td><strong>Failover proved</strong></td><td>See below</td><td align="center">☐</td></tr><tr><td>HB13</td><td>Virtual address and load balancer restored</td><td>Traffic reaches the cluster by the agreed route, and the pool is back in service</td><td align="center">☐</td></tr><tr><td>HB14</td><td>Monitoring</td><td>Back in place, and alerting on the new tiers rather than the ones that were replaced</td><td align="center">☐</td></tr></tbody></table>

{% hint style="danger" %}
**HB12 is the check that distinguishes a cluster upgrade from three host upgrades.** Take one node out deliberately and confirm the platform keeps serving: the search tier stays available, the broker keeps delivering, the cache promotes a replica, and sign-in still works. Then put it back and confirm it rejoins every tier.

Do this **before** sign-off, inside the window, while you still have the rollback position. A cluster that serves from three nodes but cannot lose one has not been upgraded successfully — and the only time you find out otherwise is when a node fails on its own.
{% endhint %}

### Rollback: all three nodes together

The shape is [as above](#rollback) — a restore, **not a reverse upgrade** — with the cluster's own constraint: **restore all three nodes and the databases to the same point in time.** A cluster rebuilt from snapshots taken at different moments will form, report healthy, and disagree with itself about state.

{% stepper %}
{% step %}
**Stop the platform on all three nodes.** Run `openiam-cli stop`, then stop the search tier, the secret backend and the graph store.
{% endstep %}

{% step %}
**Restore the VM snapshots of all three nodes.** This is the fastest and most complete route, because it brings the old secret backend, search tier, graph store, cache and binaries back together, at one point in time.
{% endstep %}

{% step %}
**Restore the databases from the backup taken at HA24**, to the same point in time as the snapshots. The databases the upgrade created — `groovy`, `auditlog`, `batchtasks`, `certification` and `pbac` — can be dropped.
{% endstep %}

{% step %}
**Without snapshots**, restore `$HOME_DIR/backup/openiam-<old>-files.tar` on each node and re-enable the old secret backend. The search tier, graph store and cache packages have to be reinstalled by hand, so raise this with OpenIAM support on your case before taking this route rather than during it.
{% endstep %}

{% step %}
**Put the virtual address and the load balancer back.** Run `systemctl enable --now keepalived`, then return the pool to service — the upgrade disabled the service that moves the address, and nothing puts it back for you.
{% endstep %}

{% step %}
**Run the HA18 to HA23 baseline checks again** as acceptance. The baseline you captured before the window is what tells you the cluster is back to where it started.
{% endstep %}
{% endstepper %}

### Sign-off: one more row

[The sign-off above](#sign-off) applies, with one row added — because on a cluster it is the easiest thing to skip and the most expensive to have skipped.

| Role                      | Name | Decision                                                        | Date and time |
| ------------------------- | ---- | --------------------------------------------------------------- | ------------- |
| Upgrade engineer          |      | Infrastructure post-checks complete, **all nodes compared**     |               |
| DBA                       |      | Migrations applied, no failed rows                              |               |
| Application owner or QA   |      | Functional post-checks accepted                                 |               |
| Infrastructure or network |      | **Failover proved, virtual address and load balancer restored** |               |
| Change manager            |      | Upgrade accepted, or rolled back                                |               |

## Related

* [Release-specific steps](https://docs.openiam.com/installation/rpm/rpm-upgrade/release-specific-steps.md) — the databases and services this upgrade crosses
* [After a major upgrade](https://docs.openiam.com/installation/rpm/rpm-upgrade/after-a-major-upgrade.md) — what to put right once it completes
* [Minor upgrade](https://docs.openiam.com/installation/rpm/rpm-upgrade/minor-upgrade.md) — staying on the line you are already on
* [Major upgrade — Kubernetes](https://docs.openiam.com/installation/kubernetes/kubernetes-upgrade/major-upgrade.md) — the same move, on a cluster
* [RPM — three nodes](https://docs.openiam.com/installation/rpm/rpm-ha.md) — the clustered shape this upgrade also covers

*Worked against OpenIAM 2026.8.2. Last reviewed 25 October 2026.*


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.openiam.com/installation/rpm/rpm-upgrade/major-upgrade.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
