Migration of index data from older ElasticSearch versions to newer one

When changing from older version of OpenIAM (v.4.2.0.x) to newer ones, might require migrating the index data from older version of Elasticsearch (namely 6.8.23) to its higher version (with OpenIAM v4.2.1.10 the ElasticSearch version is 7.17.3). To migrate data, follow the instructions below.

Installing Elasticdump

Elasticdump is a Node.js based tool used to export and import data from ElasticSearch. Hence, to successfully complete migration, make sure to install the Elasticdump utility in source and destination server following the steps below.

1. Add a Node.js repository by running the following command.

curl -fsSL https://rpm.nodesource.com/setup_18.x | sudo bash

2. Install Node.js and npm, by running

sudo yum install -y nodejs

3. Install ElasticDump Using npm, by running the following command.

sudo npm install -g elasticdump

4. Verify your installation by running

elasticdump –version

There is also an alternative way to install Elasticdump utility. Run the commands below.

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.5/install.sh | bash
source ~/.bashrc
nvm install node
npm --version
npm install elasticdump -g
elasticdump --version

Migrating the data

Now that all the required prerequisites are met, use the steps below to migrate the index data.

1. Check the running indices in source and destination server by running the command below

curl -u elastic:4AGakIGu0g1nuPzHdQfrljgsiMCEI8TZ -X GET http://localhost:9200/_cat/indices?v

2. Checking index data for audit log by running

curl -u elastic:4AGakIGu0g1nuPzHdQfrljgsiMCEI8TZ localhost:9200/auditlog/_search?pretty

3. Backup the below config data in source and destination server.
   • Backup of data in json format.

elasticdump --input=http://elastic:4AGakIGu0g1nuPzHdQfrljgsiMCEI8TZ@localhost:9200/auditlog --output=auditlog.json --type=data

• Backup of mapping in json format.

elasticdump --input=http://elastic:4AGakIGu0g1nuPzHdQfrljgsiMCEI8TZ@localhost:9200/auditlog --output=auditlog-mapping.json --type=mapping

• Backup of settings in json format.

elasticdump --input=http://elastic:4AGakIGu0g1nuPzHdQfrljgsiMCEI8TZ@localhost:9200/auditlog --output=auditlog-settings.json --type=settings

• Combine mapping and setting json.

Note: If this is a new OpenIAM installation with a higher version of ES and no data is present in the audit log, then please proceed with below steps.

1. Delete the audit log index by running the following

curl -X DELETE http://elastic:4AGakIGu0g1nuPzHdQfrljgsiMCEI8TZ@localhost:9200/auditlog

2. Verify the deletion of the audit log index and run the below command.

curl -u elastic:4AGakIGu0g1nuPzHdQfrljgsiMCEI8TZ -X GET http://localhost:9200/_cat/indices?v

3. Create the new audit log index with same name and combined json of setting and mapping of destination server by running

curl -X PUT "http://elastic:4AGakIGu0g1nuPzHdQfrljgsiMCEI8TZ@localhost:9200/auditlog" -H 'Content-Type: application/json' -d @/root/auditlog-settings-mappings.json

4. Change the format of old auditlog.json data to new format, as shown below.

Sample new format of auditlog.json from destination server.

Sample old format of auditlog.json from the old server.

5. Change below data in old auditlog.json.

• _type":"auditlog" to _type":"_doc";
• date and time setting from all places in old audiglog.json;
• “ timestamp":1724990404488/”lastIndexDateTime":1724990403103 to "timestamp":"2024-08-27T18:20:03.465Z"/”lastIndexDateTime":"2024-08-27T18:20:05.507Z”

Sample old auditlog.json after changes.

.

6. Import the final auditlog.json data using the below command in the destination server.

elasticdump --input=manish.json --output=http://elastic:4AGakIGu0g1nuPzHdQfrljgsiMCEI8TZ@localhost:9200/auditlog --type=data

Verify the changes in OpenIAM UI.