Reindex Elasticsearch

Elasticsearch is one the infrastructure elements used by OpenIAM. It is needed for storing data and providing quick access to it. For instance, when calling API, instead of contacting a database, in some cases Elasticsearch is contacted to make the whole process faster since calling database might be quite time consuming, as compared to calling Elasticsearch since Elasticsearch is meant to conduct a quick search in data stored in it.

There are several types of indices in Elasticsearch. You can view them by calling the API using the following command (in RPM).

curl localhost:9200/_cat/indices

To access indices in Docker, perform the command above after entering the Elasticsearch container.

After executing the command, you will see a list of existing indices, as shown below.

There are three indices being curated by Elasticsearch itself, it means Elasticsearch refreshes them on its own. These are provisionconnectorrequest, connectorreply and provisionrequest. For each of these indices there is a curating script with an index setting, e.g. its size (up to 5Gb) or its age (up to 10 days of life), that is being called and closing the index and opening a new one when the setting becomes true. The indexes are closed since we believe they store some temporary data that are needed for some limited time only and not necessary further, hence, needed to be erased. More info on indexes can be found in Elasticsearch Curator reference

There are also indices that are connected to an entity or a database. These are provided in a table below.

There are some indices that are allowed to be reindexed, i.e. cleared, and re-filled by OpenIAM upon clearing in case this index is connected to an entity or a database. In this case, upon clearing OpenIAM will start searching for data and refill Elasticsearch with the new data based on the index. In case an index is not connected to a database (e.g. org.openiam.elasticsearch.model.orphanDoc), when reindexing it the data will be lost without further restoration.

Reindexing is done to refill indices upon some change, e.g. making a new release of OpenIAM and changing fields in tables in a database. The need to reindex is always noted in the release notes that can be found in this section of the documents. To reindex any of the docs, go to the tab and click on the index of interest and click Ok in the pop-up window.

Note: that audit logs in OpenIAM are stored exclusively in Elasticsearch and they are not copied or stored in a database. Hence, use caution when reindexing org.openiam.elasticsearch.model.IdmAuditLogDoc index, since clearing it means losing all audit logs till now.

Note also that indices are added to the list by default, you can't delete or add indices on your own.