Elasticsearch read-only state

Overview

There might be an issue where the customer is unable to log into production environment, receiving an Invalid login/password error. Upon investigating the logs, one can discover that the Elasticsearch instance had entered a read-only state, likely due to running out of disk space. Elasticsearch uses the /var/lib/ partition for its configuration and the /var/log/ directory for log storage. The /var/lib/ partition had reached approximately 90% utilization, which caused all Elasticsearch indices to enter a read-only mode.

Example of disk space utilization is given below.

[root@nyvpopeniam elasticsearch]# df -h /var/lib/
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/rhel-root     50G   45G  5.7G  89% /

Recommended actions

1. Free up disk space. Remove old or unused data from the /var/lib/ partition, or transfer files to another location to free up disk space for Elasticsearch.
2. Extend the partition and add additional disk space. Extend the partition by allocating more disk space to the Elasticsearch instance.

Recommendation: Allocate 100GB to the /var/lib/ partition to prevent future issues with Elasticsearch running out of disk space.

3. Manually remove the read-only mode. Once sufficient disk space is freed up, manually remove the read-only mode from all Elasticsearch indices. To do it, follow the steps below.

• Check Disk Space. Ensure there is adequate free space in the /var/lib/ partition before proceeding.
• Backup Indices. Take a backup of all Elasticsearch indices. Detailed steps for backup in Docker are given here and for RPM deployment here.
• List all running indices. Run the following command to list all running indices.

[root@nyvpopeniam vault]# curl -X GET "localhost:9200/_cat/indices?v&h=index,health,status,docs.count,store.size"
index                     health status docs.count store.size
provisionstream           yellow open         1703      6.6mb
email                     yellow open         7615      1.3mb
metadatatype              yellow open          178     44.7kb
reconsynchstorageobject   yellow open         9642     19.7mb
reconsynchstorage         yellow open         1901    191.2kb
provisionconnectorrequest yellow open        79900     18.8mb
organization              yellow open          161     30.1kb
resource                  yellow open         1538      1.1mb
managedsystem             yellow open          217     94.6kb
phone                     yellow open         1729    337.3kb
role                      yellow open         2769      2.1mb
users                     yellow open       267136     47.7mb
group                     yellow open         6842        5mb
connectorreply            yellow open        79875       20mb
auditlog                  yellow open      7461268      2.1gb
userattributes            yellow open            0       261b
logins                    yellow open        26727      3.7mb
provisionrequest          yellow open         4101    181.3mb
orphan                    yellow open         2464    313.7kb

• Verify read-only setting of indices, using the following command.

[root@nyvpopeniam asingla]# curl -X GET "localhost:9200/provisionstream/_settings"
Example output with read-only status:
{
  "provisionstream": {
    "settings": {
      "index": {
        "blocks": {
          "read_only_allow_delete": "true"
        }
      }
    }
  }
}

4. Change read-only mode to write mode.

For a single index, run the following command to change its mode.

curl -X PUT "localhost:9200/provisionstream/_settings" -H 'Content-Type: application/json' -d '{ "index.blocks.read_only_allow_delete": null }'

To change the mode for all indices, run the following.

curl -X PUT "localhost:9200/_all/_settings" -H 'Content-Type: application/json' -d '{ "index.blocks.read_only_allow_delete": "false" }'

5. After changing the mode, verify the settings again by running the following command.

[root@nyvpopeniam ~]# curl -X GET "localhost:9200/provisionstream/_settings"

Ensure that the read_only_allow_delete setting is now false.

6. Attempt to log into OpenIAM again. If login attempts still fail, the user account may be locked in the database. Unlock the user account and retry.

Additional notes

1. For higher versions (>6) of Elasticsearch, username and password will require to get information from Elasticsearch cluster. Hence, below you can find sample commands to the above mentioned command.

curl -u username:password -X GET "localhost:9200/_cat/indices?v&h=index,health,status,docs.count,store.size"

curl -u username:password -X GET "localhost:9200/provisionstream/_settings"

1. For Docker or K8 deployment of OpenIAM, where Elasticsearch will be running as container/pod, follow the below steps.

To check mount point and disk usage on those specific locations

Docker

• Inspect the Elasticsearch container.

docker inspect openiam-elasticsearch-storage_storage

• Enter the container and perform curl command to get data related with Elasticsearch cluster.

docker exec -it <container-ID> /bin/bash/

Kubernetes

• Describe the Elasticsearch pod and PVC and get the details about PVC/SC/capacity.

kubectl describe pod <pod-name> -n <namespace>

kubectl describe pvc <pvc-name> -n <namespace> — Information about the PVC, such as its status, storage class, capacity.

• Enter the pod and run the curl command to get data related with Elasticsearch cluster.

Kubectl exec -it <elasticsearch container id> /bin/bash/