> For the complete documentation index, see [llms.txt](https://docs.openiam.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.openiam.com/troubleshooting/signing-in/elasticsearch-readonly-state.md).

# Elasticsearch read-only state

Nobody can sign in because the search tier has put its indices into read-only mode after running low on disk.

| Symptom                                                                          | Likely cause                                                                        | Resolution                                                                                                      |
| -------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- |
| Every user gets `Invalid login/password`, including accounts known to be correct | The search tier has gone read-only after its disk filled, so nothing can be written | Free the disk, then [clear the read-only block](#clear-the-read-only-block)                                     |
| One user still cannot sign in after the block is cleared                         | That account locked itself during the outage                                        | Unlock the account — [Unlock sysadmin](/troubleshooting/signing-in/unlocksysadmin.md) covers the same mechanism |

## Why a full disk looks like a password failure

Elasticsearch protects itself when the disk it is on gets close to full: once the **flood-stage watermark** is crossed — 95% of the filesystem by default — it sets `index.blocks.read_only_allow_delete` on every index. Writes then fail while reads keep working, and sign-in surfaces that as an invalid-credentials error rather than as a storage problem.

Elasticsearch keeps its data under `/var/lib/` and its logs under `/var/log/`, so it is that filesystem that matters:

{% code overflow="wrap" %}

```
# df -h /var/lib/
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/rhel-root     50G   45G  5.7G  89% /
```

{% endcode %}

## Free the disk first

Clearing the block without freeing space just lets Elasticsearch set it again.

{% stepper %}
{% step %}
**Reclaim or extend.** Remove or move data off the `/var/lib/` filesystem, or extend the partition. [Running out of disk space](/troubleshooting/performance/lackof_disk_space.md) covers finding what is holding it.
{% endstep %}

{% step %}
**Back the indices up** before changing index settings — [Back up and restore](/installation/rpm/rpm-single-node/back-up-and-restore.md) for RPM, and your cluster's own snapshot procedure on Kubernetes.
{% endstep %}
{% endstepper %}

{% hint style="success" %}
**100 GB on `/var/lib/` keeps this from recurring** on a deployment of the size shown above.
{% endhint %}

## Clear the read-only block

{% stepper %}
{% step %}
**List the indices**, to see what is there and how large it is.

{% code overflow="wrap" expandable="true" %}

```
# curl -X GET "localhost:9200/_cat/indices?v&h=index,health,status,docs.count,store.size"
index                     health status docs.count store.size
provisionstream           yellow open         1703      6.6mb
email                     yellow open         7615      1.3mb
reconsynchstorageobject   yellow open         9642     19.7mb
provisionconnectorrequest yellow open        79900     18.8mb
users                     yellow open       267136     47.7mb
connectorreply            yellow open        79875       20mb
auditlog                  yellow open      7461268      2.1gb
logins                    yellow open        26727      3.7mb
provisionrequest          yellow open         4101    181.3mb
```

{% endcode %}

`auditlog` is usually much the largest, and is the first place to look when the disk filled.
{% endstep %}

{% step %}
**Confirm an index is actually blocked.**

{% code overflow="wrap" %}

```bash
curl -X GET "localhost:9200/provisionstream/_settings"
```

{% endcode %}

A blocked index reports:

{% code overflow="wrap" %}

```json
{ "provisionstream": { "settings": { "index": { "blocks": {
  "read_only_allow_delete": "true" } } } } }
```

{% endcode %}
{% endstep %}

{% step %}
**Clear it.** For every index at once:

{% code overflow="wrap" %}

```bash
curl -X PUT "localhost:9200/_all/_settings" -H 'Content-Type: application/json' -d '{ "index.blocks.read_only_allow_delete": null }'
```

{% endcode %}

Or for one:

{% code overflow="wrap" %}

```bash
curl -X PUT "localhost:9200/provisionstream/_settings" -H 'Content-Type: application/json' -d '{ "index.blocks.read_only_allow_delete": null }'
```

{% endcode %}
{% endstep %}

{% step %}
**Verify**, by reading the settings again — `read_only_allow_delete` should be gone or `false` — and sign in.
{% endstep %}
{% endstepper %}

{% hint style="info" %}
**A secured search tier needs credentials on every command above.** Pass them with `-u`, reading them from the secret store as `vault.secret.opensearch.username` and `vault.secret.opensearch.password` rather than typing them inline — see [Reading a credential from the secret store](/installation/commands.md).

```bash
curl -u <user>:<password> -X GET "localhost:9200/_cat/indices?v"
```

{% endhint %}

## On a container deployment

The commands are the same; reaching the search tier is not.

The pod and its claim tell you the storage class and capacity:

{% code overflow="wrap" %}

```bash
kubectl describe pod <pod-name> -n <namespace>
kubectl describe pvc <pvc-name> -n <namespace>
```

{% endcode %}

Then run the `curl` commands from inside the pod:

```bash
kubectl exec -it <pod-name> -n <namespace> -- /bin/bash
```


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.openiam.com/troubleshooting/signing-in/elasticsearch-readonly-state.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.