> For the complete documentation index, see [llms.txt](https://docs.openiam.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.openiam.com/administration-guide/gateway/troubleshoot-the-gateway.md). # Troubleshoot the Access Gateway Diagnose a request that fails through the OpenIAM Access Gateway — locate the failing hop, collect the four pieces of evidence that settle it, and work the symptom/cause/fix table. ## What this is, and when you need it A gateway failure looks the same from the browser almost regardless of cause: a redirect loop, an error page, or the application's own login form coming back. The diagnosis is always the same shape — find which hop failed, then read the one line that says why. Work this page in order. The first section places the failure on the request path, the second collects the evidence, and the tables after that name the fix. ## Locate the failure on the request path The request path is described in [How the Access Gateway works](/administration-guide/gateway/access-gateway.md#where-it-sits-and-how-one-request-travels). Each hop has a distinct failure mode: | Hop | If it fails | | ------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | | Browser → gateway | Nothing OpenIAM can see. `x-openiam-login-uri` absent from the response means the gateway is not in the path at all | | Gateway → OpenIAM, "may this user have this URI" | **Service Not Available** on every URL, with nothing in the OpenIAM log, because the call never arrived | | OpenIAM's answer | A deny becomes the `401` error page; "authenticate first" becomes a redirect | | Gateway → browser, the sign-in redirect | A redirect loop when the error and login paths are themselves protected; a redirect to `/webconsole/setup` when no content provider matched the host at all | | Browser → UI tier, authentication | A loop when the cookie names disagree at the two ends | | Gateway → OpenIAM, "what should I inject" | Injection silently absent; the application sees an anonymous request | | Gateway → application | The application receives the request but not the identity | | Gateway → browser, the response | Missing security headers, or absolute URLs that leave the gateway | ## Diagnosing a request that fails through the gateway Almost every gateway investigation needs the same evidence, in the same order. Collecting it up front is faster than one round trip per question, and three of the four steps cost nothing at run time. {% stepper %} {% step %} ### Ask for the proxy's debug output as one block Enabling the module's debug directives is a proxy-host change and a restart — see the [Installation Guide](/installation/installation.md). Ask for them together, ask for one reproduction of the failure, and ask for them to be turned off again afterwards. The names to quote are `OPENIAM_Verbose`, `OPENIAM_DebugESB`, `OPENIAM_DebugCookies`, `OPENIAM_DebugPatterns`, `OPENIAM_DebugRespHeaders`, `OPENIAM_PrintTimings`, `OPENIAM_DumpRequests`, `OPENIAM_DumpResponses` and `OPENIAM_AllowDumpHeaders`, with the log level at `debug`. Ask for `OPENIAM_DebugCertAuth` only when the question is specifically about certificate authentication. In a container deployment the equivalents are `OPENIAM_RPROXY_VERBOSE`, `OPENIAM_RPROXY_DEBUG_ESB` and their siblings. {% hint style="danger" %} **Do not ask for the body dumps unless the question is about payload content.** `OPENIAM_DumpRequestsBody` and `OPENIAM_DumpResponsesBody` — and `OPENIAM_RPROXY_DUMP_ALL` in a container, which **ships as `1`** — write request and response bodies to the log, which means credentials, tokens and personal data in whatever aggregates that log. Confine them to a single reproduction on a non-production host, and treat any log captured with them on as a secret. {% endhint %} Collect alongside the proxy log: a HAR capture of the failing navigation, the UI tier's log at `DEBUG` for the request, and the OpenIAM ESB log for the same seconds. A HAR on its own rarely settles anything, because the interesting decision happens between the proxy and OpenIAM where the browser cannot see it. {% endstep %} {% step %} ### Read the four headers the gateway sends the UI The gateway tells the UI what it decided in four request headers — `x-openiam-cp-id`, `x-openiam-uri-pattern-id`, `x-openiam-uri-method-id` and `x-openiam-auth-level-ids`, all listed in [Access Gateway reference](/administration-guide/gateway/access-gateway-reference.md#the-headers-the-gateway-sends-the-ui). Read them off a HAR capture or a request dump. They answer the first question of any gateway investigation — *did the request match the pattern I think it matched* — without reading a single log line. The UI logs the same conclusion at `DEBUG`: {% code overflow="wrap" %} ``` DEBUG o.o.ui.web.filter.ContentProviderFilter : URI Pattern ID: , Content Provider ID: , is From Proxy: true. User ID: null ``` {% endcode %} `is From Proxy: false` on a request you expected to come through the gateway means the proxy is not in the path at all — start there rather than with the pattern. From the Webconsole, the same question is answered by the **Federate URI Debug Tool** at `/webconsole-am/federateURIDebug.html`, and "which identity would this URL be given" by the **Principal Debug Tool** at `/webconsole-am/proxyURIPrincipalDebug.html`. {% endstep %} {% step %} ### Translate the status code back into an authorization decision `401` is `UNAUTHORIZED`, `404` is `NOT_FOUND` and `500` is `INVALID_URI` — the full mapping is in [Access Gateway reference](/administration-guide/gateway/access-gateway-reference.md#status-codes-and-authorization-decisions). `404` is the one that misleads. It does not mean the application returned `404`; it means authorization never got as far as asking the application, because nothing in the content provider claimed the path. A long URL, a URL carrying an unencoded nested URL, or a path a little different from the pattern you wrote all produce it. Apache's own request-line and header-size limits sit in front of all of this, so a genuinely enormous URL can be refused by the web server before the module sees it — a host setting, not an OpenIAM one. A fourth possibility is none of the three: the log line `Could not determine authentication level grouping for ::` means the pattern matched but no authentication level grouping resolved for it, which is a configuration gap on the pattern rather than an entitlement decision. Fix it on the pattern's authentication levels rather than on the entitlement. {% endstep %} {% step %} ### Read the one OpenIAM line that contains the whole answer Raise the ESB's URI-federation logger to `DEBUG` and every authorization call logs its inputs, its full result and its cost on one line: {% code overflow="wrap" %} ``` DEBUG o.o.e.a.s.impl.URIFederationServiceImpl : URI Fedration. Proxy URI: 'https://legacy.example.com/webconsole/rest/api/accessrights/search?from=0&size=100'. UserId: ''. Output: 'URIFederationResponse(super=Response(status=FAILURE, errorCode=UNAUTHORIZED, responseValue=/webconsole/rest/api/*), server=AbstractServer(serverURL=http://localhost:8080), proxyPassLocation=null, redirectTo=null)'. Time: 12 ms ``` {% endcode %} Grep for `URI Fedration` — the typo is the product's, and there are two spaces after the full stop. Read it in this order: `responseValue` is the pattern that matched, `errorCode` is why, `server` is where the request would have gone, `redirectTo` is set when the pattern carries a redirect, and `Time` is OpenIAM's own service time for the decision. An empty `UserId` with `errorCode=UNAUTHORIZED` is the ordinary first-hit-before-login case; the same line repeating for a user who *is* signed in is the finding. `Time` is also how you decide whether a slow page is the gateway's fault at all. Single-digit-millisecond decision times with a page that takes tens of seconds means the cost is in the number of calls the page makes, or in the tier behind the gateway — not in authorization. Compare a slow context with a fast one on the same gateway and count the authorized requests in the HAR before tuning anything. {% endstep %} {% endstepper %} ## Symptom, cause, fix | Symptom | Cause | Fix | | ------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Service Not Available** on every URL, nothing in the OpenIAM log | The proxy cannot reach the ESB — a wrong ESB path, or a host security policy blocking outbound connections from the web server | Proxy-host configuration — [Installation Guide](/installation/installation.md). Nothing in OpenIAM is involved yet | | Redirected to `/webconsole/setup` forever | No content provider matches the request's host | Create the content provider; its **Domain Pattern** must equal the `Host` header the browser sends, not a container name | | Rule saved but behaviour unchanged | The cache has not swept | Wait out `org.openiam.am.uri.federation.threadsweep` (30 s) and `org.openiam.content.provider.threadsweep` (300 s) | | Login loop: `/idp/login` redirects to itself | The cookie name the proxy reads and the content provider's **Authentication Cookie Name** disagree | Make both `OPENIAM_AUTH_TOKEN`, or both the same override | | Login loop after protecting `/idp/*` | `/idp/401`, `/idp/404` or the actuator health patterns are not public | Run **Add Default Patterns** on the content provider | | Redirects and email links point at an internal container name | `x-openiam-proxy-host` stripped between the gateway and the UI | Restore the header at the load balancer; do not compensate by rewriting `Host` | | Websocket features — push, live notifications — dead | The socket path is authenticated, or its negotiation endpoint is not | The socket and its `/info` endpoint need opposite treatment; a proxy configuration fix — [Installation Guide](/installation/installation.md) | | Injected header arrives twice, or with a browser value | No **Exclude Headers** entry for that header name | Add one naming the same header on the same pattern — [Give the application the authenticated identity](/administration-guide/gateway/inject-identity.md) | | Form fill posts an empty password | No recoverable password for that principal on the authentication provider's managed system | Check the user's **Identities**; otherwise supply the value from a **Groovy Script** — [Form fill](/administration-guide/gateway/inject-identity.md#form-fill) | | Kerberos falls back to the login form | A keytab or SPNEGO problem, not an OpenIAM one | [Kerberos and desktop SSO](/administration-guide/gateway/kerberos.md) | | Passwords visible in the log aggregator | `OPENIAM_RPROXY_DUMP_ALL=1`, the shipped starter default | Ask for it and the other four debug variables to be set to `0`, and the log level to `warn` | | A directive appears to be ignored | Three shipped names carry typos — `OPENIAM_CSPReportEnalbed`, `OPENIAM_AutnHeaderPassword`, `OPENIAM_WorkerPooEnabled` | Use the misspelled names exactly as written | | Internal Server Error plus `AH01144: No protocol handler was valid for the URL /app/ (scheme 'https')` | The pattern's application server is `https://` but the web server has no module willing to speak HTTPS upstream | [Proxying to an HTTPS upstream](#proxying-to-an-https-upstream) | | `404` on one URL while the rest of the application works | No URI pattern matched that URL — `NOT_FOUND` from OpenIAM, not a `404` from the application | Widen or add the pattern. Long URLs and URLs carrying an unencoded nested URL are the usual causes | | `500` and `URI Syntax Exception:` in the OpenIAM log | The proxied URL is not a legal URI, so the call returned `INVALID_URI` | Percent-encode the offending parameter value at the application that generates the link | | Authentication succeeds, then the browser lands on a `404` under the OpenIAM context | Nothing supplied a postback URL, so the IdP used its `/selfservice/` fallback | [Where the user lands after authenticating](/administration-guide/gateway/request-and-response-transformations.md#where-the-user-lands-after-authenticating) | | `401` with `not valid - doesn't start with a '/' - XSS detected. Returning 401` | The postback URL failed OpenIAM's validation | Make it a relative path and percent-encode nested URLs — [Postback URL validation](/administration-guide/gateway/access-gateway-reference.md#postback-url-validation) | | **Change password** in SelfService lands on the default page, or drops from HTTPS to HTTP | `org.openiam.ui.password.change.url` holds a full URL instead of a relative path | Restore the relative path; shipped value resolves to `/idp/changePassword` | | Certificate sign-in fails, log ends `Could not parse certificate: java.io.IOException: Empty input` | The gateway posted an **empty** certificate body — the certificate never reached the module | [Certificate authentication fails](#certificate-authentication-fails-and-the-login-page-says-nothing) | | Certificate sign-in fails and the login page shows nothing at all | By design the module denies, sets no cookie, and returns to the login page | [Certificate authentication fails](#certificate-authentication-fails-and-the-login-page-says-nothing) | | `FATAL ERROR. No Authentiation Provider was found …` repeating, surviving a restart | The cached authentication providers are empty and no default policy provider is flagged | ["No Authentiation Provider was found"](#no-authentiation-provider-was-found-what-actually-broke) | | A mobile or native client's session expires while idle | Renewal is a sliding window driven by requests; a client that makes none is not renewed | [Session lifetime and renewal](/administration-guide/gateway/access-gateway-reference.md#session-lifetime-and-renewing-the-session-cookie) | | A step-up rule was attached to a path and nothing changed | The rule went on a pattern the request does not match, or the pattern's rule list is empty so the content provider's rules are in force | Check the matched pattern with the **Federate URI Debug Tool**; an empty list reads *"The authentication grouping from the parent content provider will be used"* | | Adding an authentication rule changed authentication for the whole site | The rule was attached to the content provider under **Supported Authentication Rules**, not to one pattern | Remove it there and attach it to the specific URI pattern under **Supported Authentication Levels** | | A path you believed protected lets anyone in | The pattern carries **Def. None Auth Rule**, or **Is Authorization Deactivated** is ticked on it | Both are per pattern; read them back on the pattern the URL actually resolves to | | `/rabbitmq` or `/grafana` authorizes and then fails to load | The upstream URL for that prefix is unset or wrong, and neither is visible on any Webconsole screen | Ask the deployment owner for `rabbitmq.full.url` / `grafana.full.url`; the Grafana value has no shipped default | | `/mcp` answers 502, 503 or nginx's 404 page although its patterns exist | The nginx in front of the UI applications does not forward `/mcp` to the MCP server | Use the 2026.10.2 nginx image, or add its locations — see [Connect AI assistants through the MCP server](/administration-guide/federation/mcp-server.md#publish-it-through-the-access-gateway) | | `/mcp` or `/.well-known/oauth-protected-resource` returns the gateway's 401 page, or 404, on a deployment upgraded to 2026.10.2 | The content provider was seeded before these patterns existed; the seed is not re-applied on upgrade | Press **Create default URI patterns** on the content provider, or add the patterns by hand — see [the seeded groups](/administration-guide/gateway/access-gateway-reference.md#what-add-default-patterns-seeds). The MCP server's own failures are in [Connect AI assistants through the MCP server](/administration-guide/federation/mcp-server.md#when-it-goes-wrong) | ## Proxying to an HTTPS upstream A URI pattern whose **Application Servers** entry begins `https://` needs the web server's SSL proxy support switched on. In the shipped proxy configuration that switch is conditional on the SSL module being loaded, and a web server that has not loaded it skips the whole block silently — no complaint at startup, and the setting is simply never applied. The symptom is an Internal Server Error in the browser and this in the gateway error log: {% code overflow="wrap" %} ``` [proxy:warn] [pid 17918:tid 17925] [client 203.0.113.10:57660] AH01144: No protocol handler was valid for the URL /app/ (scheme 'https'). If you are using a DSO version of mod_proxy, make sure the proxy submodules are included in the configuration using LoadModule. ``` {% endcode %} `AH01144` is the web server's message, not OpenIAM's, and it is emitted late: the request was matched, the upstream was resolved from the URI pattern, and only then was no loaded module willing to speak `https` to it. **So the content provider and the pattern are working — the module list is not.** That is the finding to hand over: ask the deployment owner to confirm that the SSL, proxy and proxy-HTTP modules are all loaded alongside the OpenIAM module, and to load the SSL module if it is missing. Loading it is proxy-host work — [Installation Guide](/installation/installation.md) — and in a container deployment the image renders the configuration, so the question has to be answered inside the running container. ## Certificate authentication fails and the login page says nothing Certificate sign-in through the gateway takes a second route to OpenIAM: the module posts the certificate to the ESB's certificate-identity endpoint, and any failure there is logged as `Cannot get identity from cert`. OpenIAM rejects in a fixed order, and the **first** response code you see is the earliest thing wrong — the ordered table is in [Access Gateway reference](/administration-guide/gateway/access-gateway-reference.md#status-codes-and-authorization-decisions). The last row of that table is the one that looks like a bad certificate and usually is not. An **empty** certificate body produces `java.io.IOException: Empty input`, so a log line ending {% code overflow="wrap" %} ``` Could not parse certificate: java.io.IOException: Empty input ``` {% endcode %} means the gateway posted an **empty** certificate: the certificate never reached the module, rather than reaching it and being rejected. Check that the TLS layer is actually requesting one — `OPENIAM_SSL_VERIFY_CLIENT` in a container deployment — and where the certificate is expected in a header instead of the handshake, that `OPENIAM_ReadClientCertFromHeader` is `on` and the header named by `OPENIAM_ClientCertHeaderName` (default `x-openiam-client-cert`) is present *and non-empty* on the request that fails. An intermediary that forwards the header name with an empty value produces exactly this. {% hint style="info" %} **Unverified field guidance on `can't allocate tokens`.** A module-side error of the form `[openiam:error] [pid 12271] can't allocate 105 tokens` is reported in the field during certificate authentication. The count varies with the payload. It comes from inside the compiled module, so its exact trigger cannot be established from anything you can inspect; the observed pattern is that the module could not parse OpenIAM's response to one of the calls above. Treat it as a symptom, not a cause: have the ESB and certificate debug output enabled, find the response that immediately precedes it, and work the response-code table. Reproducing it with a header-supplied certificate as well as a handshake certificate is worth doing before escalating. {% endhint %} **Why the login page stays silent.** When OpenIAM declines a certificate, the module denies the request, issues no session cookie, and sends the browser back to the login page — and the login page has no way to know why unless it is told. The mechanism for telling it exists: the login URL accepts a `loginErrorFromProxy=` parameter, which the login screen renders as an inline message, and the server-rendered path additionally requires `openiamPrincipal` on the request. Its limits are worth knowing before you plan around it: exactly one code, `RESET_TOKEN_LINK_EXPIRED`, maps to a specific translation, and everything else falls back to the generic message key `openiam.idp.authentication.error`. As shipped, the parameter is produced by the UI rather than by the module, and **whether your gateway build appends it on a certificate failure cannot be confirmed from anything on this side** — test it on your own release. So: expect a generic message even when it works, keep the specific cause a log-side answer, and if the end-user experience matters, put a customer-facing explanation on your own customised login page rather than relying on a code-to-message mapping that covers one code. ## "No Authentiation Provider was found": what actually broke {% code overflow="wrap" %} ``` java.lang.IllegalStateException: FATAL ERROR. No Authentiation Provider was found for the current HTTP Request. There is also no any default providers in DB also. This is not a valid scenario. Seems DB or communcations to the ESB is corrupted. ``` {% endcode %} Two typos, both the product's — `Authentiation`, `communcations` — so grep for a fragment, not the sentence. OpenIAM raises it after four attempts to resolve an authentication provider for the request have all returned nothing: the matched URI pattern's provider, the provider on the federation context, the content provider's provider, and finally the cached **default** provider. The message blames the database, and that is the wrong place to look first. The default provider is read from a cache rather than from the database, and that cache is populated by a sweep that only enters a provider when it is both marked default **and** of type `OPENIAM_POLICY`. Three consequences: * **A default SAML, OAuth or social provider does not count.** Exactly one `OPENIAM_POLICY` provider must carry the default flag, or the last fallback is empty by design. * **A failed sweep empties the cache rather than leaving it stale.** The sweep fetches every provider from the ESB over the message bus; if that comes back empty — ESB down, RabbitMQ down — nothing is written, but the eviction pass still runs, and with an empty input it deletes every key it finds, including the default one. So the cache is worse after a failed sweep than before it. **There is no screen that shows this cache**, so the way you confirm it is the error stopping on its own once the ESB answers again. * **Restarting the UI does not fix it.** The next sweep repopulates the cache only once the ESB answers again, which is why the error survives a restart and then stops on its own. The sweep interval is `org.openiam.auth.provider.sweep.time`, `60000` ms. So read it as *"this UI instance currently has no authentication provider cached, and no default to fall back on"*. Check ESB and RabbitMQ health for the window in which it started, confirm one `OPENIAM_POLICY` provider is flagged default, and only then consider the database. Two things commonly make it loud rather than rare. A health check that reaches the UI on a path with no content provider match — an actuator path requested directly on the container rather than through the gateway — has nothing to resolve a provider from, so it takes the same path and can fill the log at the probe interval; point liveness and readiness probes at the seeded actuator patterns instead. And a *customised* login page can turn the same failure into an unrelated-looking template error: the content provider's override-stylesheet attribute is set only when the matched pattern's content provider actually defines one, so on a request where no content provider matched it is absent. The shipped templates guard on it being non-empty before using it, and a custom page must do the same. A custom page that uses it unconditionally fails on precisely the requests where no content provider matched — the same set of requests that produce the message above — so the template error you are looking at is a consequence, not the cause. ## Behind a load balancer Standing up the load balancer and writing its configuration are installation work — see the [Installation Guide](/installation/installation.md). Four things about it are yours, and all four are places a working balancer still breaks OpenIAM. **Both ends have to name the same address.** The proxy's own backend setting decides where the gateway sends the request; the content provider's **Application Servers** entry decides where OpenIAM believes the application lives. When they disagree the browser is redirected to an address it cannot reach. The content-provider half is a Webconsole change, so allow the five-minute cache sweep before concluding it did not work — and change both halves in the same window, because between them the configuration is wrong either way. **The health check has to be the actuator health path, not a TCP check.** `/idp/actuator/health` covers the UI tier, and the ESB has its own equivalent — the [Installation Guide](/installation/installation.md) has the address, and the balancer configuration belongs to the deployment owner in any case. Both answer without authentication, and the UI-side actuator paths are seeded as anonymous patterns by **Add Default Patterns**, so the gateway does not require a session for them. A bare TCP check routes traffic to an ESB that is listening but not ready, which presents as intermittent `503`s that clear on retry. **The balancer must forward `x-openiam-proxy-host` and `x-openiam-proxy-scheme` unchanged.** A balancer that strips headers it does not recognise takes both out, and the symptom is redirects and links in email pointing at an internal container name. This is the single commonest way a correct balancer configuration produces an incorrect OpenIAM. **Point liveness and readiness probes at the seeded actuator patterns.** A probe that reaches the UI on a path no content provider matches has nothing to resolve an authentication provider from, and fills the log at the probe interval — see the section above. ## Recovering from a lockout Deleting a content provider or a URI pattern removes authorization for everything under it, and the change takes several minutes to propagate — so a bad change and its rollback each have a five-minute blast radius. Capture the current content provider and its patterns before editing, per [Change configuration safely](/administration-guide/start/change-safely.md). If you have locked yourself out of the Webconsole entirely, the way back is the bootstrap path to `/webconsole/setup`, which is a proxy-side setting: in a container deployment `OPENIAM_DISABLE_CONFIGURE` set back to `0`, and on a host deployment the bootstrap directives restored and the proxy restarted. Both are the deployment owner's change — [Installation Guide](/installation/installation.md). Without a break-glass administrator whose route does not depend on the pattern you changed, this is where a gateway mistake becomes an outage, which is why the break-glass route is worth testing before the first change rather than after the first mistake. ## Related * [How the Access Gateway works](/administration-guide/gateway/access-gateway.md) — the request path these hops belong to * [Access Gateway reference](/administration-guide/gateway/access-gateway-reference.md) — the status codes, headers, properties and directive names used above * [Error and symptom index](/administration-guide/error-and-symptom-index.md) — every log line in the guide, indexed * [Diagnose a failed operation](/administration-guide/operations/diagnose-a-failed-operation.md) — reading the gateway, nginx and OpenIAM logs when a request does not arrive * [Which log answers which question](/administration-guide/operations/which-log-to-read.md) *Applies to OpenIAM 2026.8.1. Last reviewed 17 October 2026.* --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://docs.openiam.com/administration-guide/gateway/troubleshoot-the-gateway.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.