> For the complete documentation index, see [llms.txt](https://docs.openiam.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.openiam.com/troubleshooting/logs-and-diagnostics/audit-doc-timestamp.md). # Audit document timestamp issue This section describes the resolution for audit document timestamp issue. | Symptom | Likely cause | Resolution | | ------------------------------------------------------------------------------------------ | ---------------------------------------------------------------- | ---------------------------------- | | Audit documents hold `lastIndexDateTime` as an epoch number rather than an ISO 8601 string | The documents were indexed before the field's format was settled | Export, convert, re-import — below | {% hint style="info" %} **Two different timestamp problems, and the tell is the shape of the wrongness.** This page is for a value stored in the wrong *format* — an epoch number where an ISO 8601 string belongs. If instead the timestamps are readable but **out by a whole number of hours**, that is timezone interpretation, not format, and there are three separate settings that cause it — see [Timestamps off by a fixed offset](/administration-guide/operations/timestamps-off-by-a-fixed-offset.md). {% endhint %} The conversion is a full export and re-import of the `auditlog` index, so it is an outage rather than a maintenance task. Read the warning below before starting. {% hint style="info" %} **Before starting** ensure to do a backup of the server if possible and if not possible to back up the whole server, do a backup for Elasticsearch volume data. It is recommended to perform during a day off, because some audit events may be lost while converting documents timestamp formats. For \~100K of documents the whole process will take \~1h (timing depends on amount of data), hence it is worth stopping the `ui/proxy/` or redirect users to 'Under construction' page, so they can't login and perform actions. {% endhint %} {% hint style="warning" %} **You need `elasticdump` and a way to run it against the search tier.** OpenIAM used to ship it as a container in the Docker bundle; with Docker retired, install the tool yourself and point it at the OpenSearch endpoint. The conversion script below is independent of how the export and import are run. {% endhint %} ## Prerequisite Make sure to install **Python3** if not yet installed. If missing use the following command to install it. {% tabs %} {% tab title="RHEL, Rocky, CentOS" %} ```bash yum install python3-pip pip3 install python-dateutil ``` {% endtab %} {% tab title="Debian, Ubuntu" %} ```bash apt install python3-pip pip3 install python-dateutil ``` {% endtab %} {% endtabs %} {% hint style="info" %} The supported platform is Enterprise Linux, so `yum` is the usual one here — the `apt` form is for a container host running Ubuntu. {% endhint %} ## Fixing the issue {% stepper %} {% step %} **Export the `auditlog` index to a file** with `elasticdump`, against the search tier. Wait for it to finish and read its output to confirm no errors occurred — the export is the only copy of the data you are about to delete. A good run looks like this: {% code expandable="true" %} ``` Wed, 06 Nov 2024 11:10:55 GMT | sent 64 objects to destination file, wrote 64 Wed, 06 Nov 2024 11:10:55 GMT | got 0 objects from source elasticsearch (offset: 84364) Wed, 06 Nov 2024 11:10:55 GMT | Total Writes: 84364 Wed, 06 Nov 2024 11:10:55 GMT | dump complete Going to output what's in the target directory: /data/openiam/conf/elasticdump total 172712 drwxr-xr-x 2 root root 4096 Nov 6 10:56 . drwxr-xr-x 3 root root 4096 Nov 6 10:56 .. -rw-r--r-- 1 root root 176843787 Nov 6 11:10 auditlog.json ``` {% endcode %} Here 84,364 documents were exported. Check the count the index itself holds: {% code overflow="wrap" expandable="true" %} ``` curl -X GET "localhost:9200/auditlog/_count?pretty" { "count" : 84364, "_shards" : { "total" : 1, "successful" : 1, "skipped" : 0, "failed" : 0 } } ``` {% endcode %} **The two counts must match** — the number `elasticdump` wrote and the number the index holds. Every audit document is now in the exported file. {% endstep %} {% step %} **Convert the timestamps.** Create `auditLogDateConversion.py` from [the listing below](#the-contents-of-python-script), put the exported `auditlog.json` beside it, and run it to convert the epoch values to ISO 8601. No errors should appear. When it finishes, open `converted_file.json` and confirm its line count matches the number of documents exported — 84,364 in this example. {% endstep %} {% step %} **Delete the index.** {% code overflow="wrap" %} ``` curl -X DELETE "localhost:9200/auditlog/" ``` {% endcode %} Restart the ESB service, which recreates the index empty. Wait for it to come up before importing. {% endstep %} {% step %} **Import the converted file** back into the empty index, again with `elasticdump`. Its output should account for every document: {% code expandable="true" %} ``` Wed, 06 Nov 2024 13:13:04 GMT | got 100 objects from source file (offset: 84200) Wed, 06 Nov 2024 13:13:04 GMT | sent 100 objects to destination elasticsearch, wrote 100 Wed, 06 Nov 2024 13:13:04 GMT | got 64 objects from source file (offset: 84300) Wed, 06 Nov 2024 13:13:04 GMT | sent 64 objects to destination elasticsearch, wrote 64 Wed, 06 Nov 2024 13:13:04 GMT | got 0 objects from source file (offset: 84364) Wed, 06 Nov 2024 13:13:04 GMT | Total Writes: 84364 Wed, 06 Nov 2024 13:13:04 GMT | dump complete Going to output what's in the target directory: /data/openiam/conf/elasticdump total 179720 drwxr-xr-x 2 root root 4096 Nov 6 10:56 . drwxr-xr-x 3 root root 4096 Nov 6 12:59 .. -rw-r--r-- 1 root root 184021746 Nov 6 12:57 auditlog.json ``` {% endcode %} The number written must match the number exported, and the index count afterwards should equal it too. {% endstep %} {% endstepper %} ### The contents of Python script `auditLogDateConversion.py` {% code overflow="wrap" expandable="true" %} ``` import json from datetime import datetime, timezone from dateutil import parser # Import the parser from dateutil to handle ISO 8601 strings # Function to convert epoch to ISO 8601 datetime format with milliseconds and 'Z' suffix def convert_to_datetime(value): # If the value is already in ISO 8601 format (check for 'Z' at the end for UTC) if isinstance(value, str) and value.endswith('Z'): try: # Use dateutil.parser.isoparse to handle ISO 8601 format for all Python versions return parser.isoparse(value).strftime('%Y-%m-%dT%H:%M:%S.') + f"{parser.isoparse(value).microsecond // 1000:03d}Z" except ValueError: pass # If it doesn't match ISO format, proceed to try epoch conversion # If value is numeric or a string that can be cast to a number (epoch) try: # If the value is a valid integer (or can be converted to int), it's epoch value = int(value) # Convert the epoch to a datetime object dt = datetime.fromtimestamp(value / 1000, tz=timezone.utc) # Format it to include milliseconds and append 'Z' for UTC return dt.strftime('%Y-%m-%dT%H:%M:%S.') + f"{dt.microsecond // 1000:03d}Z" except (ValueError, TypeError): # If it's neither an epoch nor a valid ISO datetime, return None print(f"Invalid datetime value '{value}' encountered.") return None # Recursive function to traverse nested structures and convert 'lastIndexDateTime' in epoch format def convert_nested_data(data): if isinstance(data, dict): # If it's a dictionary, check each key for key, value in data.items(): if key == 'lastIndexDateTime' or key == 'timestamp': # Check for the specific keys converted_value = convert_to_datetime(value) if converted_value: data[key] = converted_value else: # Recursively call the function for nested objects convert_nested_data(value) elif isinstance(data, list): # If it's a list, check each item for item in data: convert_nested_data(item) # Define input and output file paths input_file_path = "auditlog.json" # Replace with the path to your input file output_file_path = "converted_file.json" # The path for the output file # Open the input file and process each line with open(input_file_path, 'r') as infile, open(output_file_path, 'w') as outfile: for line in infile: # Parse the JSON line record = json.loads(line) # Access '_source' data where the target fields are located source_data = record.get('_source', {}) # Convert top-level fields 'timestamp' and 'lastIndexDateTime' if 'timestamp' in source_data: converted_timestamp = convert_to_datetime(source_data['timestamp']) if converted_timestamp: source_data['timestamp'] = converted_timestamp else: print(f"Error converting 'timestamp' for record ID: {record.get('_id')}") if 'lastIndexDateTime' in source_data: converted_last_index = convert_to_datetime(source_data['lastIndexDateTime']) if converted_last_index: source_data['lastIndexDateTime'] = converted_last_index else: print(f"Error converting 'lastIndexDateTime' for record ID: {record.get('_id')}") # Convert nested fields like 'attributes' or 'targetUsers' convert_nested_data(source_data) # Write the modified record to the output file as a single line of JSON outfile.write(json.dumps(record) + '\n') ``` {% endcode %} --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://docs.openiam.com/troubleshooting/logs-and-diagnostics/audit-doc-timestamp.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.