Backing up and restoring the vault unseal keys

At some point during the installation process, you will be asked to save your Vault unseal keys. These are important as access to the vault backend data (Consul/ETCD) can be restored with their help. To create a backup of the unseal keys, follow the instructions below.

Backing up the unseal keys

1. Create a backup using the following command.

kubectl exec -it <vault_pod> -- cat /data/openiam/conf/vault/seal/unseal.keys

For example, the output may look as follows.

kubectl get pods | grep vault
test2021-vault-0                          1/1     Running   0              17h

kubectl exec -it test2021-vault-0 -- cat /data/openiam/conf/vault/seal/unseal.keys

{"keys":["65fa1cdffb74fd1e9d9872cd81464868d710329aca27cf8d395944048a800a33f7","1374d9698986b135356336f25688b840830b065d417bcfb2066bfe56c552fdf166",
"0e3fb6ffc3ea7130a4f0cd31fdea3e9aeca305c775ace708ba0b2789df839731a6","6d5a6cd3e9b50d4d6562fd4a5d749a87c75bfd46e4d87191b21f736a9783dea8ea","95dc001966103ad985816662e085dc459e807b0b91b83995b8e0cc4fffc4a4b604"],"keys_base64":["Zfoc3/t0/R6dmHLNgUZIaNcQMprKJ8+NOVlEBIqACjP3","E3TZaYmGsTU1YzbyVoi4QIMLBl1Be8+yBmv+VsVS/fFm","Dj+2/8PqcTCk8M0x/eo+muyjBcd1rOcIugsnid+DlzGm","bVps0+m1DU1lYv1KXXSah8db/Ubk2HGRsh9zapeD3qjq","ldwAGWYQOtmFgWZi4IXcRZ6AewuRuDmVuODMT//EpLYE"],"root_token":"s.zLyftXS2zpOMDFjOjnVf3Tyk"}


Save these! Distribute the unseal keys individually to different trusted members of your team.

Restoring the unseal keys

If the unseal keys are lost or destroyed, they can be restored using the following steps:

1. Create the Busybox pod with the below content.

vi busybox.yaml  
apiVersion: v1
kind: Pod
metadata:
 name: busybox
spec:
 containers:
 - name: busybox
   image: busybox
   command: ['sleep', '3600']   # Keeps the Pod running
   volumeMounts:
   - name: seal
     mountPath: /data         # Mount point inside the container
 volumes:
 - name: seal
   persistentVolumeClaim:
     claimName: openiam-pvc-vault-seal

2. Apply the changes with the following command.

kubectl apply -f busybox.yaml

3. Access the Busybox pod command line.

kubectl exec -it busybox -- sh

4. Restore the saved keys.

echo '<saved_unseal_keys>' > /data/unseal.keys

Example output:

echo '{"keys":["65fa1cdffb74fd1e9d9872cd81464868d710329aca27cf8d395944048a800a33f7","1374d9698986b135356336f25688b840830b065d417bcfb2066bfe56c552fdf166",
"0e3fb6ffc3ea7130a4f0cd31fdea3e9aeca305c775ace708ba0b2789df839731a6","6d5a6cd3e9b50d4d6562fd4a5d749a87c75bfd46e4d87191b21f736a9783dea8ea","95dc001966103ad985816662e085dc459e807b0b91b83995b8e0cc4fffc4a4b604"],"keys_base64":["Zfoc3/t0/R6dmHLNgUZIaNcQMprKJ8+NOVlEBIqACjP3","E3TZaYmGsTU1YzbyVoi4QIMLBl1Be8+yBmv+VsVS/fFm","Dj+2/8PqcTCk8M0x/eo+muyjBcd1rOcIugsnid+DlzGm","bVps0+m1DU1lYv1KXXSah8db/Ubk2HGRsh9zapeD3qjq","ldwAGWYQOtmFgWZi4IXcRZ6AewuRuDmVuODMT//EpLYE"],"root_token":"s.zLyftXS2zpOMDFjOjnVf3Tyk"}' > /data/unseal.keys

5. Verify that the keys are in place.

cat /data/unseal.keys

6. Exit from the Busybox container.

7. Restart services with the command below

terraform apply