Recommended Storage Solutions and Data Reclamation Strategies for NATS JetStream in a Red Hat OpenShift Environment (on premise)

[mohamedsaleem18]

We are currently using a 3-node NATS cluster (container storage) on Red Hat OpenShift (on premise). Since NFS is not recommended as external storage for NATS, what are the recommended storage options for a NATS cluster (JetStream) in production ?

If a StatefulSet of the NATS cluster is deleted due to an unknown scenario, it also deletes the PVC. When the StatefulSet is re-provisioned, the PV will not be reclaimed by the new PVC, resulting in loss of JetStream data. Can you provide a solution for automatically reclaiming data from the existing PV when the NATS cluster (StatefulSet) is brought up again?

[wallyqs]

(Moving this to nats-io/k8s) I think deletion of the statefulset should not delete the pvc so would confirm the setting being used for persistentVolumeReclaimPolicy first.

[mohamedsaleem18]

persistentVolumeReclaimPolicy as Retain will not delete the data on PV incase PVC is deleted but when PVC is recreated, same PV with data cannot be mounted automatically. It requires manual intervention.

We tested statefulset with PV and PVC. Deleting statefulset, deletes PVC as well.

[wallyqs]

in my experience deleting a statefulset does not delete the volumes and they have to be deleted explicitly but not sure if this is different in openshift...

[mohamedsaleem18]

PV (volume) will not be deleted but PVC will be deleted. New PVC to claim existing PV requires manual intervention. Automatically PVC will claim new PVs.

[mohamedsaleem18]

In addition that, external storage options for NATS, NFS and local storage are ruled out. Can you please let me know the other storage options available with Kubernetes for production deployment (on-premise)? How to ensure that data are not lost in case statefulset is deleted due to any reasons?