Restoring Postgres databases in Kubernetes

Expected use case: Restoring a previous snapshot for disaster recovery

For restoring Postgres databases in our Kubernetes deployments you should have backups of both the primary db and the codeintel-db. The primary db contains all user information.

To restore persistent volumes you need to

Understand how your Kubernetes cluster provides Persistent Volume (PV) Storage GKE , EKS , AKS
Scale down all pods that are using the PersistentVolumeClaim (PVC)
Backup the current PVCs
Create new PVs that reference the restored snapshots

Steps

Backup the PVCs we will be operating on

SHELL
kubectl get pvc pgsql codeintel-db codeinsights-db -oyaml > backupPVC.yaml

Ensure that you have snapshots of the necessary PVs (this depends on your deployment environment)

SHELL
kubectl get pv $(kubectl get pvc pgsql codeintel-db codeinsights-db -o=jsonpath='{.items[*].spec.volumeName}')

Restore the snapshots to new disks that meet the requirements to be consumed by your cluster (same region, etc).
Scale down all deployments to zero (note: both approaches involve downtime, the first approach is preferred)
```
SHELL
kubectl scale deployment  -l deploy=sourcegraph --replicas=0
```
or (if you are unable to scale deployments to their previous values easily)
```
SHELL
 kubectl scale deployment pgsql codeinsights-db codeintel-db  --replicas=0
```
Next, you need to replace the existing PVC and PV.
First, delete the PVC kubectl delete pvc pgsql
Then create the new PV & PVC. The options used to configure the PV will be specific to every provider.

YAML
apiVersion: v1
kind: PersistentVolume
metadata:
  name: example-pgsql-volume
spec:
  storageClassName: "sourcegraph"
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 200Gi
  claimRef:
    namespace: ns-sourcegraph
    name: pgsql
  # You will need to customize the fields below most likely
  gcePersistentDisk:
    pdName: disk-restored-from-snapshot
    fsType: ext4
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pgsql
  namespace: ns-sourcegraph
spec:
  storageClassName: "sourcegraph"
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 200Gi

Ensure PVCs become Bound kubectl get pvc -w
(Optional) Commit PVs and PVCs back to deployment repo into the base/${deployment} directory.
Scale deployments back up
If you use our deployment scripts, after committing the change you may run ./kubectl-apply-all.sh again to restore all deployments to their set number of replicas.
Otherwise, kubectl scale deployment pgsql codeinsights-db codeintel-db --replicas=1
After ensuring Sourcegraph is functional, you may delete the previous PVs. If the Reclaim Policy is not set to Delete you will need to manually delete the disk from your provider as well.