How To Run Databases in Kubernetes

Many people successfully run their databases in Kubernetes, and the number of such deployments is growing daily.

Jul 23rd, 2024 10:00am by Kolawole Olowoporoku

Featued image for: How To Run Databases in Kubernetes

Image by Gerd Altmann from Pixabay.

The debate about where databases should run in Kubernetes has been a hot topic in the tech community. The prevailing argument is about “building stateless applications,” suggesting that databases are best suited as managed services with cloud providers. However, there are practical design patterns for successfully running databases in Kubernetes. On most cloud providers, volumes are constrained to a single Availability Zone (AZ), which means the databases are also constrained to that AZ by design. Most production clusters are likely regional or multi-AZ, especially for stateless applications. Using node selectors to ensure the database pods are located in the AZs where their volumes can be mounted is important. Example:

nodeSelector:
  topology.kubernetes.io/zone: europe-west6-b

This configuration specifies that the database pod should run in the ‘europe-west6-b’ AZ.

Plan Resource Usage

Since our databases are constrained to one AZ, we must carefully plan our node-to-AZ design to avoid scheduling errors and unavailability issues. One effective strategy is to run separate node groups or node pools specifically for database workloads. This ensures that sufficient resources are always available in the required AZ. Example:

Create a dedicated node pool for database workloads.
Use taints and tolerations to ensure only database pods are scheduled on these nodes.

# Taint nodes to be dedicated to databases

spec:
&nbsp;&nbsp;taints:
&nbsp;&nbsp;- key: "dedicated"
&nbsp;&nbsp;&nbsp;&nbsp;value: "database"
&nbsp;&nbsp;&nbsp;&nbsp;effect: "NoSchedule"

# Toleration in the DB pod spec
spec:
  tolerations:
  - key: "dedicated"
    operator: "Equal"
    value: "database"
    effect: "NoSchedule"

High Availability

Managed database services often provide built-in high availability and failover capabilities. To achieve similar resilience in Kubernetes, meticulous planning for recovery and availability strategies is essential. Here are two approaches:

Using Kubernetes Operators:

Kubernetes operators like the Zalando Postgres Operator offer advanced features like read replicas and automatic failovers, similar to managed database services. These operators can significantly simplify the setup and management of high availability for your databases. The Zalando Postgres Operator allows you to specify the number of read replicas and automatically manages failovers. This operator provides a UI where you can configure these settings, making it an intuitive and powerful tool for managing database high availability in Kubernetes. Here is a list of some other Operators, some of which are managed by their respective communities

Self-Service Approach:

For those who prefer a more hands-on approach, particularly for NoSQL databases, here’s a step-by-step method:

Mount Data Volumes on Both Pods: Ensure that the data volume is accessible by both the primary and secondary pods.
Pod Affinity: Use pod affinity rules to ensure that the primary and secondary pods are placed together, respecting volume constraints.
Init Container: At startup, use an init container in the secondary pod to copy all data from the primary pod.
Volume Mount Constraint: Set the volume mount on the secondary pod to read-only to prevent data corruption.
Use cronjob for restarts: Create a simple CronJob that deletes the old pod every six hours, allowing the init container to run and copy new data.

Example Configurations

The example below shows how to set up a Neo4j read replica using pod affinity, an init container for data copying, and mount volumes with read-only constraints to ensure data integrity.

affinity:
  podAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
            - key: app
              operator: In
              values:
                - primary-db
        topologyKey: "kubernetes.io/hostname"

initContainers:
  - name: copy-data
    image: busybox
    command: ["sh", "-c", "cp -r /data/* /backup/"]
    volumeMounts:
      - name: data-volume
        mountPath: /data
      - name: backup-volume
        mountPath: /backup

volumes:
  - name: data-volume
    persistentVolumeClaim:
      claimName: primary-db-pvc
  - name: backup-volume
    persistentVolumeClaim:
      claimName: secondary-db-pvc

containers:
  - name: secondary-db
    image: neo4j:latest
    volumeMounts:
      - name: backup-volume
        mountPath: /data
        readOnly: true

Backups and Restore

Many service providers offer ways to schedule recurring snapshots on disk-based volumes. This is often the preferred method because it is easier to set up, and the recovery process is faster. We can back up the volumes hosting the DB data regularly in this scenario. Another approach involves combining the database’s proprietary tools, such as pg_dump for PostgreSQL. Here is an example Configuration for PostgreSQL using Kubernetes Cron Jobs to backup to s3

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: postgres-backup
spec:
  schedule: "0 0 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: backup
            image: postgres
            command: ["sh", "-c", "pg_dumpall -c -U $PGUSER | gzip > /backup/db_backup.gz && aws s3 cp /backup/db_backup.gz s3://your-bucket/db-backup-$(date +\%F).gz"]
            volumeMounts:
            - name: backup-volume
              mountPath: /backup
          restartPolicy: OnFailure
          volumes:
          - name: backup-volume
            emptyDir: {}

Summary

Even though the initial setup or learning curve might be steep, running your database in Kubernetes provides plenty of advantages. One not-so-talked-about benefit is cost. Running a db.m4.2xlarge (4vCPUs, 32GB RAM) instance in RDS costs approximately $1200/month, while running a similarly sized EC2 instance costs around $150/month. A node in Kubernetes will likely also run more than one pod, further optimizing resource use. Vendor agnosticism is another key motivation for many people running databases in Kubernetes. Moving your workloads across any platform with minimal tweaks is incredibly appealing. In conclusion, consider the pros and cons before deciding where to run your production database. Many people successfully run their databases in Kubernetes, and the number of such deployments is growing daily.

Kolawole Olowoporoku has over 7 years of experience managing distributed systems. Currently, he works as a Site Reliability Engineer (SRE) at SEKAI, a Swiss-based startup. Kolawole has also worked for several large organizations, including AWS. He is a passionate advocate...