Table of contents
No headings in the article.
In this post, I am going to explain the step I followed to deploy PostgreSQL running in docker containers to AWS EKS. The YAML files I used for this deployment can be found in my GitHub repository:
https://github.com/henryeleonu/spark-kubernetes/tree/jupyter-spark-kube
In this YouTube video, I demoed the deployment of Apache Spark and PostgreSQL on EKS:
Before your deploy PostgreSQL, you must deploy the EKS cluster on AWS. Click this link to see how we deploy EKS cluster.
Create a Storage Volume
As Docker containers are ephemeral. All the data which is generated by or in the container will be lost after the termination of the container instance. To save the data, we will be using Persistent volumes and persistent volume claim resources within Kubernetes to store the data on persistent storages. We use the manifest below to specify persistent volumes and persistent volume claim resources. Note that since we are deploying this database container to AWS EKS, the instance storage in the EC2 will be used. This means on termination of the EC2, your data will be lost. For a production database, the best approach is to use storage external to the EC2 instance on which the database pod is running such as Elastic Block Store (EBS) volume or Elastic File System (EFS).
To create a persistent volume and persistent volume you run the command on the terminal:
kubectl create -f postgres-storage.yaml
OR
kubectl apply -f postgres-storage.yaml
To check the Persistent Volume, run the command:
kubectl get pv
To check the Persistent Volume Claim, run the command:
kubectl get pvc
Create Secret
A Secret is an object that contains a small amount of sensitive data such as a password, a token, or a key. By using secret, you don't need to include confidential data in your application code. Secrets are created independently of the Pods that use them, therefore, there is less risk of confidential data being exposed during the workflow of creating and viewing and editing Pods. Kubernetes, and applications that run in your cluster, can also take additional precautions with Secrets, such as avoiding writing secret data to nonvolatile storage. I used base64 encoding to encode the values for POSTGRES_DB, POSTGRES_USER AND POSTGRES_PASSWORD in the secret manifest. Below is the secret manifest we used.
To create the secret, you run the command on your terminal:
kubectl apply -f postgres-login-secret.yaml
Creating PostgreSQL Deployment
In the deployment manifest below, the pod is specified under template, which has the image of postgres:14-alpine. It has the information from the secret associated to the pod and also mounts the volume created from the persistent volumes and claims. Set the replica of the deployment to 1. If you need to run multiple pods of the database in the Kubernetes cluster, then do not use a deployment for it because it will not be able to synchronize the data in the multiple databases which will compromise the integrity of data. If you need multiple databases, like one master and one or more read replicas, then use a StatefulSet. A manifest of the kind: StatefulSet will ensure the synchronization of read replicas with the master pod.
To create Postgres deployment, run the command on your terminal:
kubectl create -f postgres-deployment.yaml
To get the list of Kubernetes deployments run the command:
kubectl get deployments
Create PostgreSQL Service
To access the deployment or container, we need to expose PostgreSQL service. Kubernetes provides different types of services like ClusterIP, NodePort and LoadBalancer. With ClusterIP we can access the PostgreSQL service within a Kubernetes cluster. NodePort gives the ability to expose a service endpoint on the Kubernetes node, which is the EC2 instance in the case of AWS. For accessing PostgreSQL externally, you need to use a Load Balancer service type which exposes the service externally. Apart from service, you can also use Ingress to expose a service externally.
In our case, we don't need to expose the database externally and it is a good practice never to do this because the database should not be access directly but through a front end. Therefore the manifest below which we used specifies a service of type ClusterIP.
To create Postgres Service, run the command in the terminal:
kubectl create -f postgres-service.yaml
To verify the Kubernetes service, run the command:
kubectl get svc
Connecting to PostgreSQL via kubectl Command
To get into the PostgreSQL pod, run the kubectl exec command below. Be sure to change the pod name, the username and the database name.
kubectl exec -it <name of postres pod> --psql -h localhost -U <database username> --password -p <postgres port number> <database name>
kubectl exec -it postgres-574d8d5f-2488v --psql -h localhost -U postgres --password -p 5432 postgres
Delete PostgreSQL Deployments
For the deletion of PostgreSQL resources, we need to use the below commands.
kubectl delete -f postgres-deployment.yaml
kubectl delete -f postgres-login-secret.yaml
kubectl delete -f postgres-service.yaml
kubectl delete -f postgres-storage.yaml