Getting started with the YugabyteDB Kubernetes Operator and Portworx

Background

Over the last few years I have seen an increasing interest within the Oracle community with PostgreSQL and more recently with Yugabyte.

During a recent User Group conference I took the opportunity to learn a bit more about YugabyteDB by attending an excellent presentation by my good friend Franck Pachot.

Using my new found knowledge I thought I would try and build my own YugabyteDB environment using Oracle Kubernetes Engine (OKE), Portworx storage and the YugabyteDB Kubernetes Operator.

Kubernets Environment

Before installing the YugabyteDB Operator, lets have a quick look at my Oracle Cloud Infrastructure (OCI) Oracle Kubernetes Engine (OKE) environment which I built using a Terraform script which I have made available within GitHub.

I am using Kubernetes version 1.20.11, we can check version with kubectl version, note the client versions need to be within one minor version of the server, so you may need to upgrade / downgrade.

% kubectl version --short | awk -Fv '/Server Version: / {print $3}'
1.20.11

Number of nodes, operating system, and container runtime in the Kubernetes Cluster with kubectl get nodes.

% kubectl get nodes -o wide
NAME         STATUS   ROLES   AGE   VERSION    INTERNAL-IP   EXTERNAL-IP     OS-IMAGE                  KERNEL-VERSION                     CONTAINER-RUNTIME
10.0.1.157   Ready    node    11d   v1.20.11   10.0.1.157    132.145.33.86   Oracle Linux Server 7.8   4.14.35-1902.306.2.el7uek.x86_64   cri-o://1.20.2
10.0.1.163   Ready    node    11d   v1.20.11   10.0.1.163    144.21.51.173   Oracle Linux Server 7.8   4.14.35-1902.306.2.el7uek.x86_64   cri-o://1.20.2
10.0.1.205   Ready    node    11d   v1.20.11   10.0.1.205    132.145.69.16   Oracle Linux Server 7.8   4.14.35-1902.306.2.el7uek.x86_64   cri-o://1.20.2

We can also check the OCI regions and Availability Domains kubectl get nodes –show-labels

% kubectl get nodes --show-labels
NAME         STATUS   ROLES   AGE   VERSION    LABELS
10.0.1.157   Ready    node    11d   v1.20.11   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=VM.Standard2.1,beta.kubernetes.io/os=linux,displayName=oke-cgr3bekcbka-n5k6rizv7zq-sbgpx554ada-1,failure-domain.beta.kubernetes.io/region=uk-london-1,failure-domain.beta.kubernetes.io/zone=UK-LONDON-1-AD-3,hostname=oke-cgr3bekcbka-n5k6rizv7zq-sbgpx554ada-1,internal_addr=10.0.1.157,kubernetes.io/arch=amd64,kubernetes.io/hostname=10.0.1.157,kubernetes.io/os=linux,node-role.kubernetes.io/node=,node.info.ds_proxymux_client=true,node.info/compartment.id_prefix=ocid1.compartment.oc1,node.info/compartment.id_suffix=aaaaaaaauktztjuam57uhird7up5gxp32svboo6bmjfxvotkja75tfndc2sq,node.info/compartment.name=tf-compartment,node.info/kubeletVersion=v1.20,oci.oraclecloud.com/fault-domain=FAULT-DOMAIN-2,oke.oraclecloud.com/node.info.private_subnet=false,oke.oraclecloud.com/node.info.private_worker=true,oke.oraclecloud.com/tenant_agent.version=1.38.5-8f5c194266-712,px/enabled=true,px/metadata-node=true

10.0.1.163   Ready    node    11d   v1.20.11   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=VM.Standard2.1,beta.kubernetes.io/os=linux,displayName=oke-cgr3bekcbka-n5k6rizv7zq-sbgpx554ada-0,failure-domain.beta.kubernetes.io/region=uk-london-1,failure-domain.beta.kubernetes.io/zone=UK-LONDON-1-AD-2,hostname=oke-cgr3bekcbka-n5k6rizv7zq-sbgpx554ada-0,internal_addr=10.0.1.163,kubernetes.io/arch=amd64,kubernetes.io/hostname=10.0.1.163,kubernetes.io/os=linux,last-migration-failure=get_kubesvc_failure,node-role.kubernetes.io/node=,node.info.ds_proxymux_client=true,node.info/compartment.id_prefix=ocid1.compartment.oc1,node.info/compartment.id_suffix=aaaaaaaauktztjuam57uhird7up5gxp32svboo6bmjfxvotkja75tfndc2sq,node.info/compartment.name=tf-compartment,node.info/kubeletVersion=v1.20,oci.oraclecloud.com/fault-domain=FAULT-DOMAIN-1,oke.oraclecloud.com/node.info.private_subnet=false,oke.oraclecloud.com/node.info.private_worker=true,oke.oraclecloud.com/tenant_agent.version=1.38.5-8f5c194266-712,px/enabled=true,px/metadata-node=true

10.0.1.205   Ready    node    11d   v1.20.11   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=VM.Standard2.1,beta.kubernetes.io/os=linux,displayName=oke-cgr3bekcbka-n5k6rizv7zq-sbgpx554ada-2,failure-domain.beta.kubernetes.io/region=uk-london-1,failure-domain.beta.kubernetes.io/zone=UK-LONDON-1-AD-1,hostname=oke-cgr3bekcbka-n5k6rizv7zq-sbgpx554ada-2,internal_addr=10.0.1.205,kubernetes.io/arch=amd64,kubernetes.io/hostname=10.0.1.205,kubernetes.io/os=linux,node-role.kubernetes.io/node=,node.info.ds_proxymux_client=true,node.info/compartment.id_prefix=ocid1.compartment.oc1,node.info/compartment.id_suffix=aaaaaaaauktztjuam57uhird7up5gxp32svboo6bmjfxvotkja75tfndc2sq,node.info/compartment.name=tf-compartment,node.info/kubeletVersion=v1.20,oci.oraclecloud.com/fault-domain=FAULT-DOMAIN-3,oke.oraclecloud.com/node.info.private_subnet=false,oke.oraclecloud.com/node.info.private_worker=true,oke.oraclecloud.com/tenant_agent.version=1.38.5-8f5c194266-712,px/enabled=true,px/metadata-node=true

My three node Kubernetes cluster is spread across three Availability Domains (ADs) located with the UK-London-1 Region, as per the diagram below.

Oracle Kubernetes Engine (OKE)

Note: UK-London-region has 3 Availability Domains (ADs), but some regions are limited to a single AD, check OCI Region documentation for availability in your region.

I have already installed Portworx into my OKE cluster, we can check the version using pxctl –version

% PX_POD=$(kubectl get pods -l name=portworx -n kube-system -o jsonpath='{.items[0].metadata.name}')
  kubectl exec $PX_POD -n kube-system -- /opt/pwx/bin/pxctl --version
Defaulting container name to portworx.
Use 'kubectl describe pod/px-cluster-21a6ec62-0ed7-48b8-b6ac-b6833005b533-c6tq6 -n kube-system' to see all of the containers in this pod.
pxctl version 2.8.0.0-1ef62f8

If you are not already using Portworx and want to install the Free Forever Portworx version into your OKE environment you may want to read blog post: Portworx Essentials installation on Oracle Kubernetes Engine (OKE)

Storage Class

Let’s start by creating a Portworx Kubernetes Container Storage Interface (CSI) Storage Class, as I will be using Yugabyte replication I will use a Portworx replication of 1.

% cat px-yb-csi-sc.yaml 
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: px-yb-csi-sc
provisioner: pxd.portworx.com
parameters:
  repl: "1"
  io_profile: "auto"
allowVolumeExpansion: True

And apply with kubectl apply

% kubectl apply -f px-yb-csi-sc.yaml 
storageclass.storage.k8s.io/px-yb-csi-sc created

We can examine the settings with kubectl describe, for example, from the below you can see I have a replication factor of 1 and io_profile of auto.

% kubectl describe sc/px-yb-csi-sc
Name:            px-yb-csi-sc
IsDefaultClass:  No
Annotations:     kubectl.kubernetes.io/last-applied-configuration={"allowVolumeExpansion":true,"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{},"name":"px-yb-csi-sc"},"parameters":{"io_profile":"auto","repl":"1"},"provisioner":"pxd.portworx.com"}

Provisioner:           pxd.portworx.com
Parameters:            io_profile=auto,repl=1
AllowVolumeExpansion:  True
MountOptions:          <none>
ReclaimPolicy:         Delete
VolumeBindingMode:     Immediate
Events:                <none>

List the StorageClasses (SCs) in the cluster with kubectl get sc, the default StorageClass is marked with (default) for example:

% kubectl get sc
NAME                             PROVISIONER                       RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   
oci (default)                    oracle.com/oci                    Delete          Immediate              false                  
oci-bv                           blockvolume.csi.oraclecloud.com   Delete          WaitForFirstConsumer   false                  
px-db                            kubernetes.io/portworx-volume     Delete          Immediate              true                   
px-db-cloud-snapshot             kubernetes.io/portworx-volume     Delete          Immediate              true                   
px-db-cloud-snapshot-encrypted   kubernetes.io/portworx-volume     Delete          Immediate              true                   
px-db-encrypted                  kubernetes.io/portworx-volume     Delete          Immediate              true                   
px-db-local-snapshot             kubernetes.io/portworx-volume     Delete          Immediate              true                   
px-db-local-snapshot-encrypted   kubernetes.io/portworx-volume     Delete          Immediate              true
px-replicated                    kubernetes.io/portworx-volume     Delete          Immediate              true                  
px-replicated-encrypted          kubernetes.io/portworx-volume     Delete          Immediate              true
px-yb-csi-sc                     pxd.portworx.com                  Delete          Immediate              true                                     
stork-snapshot-sc                stork-snapshot                    Delete          Immediate              true                  

Installation

OK now let’s start by cloning the Yugabyte Operator back to our desktop.

% git clone https://github.com/yugabyte/yugabyte-operator.git
Cloning into 'yugabyte-operator'...
remote: Enumerating objects: 346, done.
remote: Counting objects: 100% (346/346), done.
remote: Compressing objects: 100% (204/204), done.
remote: Total 346 (delta 170), reused 275 (delta 121), pack-reused 0
Receiving objects: 100% (346/346), 284.64 KiB | 1.98 MiB/s, done.
Resolving deltas: 100% (170/170), done.

Deploy YugabyteDB Cluster Operator

Navigate to yugabyte-operator and register Custom Resource Definition (CRD)

% kubectl create -f deploy/crds/yugabyte.com_ybclusters_crd.yaml
customresourcedefinition.apiextensions.k8s.io/ybclusters.yugabyte.com created

Confirm ybclusters.yugabyte.com CRD has been created.

% kubectl get crd/ybclusters.yugabyte.com
NAME                      CREATED AT
ybclusters.yugabyte.com   2021-10-21T14:00:25Z

From the same directory create the YugabyteDB Operator.

% kubectl create -f deploy/operator.yaml

namespace/yb-operator created
serviceaccount/yugabyte-operator created
clusterrole.rbac.authorization.k8s.io/yugabyte-operator created
clusterrolebinding.rbac.authorization.k8s.io/yugabyte-operator created
deployment.apps/yugabyte-operator created

From the above we can see a namespace called yb-operator has been created for us.

Wait a seconds for the Operator to start, and confirm running with the below.

% kubectl -n yb-operator get po,deployment

NAME                                     READY   STATUS    RESTARTS   AGE
pod/yugabyte-operator-5dd57b9f8c-w66qr   1/1     Running   0          119s

NAME                                READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/yugabyte-operator   1/1     1            1           119s

We can now create a YugabyteDB cluster using the Operator.

You can see I have defined 3 YugabyteDB Master Servers for management of system metadata and 3 YugabyteDB Tablet Servers for the data IO.

The Operator also provide a storage section, here I specified 1 volume per Master and TServer using count, a size of 1G and my Portworx StorageClass

Below is my example manifest

apiVersion: yugabyte.com/v1alpha1
kind: YBCluster
metadata:
  name: px-ybcluster
  namespace: yb-operator
spec:
  replicationFactor: 3
  domain: cluster.local
  master:
    replicas: 3
    storage:
      count: 1
      size: 1Gi
      storageClass: px-yb-csi-sc 
  tserver:
    replicas: 3
    storage:
      count: 1
      size: 1Gi
      storageClass: px-yb-csi-sc 

Create with kubectl create -f <filename> for example.

% kubectl create -f px-yb-cluster.yaml
ybcluster.yugabyte.com/px-ybcluster created

Verify that the cluster is up and running with below command. You should see 3 pods each for YB-Master & YB-TServer.

% kubectl get po,sts,svc -n yb-operator
NAME                                     READY   STATUS    RESTARTS   AGE
pod/yb-master-0                          1/1     Running   0          8m2s
pod/yb-master-1                          1/1     Running   0          8m2s
pod/yb-master-2                          1/1     Running   0          8m2s
pod/yb-tserver-0                         1/1     Running   0          8m2s
pod/yb-tserver-1                         1/1     Running   0          8m2s
pod/yb-tserver-2                         1/1     Running   0          8m2s
pod/yugabyte-operator-5dd57b9f8c-w66qr   1/1     Running   0          52m

NAME                          READY   AGE
statefulset.apps/yb-master    3/3     8m3s
statefulset.apps/yb-tserver   3/3     8m3s

NAME                                TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                               AGE
service/yb-master-ui                ClusterIP   10.96.18.198   <none>        7000/TCP                              8m3s
service/yb-masters                  ClusterIP   None           <none>        7000/TCP,7100/TCP                     8m4s
service/yb-tservers                 ClusterIP   None           <none>        9100/TCP,9042/TCP,6379/TCP,5433/TCP   8m4s
service/yugabyte-operator-metrics   ClusterIP   10.96.87.164   <none>        8383/TCP,8686/TCP                     51m

We should also now see 6 Persistent Volume Claims (PVCs), 1 volume per Pod in the yb-operator namespace using the Portworx px-yb-csi-sc Storage Class.

% kubectl get pvc -n yb-operator
NAME                    STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
datadir0-yb-master-0    Bound    pvc-e5f2c02e-ca27-4561-aa0e-7fc15f95f34d   1Gi        RWO            px-yb-csi-sc   2m24s
datadir0-yb-master-1    Bound    pvc-c7a3c692-11fd-4f95-8d27-6db962aaaf9c   1Gi        RWO            px-yb-csi-sc   2m24s
datadir0-yb-master-2    Bound    pvc-87fb6b66-6f88-4658-bda9-ee22c6100ed2   1Gi        RWO            px-yb-csi-sc   2m24s
datadir0-yb-tserver-0   Bound    pvc-97bbaf86-7a27-4a5d-988a-22d306127b2f   1Gi        RWO            px-yb-csi-sc   2m24s
datadir0-yb-tserver-1   Bound    pvc-7accafb2-e2c1-4636-bb5f-a4d2e2550470   1Gi        RWO            px-yb-csi-sc   2m24s
datadir0-yb-tserver-2   Bound    pvc-39ddaeb9-b852-461f-a327-54ce392ee775   1Gi        RWO            px-yb-csi-sc   2m24s

We can also see the volumes from Portworx, using pxctl volume list, for example

% kubectl exec -it $PX_POD -n kube-system -- /opt/pwx/bin/pxctl volume list 
Defaulting container name to portworx.
Use 'kubectl describe pod/px-cluster-21a6ec62-0ed7-48b8-b6ac-b6833005b533-c6tq6 -n kube-system' to see all of the containers in this pod.
ID			NAME						SIZE	HA	SHARED	ENCRYPTED	PROXY-VOLUME	IO_PRIORITY	STATUS				SNAP-ENABLED	
828496762191136181	pvc-39ddaeb9-b852-461f-a327-54ce392ee775	1 GiB	1	no	no		no		MEDIUM		up - attached on 10.0.1.163	no
925209006038916696	pvc-7accafb2-e2c1-4636-bb5f-a4d2e2550470	1 GiB	1	no	no		no		MEDIUM		up - attached on 10.0.1.205	no
231918167691463467	pvc-87fb6b66-6f88-4658-bda9-ee22c6100ed2	1 GiB	1	no	no		no		MEDIUM		up - attached on 10.0.1.163	no
965782486596909278	pvc-97bbaf86-7a27-4a5d-988a-22d306127b2f	1 GiB	1	no	no		no		MEDIUM		up - attached on 10.0.1.157	no
600324875059410877	pvc-c7a3c692-11fd-4f95-8d27-6db962aaaf9c	1 GiB	1	no	no		no		MEDIUM		up - attached on 10.0.1.205	no
852880312411130158	pvc-e5f2c02e-ca27-4561-aa0e-7fc15f95f34d	1 GiB	1	no	no		no		MEDIUM		up - attached on 10.0.1.157	no

Let’s check the version using ysqlsh

% kubectl exec --namespace yb-operator -it yb-tserver-0 -- /home/yugabyte/bin/ysqlsh
ysqlsh (11.2-YB-2.7.0.0-b0)
Type "help" for help.

yugabyte=# select version();
                                                  version                                                   
------------------------------------------------------------------------------------------------------------
 PostgreSQL 11.2-YB-2.7.0.0-b0 on x86_64-pc-linux-gnu, compiled by gcc (Homebrew gcc 5.5.0_4) 5.5.0, 64-bit
(1 row)

yugabyte=# alter role yugabyte with password 'portworx';
ALTER ROLE
yugabyte=# \q

Clean-Up

To avoid unnecessary costs we can scale compute nodes down to zero, or to remove the YugabyteDB cluster using kubectl delete, for example.

% kubectl delete -f px-yb-cluster.yaml 
ybcluster.yugabyte.com "px-ybcluster" deleted

The yb-master and yb-tserver pods have now terminated.

% kubectl get po -n yb-operator 
NAME                                 READY   STATUS    RESTARTS   AGE
yugabyte-operator-5dd57b9f8c-w66qr   1/1     Running   0          21h

This will not delete the Persistent Volume Claims (PVCs), therefore the data will persist if you recreate the cluster.

However if you no longer require the data, delete the PVCs with kubeclt delete pvc, for example:

% kubectl get pvc -n yb-operator
NAME                    STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
datadir0-yb-master-0    Bound    pvc-e5f2c02e-ca27-4561-aa0e-7fc15f95f34d   1Gi        RWO            px-yb-csi-sc   22h
datadir0-yb-master-1    Bound    pvc-c7a3c692-11fd-4f95-8d27-6db962aaaf9c   1Gi        RWO            px-yb-csi-sc   22h
datadir0-yb-master-2    Bound    pvc-87fb6b66-6f88-4658-bda9-ee22c6100ed2   1Gi        RWO            px-yb-csi-sc   22h
datadir0-yb-tserver-0   Bound    pvc-97bbaf86-7a27-4a5d-988a-22d306127b2f   1Gi        RWO            px-yb-csi-sc   22h
datadir0-yb-tserver-1   Bound    pvc-7accafb2-e2c1-4636-bb5f-a4d2e2550470   1Gi        RWO            px-yb-csi-sc   22h
datadir0-yb-tserver-2   Bound    pvc-39ddaeb9-b852-461f-a327-54ce392ee775   1Gi        RWO            px-yb-csi-sc   22h

% kubectl delete pvc -n yb-operator --all
persistentvolumeclaim "datadir0-yb-master-0" deleted
persistentvolumeclaim "datadir0-yb-master-1" deleted
persistentvolumeclaim "datadir0-yb-master-2" deleted
persistentvolumeclaim "datadir0-yb-tserver-0" deleted
persistentvolumeclaim "datadir0-yb-tserver-1" deleted
persistentvolumeclaim "datadir0-yb-tserver-2" deleted

% kubectl get pvc -n yb-operator         
No resources found in yb-operator namespace.

Summary

In this post I have shared how we can install a YugabyteDB cluster on an Oracle Kubernetes Engine (OKE) cluster and provide persistent Portworx storage to the database cluster.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Create a website or blog at WordPress.com

Up ↑

%d bloggers like this: