How to deploy an on-premises 21c database using the Oracle Database Operator for Kubernetes

Background

The Oracle Database Operator for Kubernetes (OraOpeator) provides a simple, supported method of provisioning, cloning, and patching Oracle Databases including databases deployed on Kubernetes.

I have previously posted a blog on how to use the OraOperator (v0.1.0) to deliver an Oracle database on the Oracle Cloud Infrastructure (OCI) Oracle Container Engine for Kubernetes (OKE) service and how to deliver a warm failover with Portworx persistent storage for Kubernetes.

In this post I will show how we can use the OraOperator to create a single instance Oracle 21.3 Databases on an on-premises Kubernetes cluster.

Please note: The current release of OraOperator (v0.2.0) is for development and test only. DO NOT USE IN PRODUCTION.

Environment

Before you start it’s advisable to check the latest OraOperator prerequisites to help avoid any issues down the line.

Kubernetes Version

Let’s begin by confirming the version of Kubernetes with kubectl version

[root@master-1 ~]# kubectl version --short | awk -Fv '/Server Version: / {print $3}'
1.21.11

Kubernetes Nodes

And check Kubernetes worker node details with kubectl get nodes

[root@master-1 ~]# kubectl get nodes -o wide
NAME       STATUS   ROLES                  AGE    VERSION    INTERNAL-IP      EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION                CONTAINER-RUNTIME
master-1   Ready    control-plane,master   118m   v1.21.11   10.225.115.193   <none>        CentOS Linux 7 (Core)   3.10.0-1160.45.1.el7.x86_64   docker://1.13.1
node-1-1   Ready    <none>                 117m   v1.21.11   10.225.115.190   <none>        CentOS Linux 7 (Core)   3.10.0-1160.45.1.el7.x86_64   docker://1.13.1
node-1-2   Ready    <none>                 117m   v1.21.11   10.225.115.189   <none>        CentOS Linux 7 (Core)   3.10.0-1160.45.1.el7.x86_64   docker://1.13.1
node-1-3   Ready    <none>                 117m   v1.21.11   10.225.115.191   <none>        CentOS Linux 7 (Core)   3.10.0-1160.45.1.el7.x86_64   docker://1.13.1

Cert Manager

The Oracle Database Kubernetes Operator uses webhooks for validating user input before persisting it in Etcd.

Webhooks require TLS certificates that are generated and managed by a certificate manager.

Install the certificate manager using the following command:

[root@master-1 ~]# kubectl apply -f https://github.com/jetstack/cert-manager/releases/latest/download/cert-manager.yaml
namespace/cert-manager created
customresourcedefinition.apiextensions.k8s.io/clusterissuers.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/challenges.acme.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/issuers.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/orders.acme.cert-manager.io created
serviceaccount/cert-manager-cainjector created
serviceaccount/cert-manager created
serviceaccount/cert-manager-webhook created
configmap/cert-manager-webhook created
clusterrole.rbac.authorization.k8s.io/cert-manager-cainjector created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-issuers created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificates created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-orders created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-challenges created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim created
clusterrole.rbac.authorization.k8s.io/cert-manager-view created
clusterrole.rbac.authorization.k8s.io/cert-manager-edit created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-approve:cert-manager-io created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificatesigningrequests created
clusterrole.rbac.authorization.k8s.io/cert-manager-webhook:subjectaccessreviews created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-cainjector created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-issuers created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificates created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-orders created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-challenges created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-approve:cert-manager-io created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificatesigningrequests created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-webhook:subjectaccessreviews created
role.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection created
role.rbac.authorization.k8s.io/cert-manager:leaderelection created
role.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving created
rolebinding.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection created
rolebinding.rbac.authorization.k8s.io/cert-manager:leaderelection created
rolebinding.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving created
service/cert-manager created
service/cert-manager-webhook created
deployment.apps/cert-manager-cainjector created
deployment.apps/cert-manager created
deployment.apps/cert-manager-webhook created
mutatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created
validatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created
[root@master-1 ~]# git clone https://github.com/oracle/oracle-database-operator.git
Cloning into 'oracle-database-operator'...
remote: Enumerating objects: 5447, done.
remote: Counting objects: 100% (1294/1294), done.
remote: Compressing objects: 100% (195/195), done.
remote: Total 5447 (delta 1184), reused 1099 (delta 1099), pack-reused 4153
Receiving objects: 100% (5447/5447), 3.25 MiB | 3.18 MiB/s, done.
Resolving deltas: 100% (3546/3546), done.

The Database Operator

Now clone the Oracle Database Kubernetes Operator from GitHub, for example

[root@master-1 ~]# git clone https://github.com/oracle/oracle-database-operator.git
Cloning into 'oracle-database-operator'...
remote: Enumerating objects: 5447, done.
remote: Counting objects: 100% (1294/1294), done.
remote: Compressing objects: 100% (195/195), done.
remote: Total 5447 (delta 1184), reused 1099 (delta 1099), pack-reused 4153
Receiving objects: 100% (5447/5447), 3.25 MiB | 3.18 MiB/s, done.
Resolving deltas: 100% (3546/3546), done.

Change directory to oracle-database-operator then use kubectl apply to install the database Operator

[root@master-1 oracle-database-operator]# kubectl apply -f oracle-database-operator.yaml
namespace/oracle-database-operator-system created
customresourcedefinition.apiextensions.k8s.io/autonomouscontainerdatabases.database.oracle.com created
customresourcedefinition.apiextensions.k8s.io/autonomousdatabasebackups.database.oracle.com created
customresourcedefinition.apiextensions.k8s.io/autonomousdatabaserestores.database.oracle.com created
customresourcedefinition.apiextensions.k8s.io/autonomousdatabases.database.oracle.com created
customresourcedefinition.apiextensions.k8s.io/cdbs.database.oracle.com created
customresourcedefinition.apiextensions.k8s.io/dbcssystems.database.oracle.com created
customresourcedefinition.apiextensions.k8s.io/oraclerestdataservices.database.oracle.com created
customresourcedefinition.apiextensions.k8s.io/pdbs.database.oracle.com created
customresourcedefinition.apiextensions.k8s.io/shardingdatabases.database.oracle.com created
customresourcedefinition.apiextensions.k8s.io/singleinstancedatabases.database.oracle.com created
role.rbac.authorization.k8s.io/oracle-database-operator-leader-election-role created
clusterrole.rbac.authorization.k8s.io/oracle-database-operator-manager-role created
clusterrole.rbac.authorization.k8s.io/oracle-database-operator-metrics-reader created
clusterrole.rbac.authorization.k8s.io/oracle-database-operator-oracle-database-operator-proxy-role created
rolebinding.rbac.authorization.k8s.io/oracle-database-operator-oracle-database-operator-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/oracle-database-operator-oracle-database-operator-manager-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/oracle-database-operator-oracle-database-operator-proxy-rolebinding created
service/oracle-database-operator-controller-manager-metrics-service created
service/oracle-database-operator-webhook-service created
certificate.cert-manager.io/oracle-database-operator-serving-cert created
issuer.cert-manager.io/oracle-database-operator-selfsigned-issuer created
mutatingwebhookconfiguration.admissionregistration.k8s.io/oracle-database-operator-mutating-webhook-configuration created
validatingwebhookconfiguration.admissionregistration.k8s.io/oracle-database-operator-validating-webhook-configuration created
deployment.apps/oracle-database-operator-controller-manager created

The file will perform the following operations:

  • Create Namespace (oracle-database-operator-system)
  • Create Custom Resource Definitions (CRDs)
  • Create Roles and Bindings
  • Deploy the Operator

Use kubectl get all to see the Kubernetes pods, services, deployments and replicasets.

[root@master-1 ~]# kubectl get all -n oracle-database-operator-system 
NAME                                                               READY   STATUS    RESTARTS   AGE
pod/oracle-database-operator-controller-manager-694f44ffc8-6hbx9   1/1     Running   0          3m21s
pod/oracle-database-operator-controller-manager-694f44ffc8-9pqbg   1/1     Running   0          3m21s
pod/oracle-database-operator-controller-manager-694f44ffc8-h99cj   1/1     Running   0          3m21s

NAME                                                                  TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/oracle-database-operator-controller-manager-metrics-service   ClusterIP   10.111.168.39   <none>        8443/TCP   3m22s
service/oracle-database-operator-webhook-service                      ClusterIP   10.99.58.64     <none>        443/TCP    3m22s

NAME                                                          READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/oracle-database-operator-controller-manager   3/3     3            3           3m21s

NAME                                                                     DESIRED   CURRENT   READY   AGE
replicaset.apps/oracle-database-operator-controller-manager-694f44ffc8   3         3         3       3m21s

Create Namespace

Optional, create a new Kubernetes namespace for ease of management, for example oracle-namespace.

Use kubectl create namespace to create a Kubenetes namespace, for example.

[root@master-1 ~]# kubectl create namespace oracle-namespace
namespace/oracle-namespace created

To avoid having to specify the namespace with -n each time I will use the set-context to specify a default namespace of oracle-namespace.

[root@master-1 ~]# kubectl config set-context --current --namespace=oracle-namespace
Context "kubernetes-admin@kubernetes" modified.

Kubernetes Secrets

For this blog we will use Kubernetes Secrets to store Database and Oracle Container Registry (OCR) credentials.

Note this is not the most secure method and should not be used for production as the value can be easily obtained. Oracle strongly recommends that you set and get sensitive data from Oracle Cloud Infrastructure Vault, or from other third-party Vaults.

Oracle Database Secret

Use kubectl create secret to create a database secret, this will be used to logon to the database.

[root@master-1 ~]# kubectl create secret generic admin-password --from-literal=sidb-admin-password='Kube#2022' -n oracle-namespace
secret/admin-password created

Oracle Container Registry Secret

Logon to the Oracle Container Registry (OCR) using SSO credentials, for example:

[root@master-1 ~]# docker login container-registry.oracle.com
Username: <sso username>
Password: <sso password>
Login Succeeded

Create Kubernetes secret for OCR using cached Oracle SSO credentials

[root@master-1 ~]# kubectl create secret generic regcred --from-file=.dockerconfigjson=$HOME/.docker/config.json  --type=kubernetes.io/dockerconfigjson -n oracle-namespace
secret/regcred created

Database Configuration

Create a copy of the cloned ../oracle-database-operator/config/samples/sidb/singleinstancedatabase.yaml to your working directory and update as required, for example.

[root@master-1 ~]# cp oracle-database-operator/config/samples/sidb/singleinstancedatabase.yaml pstg.yaml

And localise as required.

For this blog post, I updated 

  • name, namespace, sid and pdbName
  • secretName and secretKey
  • pullFrom and pullSecrets
  • size and storageClasss.

Now provision database using kubectl apply providing the updated .yaml file, for example.

[root@master-1 ~]# kubectl apply -f pstg.yaml -n oracle-namespace
singleinstancedatabase.database.oracle.com/pstg created

Database Status

We can use watch kubectl get singleinstancedatabase to return database status, the status will change from Pending -> Creating -> Patching -> Healthly

[root@master-1 ~]# watch kubectl get singleinstancedatabase -n oracle-namespace 

Every 2.0s: kubectl get singleinstancedatabase -n oracle-namespace                                         Mon Dec 12 12:02:05 2022

NAME   EDITION      STATUS    VERSION       CONNECT STR                  TCPS CONNECT STR   OEM EXPRESS URL
pstg   Enterprise   Pending   Unavailable   10.225.115.191:30745/PSTG1   Unavailable        https://10.225.115.191:30275/em

Wait until Healthy

[root@master-1 ~]# kubectl get singleinstancedatabase pstg -n oracle-namespace
NAME   EDITION      STATUS    VERSION      CONNECT STR                  TCPS CONNECT STR   OEM EXPRESS URL
pstg   Enterprise   Healthy   21.3.0.0.0   10.225.115.191:30745/PSTG1   Unavailable        https://10.225.115.191:30275/em

Oracle Database Log

Note, whilst the Oracle database is being created we can review the creation log with kubectl logs pod, for example.

[root@master-1 ~]# kubectl get pods -n oracle-namespace
NAME         READY   STATUS    RESTARTS   AGE
pstg-9vqzn   0/1     Running   0          23m

[root@master-1 ~]# kubectl logs pod/pstg-9vqzn -n oracle-namespace
[2022:12:12 12:10:56]: Acquiring lock .PSTG1.create_lck with heartbeat 30 secs
[2022:12:12 12:10:56]: Lock acquired
[2022:12:12 12:10:56]: Starting heartbeat
[2022:12:12 12:10:56]: Lock held .PSTG1.create_lck
ORACLE EDITION: ENTERPRISE
LSNRCTL for Linux: Version 21.0.0.0.0 - Production on 12-DEC-2022 12:10:56
Copyright (c) 1991, 2021, Oracle.  All rights reserved.
Starting /opt/oracle/product/21c/dbhome_1/bin/tnslsnr: please wait...
TNSLSNR for Linux: Version 21.0.0.0.0 - Production
...

Persistent Volume Claim

With kubectl get pvc we can confirm that a Persistent Volume Claim has been created with the size and storage class specified in our .yaml file.

[root@master-1 ~]# kubectl get pvc
NAME   STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
pstg   Bound    pvc-4f2e08d1-43b6-457f-9bc9-797ed9a1ff2c   8Gi        RWO            px-csi-db      12m

Oracle Database

Now we have a Healthy OraOperator deployed 21c database we can confirm database instance nameversion and edition by shelling into the pod using the v$instance view providing the database credentials stored in the Kubernetes secret.

[root@master-1 ~]#  kubectl get pods -n oracle-namespace -o wide
NAME         READY   STATUS    RESTARTS   AGE   IP           NODE       NOMINATED NODE   READINESS GATES
pstg-9vqzn   1/1     Running   0          32m   10.244.2.9   node-1-3   <none>           <none>

Note: the Oracle database host name is the pod name not the Kubernetes node.

[root@master-1 ~]# kubectl exec -it pods/pstg-9vqzn -n oracle-namespace -- sqlplus system/Kube#2022@PSTG1 
Defaulted container "pstg" out of: pstg, init-permissions (init), init-wallet (init)

SQL*Plus: Release 21.0.0.0.0 - Production on Mon Dec 12 12:31:57 2022
Version 21.3.0.0.0

Copyright (c) 1982, 2021, Oracle.  All rights reserved.

Last Successful login time: Mon Dec 12 2022 12:19:00 +00:00

Connected to:
Oracle Database 21c Enterprise Edition Release 21.0.0.0.0 - Production
Version 21.3.0.0.0

SQL> set linesize 200
SQL> select INSTANCE_NAME, HOST_NAME, VERSION_FULL, EDITION from v$instance;

INSTANCE_NAME	 HOST_NAME							  VERSION_FULL	    EDITION
---------------- ---------------------------------------------------------------- ----------------- -------
PSTG1		 pstg-9vqzn							  21.3.0.0.0	    EE

SQL> 

Oracle Instant Client

We can also do the same with the Oracle Instant client for Docker, start by pulling the Oracle Instant Client docker image.

[root@master-1 ~]# docker pull ghcr.io/oracle/oraclelinux8-instantclient:21
Trying to pull repository ghcr.io/oracle/oraclelinux8-instantclient ... 
21: Pulling from ghcr.io/oracle/oraclelinux8-instantclient
1ca7c848b9e5: Pull complete 
d92bd232d5b4: Pull complete 
Digest: sha256:76dd41b84f5217a66eada78b421f12dec217b69578e4e4e2a40fcf006a7f40ec
Status: Downloaded newer image for ghcr.io/oracle/oraclelinux8-instantclient:21

Determine Node Port.

[root@master-1 ~]# kubectl get svc 
NAME       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                         AGE
pstg       ClusterIP   10.106.247.177   <none>        1521/TCP                        90m
pstg-ext   NodePort    10.105.235.39    <none>        5500:30275/TCP,1521:30745/TCP   90m

And connect using the Oracle Instance Client Docker image and Kubernetes Node Port.

[root@master-1 ~]# docker run -ti --rm ghcr.io/oracle/oraclelinux8-instantclient:21 sqlplus system/Kube#2022@10.105.235.39/PSTG1 

SQL*Plus: Release 21.0.0.0.0 - Production on Mon Dec 12 12:53:05 2022
Version 21.8.0.0.0

Copyright (c) 1982, 2022, Oracle.  All rights reserved.

Last Successful login time: Mon Dec 12 2022 12:31:57 +00:00

Connected to:
Oracle Database 21c Enterprise Edition Release 21.0.0.0.0 - Production
Version 21.3.0.0.0

SQL> set linesize 200
SQL> select INSTANCE_NAME, HOST_NAME, VERSION_FULL, EDITION from v$instance;

INSTANCE_NAME	 HOST_NAME							  VERSION_FULL	    EDITION
---------------- ---------------------------------------------------------------- ----------------- -------
PSTG1		 pstg-9vqzn							  21.3.0.0.0	    EE

Summary

In this post I have shown how we can now use the Oracle Database Kubernetes Operator (OraOperator) to deploy an Oracle 21c single instance database on an on-premises Kubernetes cluster.

In my next post I will share how we can use Portworx CloudSnap to perform a storage snapshot and send it to an S3 bucket in OCI Object Storage.

Leave a Reply

Create a website or blog at WordPress.com

Up ↑

%d bloggers like this: