Oracle 19c on Kubernetes using NFS Persistent Storage

In this Blog I can going to show how you can run an Oracle 19c Database on a Kubernetes cluster using a Network File System (NFS) server for Persistent Storage.

In this example I will be using the Pure Storage Container Storage Interface (CSI) Driver Pure Service Orchestrator (PSO) to present a persistent storage to my Oracle Database container using a FlashBlade NFS (Network File System) server in my lab.

If you have not yet created an Oracle 19c Kubernetes image from the official Oracle Docker build scripts you can find detailed steps here.

Persistent NFS Storage

Let’s start by creating our database file systems, the Oracle Docker image can use 3 filesystems for storing persistent data.

  • /opt/oracle/oradata – used for Oracle database files
  • /opt/oracle/scripts/setup – used for custom scripts run on initial setup
  • /opt/oracle/scripts/startup – used for custom script run on database startup

Below is an extract from my pvc-nfs.yaml file which we will use to create our FlashBlade NFS filesystem using the PSO CSI Driver.

---
### Oracle 19c NFS
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: ora-data193-nfs
  namespace: oracle-namespace
  labels:
    app: database
    version: 19.3.0.1
spec:
  storageClassName: pure-file
  accessModes:
    - ReadWriteMany
  resources:
    requests:
     storage: 20Gi

Create Persistent Volume Claim (PVC)

$ kubectl apply -f pvc-nfs.yaml -n oracle-namespace
persistentvolumeclaim/ora-data193-nfs created
persistentvolumeclaim/ora-startup193-nfs created
persistentvolumeclaim/ora-setup193-nfs created

Get Persistent Volume Claim (PVC)

To help the track the PVC’s I will highlight the ‘/opt/oracle/oradata’ filesystem throughout this blog post.

$ kubectl get pvc -n oracle-namespace
NAME               STATUS VOLUME                              CAPACITY ACCESS MODES STORAGECLASS 
ora-data193-nfs    Bound  pvc-797e64df-4b12-4cf6-9e08-a6fa51bb4408 20Gi RWX         pure-file      
ora-setup193-nfs   Bound  pvc-35ced0e5-ec71-4454-8fa6-5cc1476a268e 1Gi  RWX         pure-file
ora-startup193-nfs Bound  pvc-73b5c2a8-080b-4979-b69b-e1206189167b 1Gi  RWX         pure-file

Describe Persistent Volume Claim (PVC)

$ kubectl describe pvc/ora-data193-nfs -n oracle-namespace 
Name:          ora-data193-nfs
Namespace:     oracle-namespace
StorageClass:  pure-file
Status:        Bound
Volume:        pvc-797e64df-4b12-4cf6-9e08-a6fa51bb4408
Labels:        app=database
               version=19.3.0.1
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: pure-csi
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      20Gi
Access Modes:  RWX
VolumeMode:    Filesystem

Delete Persistent Volume Claim (PVC)

If you no longer need your file system you can delete with kubectl delete persistentvolumeclaim or slightly easier kubectl delete pvc.

$ kubectl delete -n oracle-namespace persistentvolumeclaim ora-data193-nfs

Oracle 19c Database Creation

Using the previously created Kubernetes Oracle Docker image, ConfigMap and my Oracle 19c Database yaml file I can quickly spin up a new database with kubectl apply.

Create Oracle 19c NFS Deployment

$ kubectl apply -f database19c-nfs.yaml -n oracle-namespace
deployment.apps/oracle19c-nfs created
service/oracle19c-nfs created

List Running Database

List running databases with kubectl get pods

$ kubectl get pods -n oracle-namespace -o wide
NAME                           READY STATUS  RESTARTS AGE IP         NODE          NOMINATED NODE   
oracle19c-nfs-69b87cb877-lf4w8 1/1   Running 0        58s 10.1.92.87 z-re-uk8s02   <none>           

Database Creation Log

We can check progress of our database creation by reviewing the log output with kubectl logs

$ kubectl logs oracle19c-69b87cb877-lf4w8 -n oracle-namespace
ORACLE PASSWORD FOR SYS, SYSTEM AND PDBADMIN: Kube#2020

LSNRCTL for Linux: Version 19.0.0.0.0 - Production on 18-MAY-2020 14:16:48

Copyright (c) 1991, 2019, Oracle.  All rights reserved.

Starting /opt/oracle/product/19c/dbhome_1/bin/tnslsnr: please wait...

TNSLSNR for Linux: Version 19.0.0.0.0 - Production
System parameter file is /opt/oracle/product/19c/dbhome_1/network/admin/listener.ora
Log messages written to /opt/oracle/diag/tnslsnr/oracle19c-69b87cb877-lf4w8/listener/alert/log.xml
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1)))
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=0.0.0.0)(PORT=1521)))

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC1)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 19.0.0.0.0 - Production
Start Date                18-MAY-2020 14:16:49
Uptime                    0 days 0 hr. 0 min. 0 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /opt/oracle/product/19c/dbhome_1/network/admin/listener.ora
Listener Log File         /opt/oracle/diag/tnslsnr/oracle19c-69b87cb877-lf4w8/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=0.0.0.0)(PORT=1521)))
The listener supports no services
The command completed successfully
Prepare for db operation
8% complete
Copying database files
31% complete
Creating and starting Oracle instance
32% complete
36% complete
...

FlashBlade Network File System (NFS)

Once the database build has completed we can connect to our Kubernetes container using kubectl exec , from inside our container we can see the FlashBlade hosted NFS mount points.

$ kubectl exec -it oracle19c-nfs-69b87cb877-lf4w8 -n oracle-namespace -- /bin/bash
[oracle@oracle19c-nfs-69b87cb877-lf4w8 ~]$ df -h
Filesystem                                                     Size  Used Avail Use% Mounted on
overlay                                                        197G   23G  166G  13% /
tmpfs                                                           64M     0   64M   0% /dev
tmpfs                                                          7.9G     0  7.9G   0% /sys/fs/cgroup
tmpfs                                                          7.9G     0  7.9G   0% /dev/shm
/dev/sda2                                                      197G   23G  166G  13% /etc/hosts
192.168.4.150:/rek8s-pvc-797e64df-4b12-4cf6-9e08-a6fa51bb4408   20G  3.9G   17G  20% /opt/oracle/oradata
192.168.4.150:/rek8s-pvc-35ced0e5-ec71-4454-8fa6-5cc1476a268e  1.0G     0  1.0G   0% /opt/oracle/scripts/startup
192.168.4.150:/rek8s-pvc-73b5c2a8-080b-4979-b69b-e1206189167b  1.0G     0  1.0G   0% /opt/oracle/scripts/setup
tmpfs                                                          7.9G   12K  7.9G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs                                                          7.9G     0  7.9G   0% /proc/acpi
tmpfs                                                          7.9G     0  7.9G   0% /proc/scsi
tmpfs                                                          7.9G     0  7.9G   0% /sys/firmware

By logging onto my FlashBlade I can see my 20G /opt/oracle/oradata file system has consumed 3.85GB with a Data Reduction of 4.1 : 1

FlashBlade File Systems

Oracle Database Management

We can stop and start our database with kubectl scale –replicas , setting to zero to stop, or 1 to restart.

Stop Database

$ kubectl scale -n oracle-namespace deployment oracle19c-nfs --replicas=0

Start Database

$ kubectl scale -n oracle-namespace deployment oracle19c-nfs --replicas=0

Database Move / Relocation

We can also move / relocate our database to other nodes within our Kubernetes cluster to perfom maintenance etc. using kubectl cordon. I have already blogged on the use of cordon / uncordon, and if your interested you can read about it here.

Connect to Database

Before we can use sqlplus or other database tools, we need to know which port out listener (1521) has been mapped to by Kubernetes, we can do this with kubectl get svc

$ kubectl get svc/oracle19c-nfs -n oracle-namespace
NAME            TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)                         AGE
oracle19c-nfs   NodePort   10.152.183.196   <none>        1521:31413/TCP,5500:31836/TCP   4d5h

Now, from my desktop I can connect directly to my database using sqlplus and the Kubernetes service port.

$ sqlplus system/Kube#2020@z-re-uk8s01:31413/ORCL @database_details.sql

SQL*Plus: Release 19.0.0.0.0 - Production on Tue May 19 09:41:43 2020
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.

Last Successful login time: Tue May 19 2020 09:40:49 +01:00

Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0

Current Time   : 19/05/20 08:41:48

Database Details
===============================================
Hostname       : oracle19c-nfs-69b87cb877-lf4w8
Database Name  : ORCL
Date Created   : 18/05/20 20:25:40
Date Started   : 18/05/20 23:28:28
Resetlogs Date : 18/05/20 20:25:42
DB Status      : OPEN
Space Allocated:      1.63 GB
Space Used     :      1.39 GB

Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Create a website or blog at WordPress.com

Up ↑

%d bloggers like this: