Background
In Getting started with Oracle 21c, Oracle database Kubernetes operator (OraOperator), OCI and Portworx storage I shared details of my Terraform script which I use to build OKE clusters with Portworx storage. I detailed how to install the Oracle Database Kubernetes Operator and how to configure it for a single instance database running on Oracle Container Engine for Kubernetes (OKE) service within Oracle Cloud Infrastructure (OCI).
In the post I share how we can use Portworx replication to provide the ability to failover a Kubernetes Oracle 21c database pod across Availability Domains (ADs) with OCI.
OKE Environment
If you followed my previous post you should already have an OCI OKE and Portworx Storage cluster running an Oracle 21c database, so before we start a quick look at the OKE cluster.
Kubernetes Nodes
Use kubectl get nodes to see the Kubernetes cluster, providing OKE labels we can also confirm Region and Availability Domains (ADs) for each worker node for example.
% kubectl get nodes -L topology.kubernetes.io/region,topology.kubernetes.io/zone NAME STATUS ROLES AGE VERSION REGION ZONE 10.0.10.209 Ready node 5d2h v1.20.11 uk-london-1 UK-LONDON-1-AD-1 10.0.10.53 Ready node 5d2h v1.20.11 uk-london-1 UK-LONDON-1-AD-3 10.0.10.87 Ready node 5d2h v1.20.11 uk-london-1 UK-LONDON-1-AD-2
Portworx
Set-up an alias for pxctl and use it to check version of Portworx
% export PX_POD=$(kubectl get pods -l name=portworx -n kube-system -o jsonpath='{.items[0].metadata.name}')
% alias pxctl='kubectl exec -n kube-system ${PX_POD} -it -- /opt/pwx/bin/pxctl'
% pxctl -v
Defaulted container "portworx" out of: portworx, csi-node-driver-registrar
pxctl version 2.8.1.5-937bfdd
Oracle Database
To avoid having to specify our namespace with -n each time I will use the set-context to specify a default namespace of oracle-namespace.
% kubectl config set-context --current --namespace=oracle-namespace Context "context-c4tnd3w25yq" modified.
Kubectl will now default to using the oracle–namespace for example,
List Databases
To list Oracle databases and confirm database version and identify which worker node is being used use kubectl get singleinstancedatabase, for example
% kubectl get singleinstancedatabase -n oracle-namespace NAME EDITION STATUS VERSION CONNECT STR OEM EXPRESS URL sidb-pstg Enterprise Healthy 21.3.0.0.0 10.0.10.209:30898/PSTGCDB https://10.0.10.209:30228/em
So, from the above I have determined my OKE Oracle 21c environment looks like this:
None of my OKE worker nodes have public interfaces, however my Terraform script created a Bastion host which has sqlplus installed. So I can use my Bastion host to connect to my database using any of the worker node IPs.
Using sqlplus we can confirm database instance name, version and edition using the v$instance for example.
[opc@px-bastion ~]$ sqlplus sys/Kube#2022@10.0.10.87:30898/PSTGPDB1 as sysdba SQL*Plus: Release 21.0.0.0.0 - Production on Thu Apr 7 16:32:13 2022 Version 21.3.0.0.0 Copyright (c) 1982, 2021, Oracle. All rights reserved. Last Successful login time: Thu Apr 07 2022 16:31:40 +00:00 Connected to: Oracle Database 21c Enterprise Edition Release 21.0.0.0.0 - Production Version 21.3.0.0.0 SQL> set linesize 200 SQL> select INSTANCE_NAME, HOST_NAME, VERSION_FULL, EDITION from v$instance; INSTANCE_NAME HOST_NAME VERSION_FULL EDITION ---------------- ---------------------------------------------------------------- ----------------- ------- PSTGCDB sidb-pstg-dxlwf 21.3.0.0.0 EE
Note: the Oracle database host name is the Kubernetes pod name not the OKE worker node name.
Let’s create some test data
SQL> CREATE TABLE test_table TABLESPACE USERS AS
( SELECT LEVEL id,
SYSDATE+DBMS_RANDOM.VALUE(-1000, 1000) date_value,
DBMS_RANDOM.string('A', 20) text_value
FROM dual
CONNECT BY LEVEL <= 1000000)
/
Table created.
SQL> select count(*) from test_table;
COUNT(*)
----------
1000000
Pod Failover
Cordon Node
Before we kill the Oracle 21c pod, I will remove the Kubernetes worker node for future scheduling using kubectl cordon.
% kubectl cordon 10.0.10.209 node/10.0.10.209 cordoned % kubectl get nodes NAME STATUS ROLES AGE VERSION 10.0.10.209 Ready,SchedulingDisabled node 5h13m v1.20.11 10.0.10.53 Ready node 5h13m v1.20.11 10.0.10.87 Ready node 5h13m v1.20.11
Kill Pod
To simulate a node crash we can delete the running pod.
% kubectl delete pod/sidb-pstg-dxlwf pod "sidb-pstg-dxlwf" deleted
Check Pod Status
We can see that a new Kubernetes pod has been automatically started on a different worker node in Availability Domain 2.
% kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES sidb-pstg-khp3d 1/1 Running 0 60s 10.244.0.137 10.0.10.87 <none> <none>
Normally this would be an issue as a worker node is unable to access block storage from a different Availability Domain.
However, as my OKE cluster is using Portworx storage this is not a problem as the data has been replicated across the 3 Availability Domains by the Portworx storage class and is available for use in any AD.
Check Database Status
If I return to my Bastion host and use sqlplus with the same database connection string, we can see the host name has changed to reflect the new pod name.
[opc@px-bastion ~]$ sqlplus sys/Kube#2022@10.0.10.87:30898/PSTGPDB1 as sysdba SQL*Plus: Release 21.0.0.0.0 - Production on Tue Apr 12 13:59:08 2022 Version 21.3.0.0.0 Copyright (c) 1982, 2021, Oracle. All rights reserved. Connected to: Oracle Database 21c Enterprise Edition Release 21.0.0.0.0 - Production Version 21.3.0.0.0 SQL> set linesize 200 SQL> select INSTANCE_NAME, HOST_NAME, VERSION_FULL, EDITION from v$instance; INSTANCE_NAME HOST_NAME VERSION_FULL EDITION ---------------- ---------------------------------------------------------------- ----------------- ------- PSTGCDB sidb-pstg-khp3d 21.3.0.0.0 EE
And if we query the test table we can see how data is still available.
SQL> select count(*) from test_table; COUNT(*) ---------- 1000000
Portworx Volume
We can confirm that our Oracle 21c migrated database running on a worker node in Availability Domain 2 is using local block storage by using pxctl volume list and checking status and looking to see where it’s attached, for example up – attached on 10.0.10.87
% pxctl volume list Defaulted container "portworx" out of: portworx, csi-node-driver-registrar ID NAME SIZE HA SHARED ENCRYPTED PROXY-VOLUME IO_PRIORITY STATUS 1106215858654398303 pvc-a9dcdce1-99a2-4db6-b0e6-e7b166ddeb7c 60 GiB 3 no no. no MEDIUM up - attached on 10.0.10.87
Un-cordon Node
Before we finish, let’s un-cordon the node making it available again for Kubernetes scheduling.
% kubectl uncordon 10.0.10.239 node/10.0.10.209 uncordoned % kubectl get nodes NAME STATUS ROLES AGE VERSION 10.0.10.209 Ready node 5h19m v1.20.11 10.0.10.53 Ready node 5h19m v1.20.11 10.0.10.87 Ready node 5h19m v1.20.11
Summary
In this post I have shared how to deploy an Oracle 21c database with the Oracle Database Kubernetes Operator (OraOperator), and how to use Portworx to provide storage replication across three OCI Availability Domains (ADs) with an Oracle Cloud Infrastructure (OCI) Region.
I have demonstrated how a Portworx Kubernetes storage class can be used to protect data and provide database failover.
Please Note: The current release of OraOperator (v0.1.0) is for development and test only.
DO NOT USE IN PRODUCTION (yet!)