Fully automated Oracle Kubernetes Engine (OKE) deployment with Portworx using Terraform

Introduction

I frequently have the need to stand-up Kubernetes lab environments for R&D, learning and to develop blogs and solutions, Oracle Cloud Infrastructure (OCI) provides a great wizard to enable the rapid deployment of Oracle Kubernetes Engine (OKE) clusters.

However, if you find yourself performing the same task often you should consider automating the process, Oracle Cloud provides the OCI command line interface which can be used, however Terraform by HasiCorp should be top of your list for consideration.

The Oracle Cloud Infrastructure (OCI) Terraform provider enables us to use Terraform to interact with Oracle Cloud Infrastructure (OCI) resources and data sources.

I have previously shared how we can deploy an OKE environment with Terraform, in this post I will show how we can automate the provisioning of an Oracle Cloud Kubernetes environment on OCI with block storage configured for use by Portworx using Terraform.

Getting Started

Start by visiting https://central.portworx.com/ and use the Portworx wizard to create a specification for either Portworx Essentials or Portworx Enterprise.

If you have never previously used the Portworx specification Wizard, you may want to read my post Portworx installation on OCI which provides a detailed walkthrough including the creation of a Portworx specification.

Once you have followed the wizard, copy and save the deployment specification URLs as they we will be required in our Terraform terraform.tfvars file.

Portworx Specification Generator

The Terraform HashiCorp Configuration Language (HCL) script creates a Kubernetes configuration file in .kube/config (see kubeconfig.tf), to enable direct desktop access using kubectl add this location to your KUBECONFIG, for example:

export KUBECONFIG=${KUBECONFIG}:/Users/rekins/myTerraform/oke-px-terrafom/.kube/config

Git Repo

Clone the Terraform code from my GitHub repo, change directory, localise the terraform.tfvars file providing OCI account, tenancy, and Portworx URLs.

OKE Architecture

The Terraform HCL script builds a three node OKE cluster, with a compute instance in each of the three Availability Domains (ADs) and an attached dedicated block device for Portworx.

OCI UK-London-1 region.

The Terraform HCL script does not just automate the creation of a Kubernetes cluster but also the automates the creation of other used OCI resources, including:

  • Virtual Cloud Network (VCN)
    • Subnets
    • Security Lists
    • Route Tables
    • Intenet Gateway
    • Service Gateway
    • NAT Gateway
  • Oracle Kubernetes Engine (OKE)
    • Node Pool
  • Kubernetes ConfigFile
  • Block Volumes
  • Attachment of block volumes to compute instances
  • Bastion Host
  • Portworx Operator

Terraform Deployment

Download, install Terraform, and confirm installation and version, for example

 % terraform version
Terraform v1.1.7
on darwin_amd64

Change directory to the cloned repo, and deploy using Terraform for example.

  • terraform init
  • terraform validate
  • terraform plan

If all ok, now use terraform apply to deploy.

% terraform apply --auto-approve
...
Apply complete! Resources: 28 added, 0 changed, 0 destroyed.

Outputs:

all-availability-domains-in-your-tenancy = tolist([
  "iXvO:UK-LONDON-1-AD-1",
  "iXvO:UK-LONDON-1-AD-2",
  "iXvO:UK-LONDON-1-AD-3",
])
all-running-oke-instance-names = tolist([
  "oke-cumadkmup7q-nyu576ijn5q-sma4gmd7eca-2",
  "oke-cumadkmup7q-nyu576ijn5q-sma4gmd7eca-1",
  "oke-cumadkmup7q-nyu576ijn5q-sma4gmd7eca-0",
])
containerengine-cluster-kubernetes_version = "v1.20.11"
containerengine-cluster-name = "pxcluster"
containerengine-cluster-public-endpoint = "144.21.57.50:6443"
selected-core-services = tolist([
  "All LHR Services In Oracle Services Network",
])

From the above, you can see that Terraform created a three node private OKE cluster across three Availability Domains (ADs), with a public K8 API endpoint.

Oracle Kubernetes Engine

To validate the deployment logon to OCI, navigate using the hamburger menu to Developer Services -> Kubernetes Clusters (OKE).

Select the cluster, then node pool and confirm Availability Domain Placements, Kubernetes Node Condition and Versions, Kubernetes labels, for example.

OKE Cluster

Kubernetes Nodes

We can also check the OKE environment using kubectl, for example kubectl get nodes.

% kubectl get nodes -o wide
NAME          STATUS   ROLES   AGE     VERSION    INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                  KERNEL-VERSION                     CONTAINER-RUNTIME
10.0.10.179   Ready    node    7m32s   v1.20.11   10.0.10.179   <none>        Oracle Linux Server 7.8   4.14.35-1902.306.2.el7uek.x86_64   cri-o://1.20.2
10.0.10.190   Ready    node    7m37s   v1.20.11   10.0.10.190   <none>        Oracle Linux Server 7.8   4.14.35-1902.306.2.el7uek.x86_64   cri-o://1.20.2
10.0.10.88    Ready    node    6m52s   v1.20.11   10.0.10.88    <none>        Oracle Linux Server 7.8   4.14.35-1902.306.2.el7uek.x86_64   cri-o://1.20.2

Portworx Version

The Terraform HCL script also deploys Portworx, we can use pxctl -v to check version, or pxctl status to see overall status of the Portworx cluster, for example.

% PX_POD=$(kubectl get pods -l name=portworx -n kube-system -o jsonpath='{.items[0].metadata.name}')

% kubectl exec -it $PX_POD -n kube-system -- /opt/pwx/bin/pxctl -v                                  
Defaulting container name to portworx.
Use 'kubectl describe pod/px-cluster-5ad151aa-784e-4103-9ab5-a3529b18c12f-5l9vc -n kube-system' to see all of the containers in this pod.
pxctl version 2.8.1.5-937bfdd

Kubernetes Storage Classes

The Portworx Operator creates a number of new Kubernetes storage classes, we can see these and the two OCI storage classes using kubectl get storageclass or kubectl get sc.

% kubectl get sc           
NAME                             PROVISIONER                       RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
oci (default)                    oracle.com/oci                    Delete          Immediate              false                  12m
oci-bv                           blockvolume.csi.oraclecloud.com   Delete          WaitForFirstConsumer   true                   12m
px-db                            kubernetes.io/portworx-volume     Delete          Immediate              true                   4m48s
px-db-cloud-snapshot             kubernetes.io/portworx-volume     Delete          Immediate              true                   4m48s
px-db-cloud-snapshot-encrypted   kubernetes.io/portworx-volume     Delete          Immediate              true                   4m48s
px-db-encrypted                  kubernetes.io/portworx-volume     Delete          Immediate              true                   4m48s
px-db-local-snapshot             kubernetes.io/portworx-volume     Delete          Immediate              true                   4m48s
px-db-local-snapshot-encrypted   kubernetes.io/portworx-volume     Delete          Immediate              true                   4m48s
px-replicated                    kubernetes.io/portworx-volume     Delete          Immediate              true                   4m48s
px-replicated-encrypted          kubernetes.io/portworx-volume     Delete          Immediate              true                   4m48s
stork-snapshot-sc                stork-snapshot                    Delete          Immediate              true                   4m48s

Portworx StorageCluster

Using kubectl get storagecluster we can see the Portworx cluster status and version.

% kubectl get storagecluster -n kube-system
NAME                                              CLUSTER UUID                           STATUS   VERSION   AGE
px-cluster-5ad151aa-784e-4103-9ab5-a3529b18c12f   b27154ec-6ff4-498b-87ed-7fa8349d9f54   Online   2.8.1.5   46h

Portworx Storage Nodes

The Terraform code has deployed three Portworx Storage nodes, we can see these using kubectl get storage nodes for example.

% kubectl -n kube-system get storagenodes -l name=portworx
NAME          ID                                     STATUS   VERSION           AGE
10.0.10.179   b4eca6dc-b598-49fe-8dc1-d4ff88537516   Online   2.8.1.5-937bfdd   52m
10.0.10.190   fcab60ca-d749-4762-a8df-bd8a3642422e   Online   2.8.1.5-937bfdd   52m
10.0.10.88    3f0f7d6e-7c9a-4af5-8499-2f9914a58491   Online   2.8.1.5-937bfdd   52m

Clean-Up

Finally, once you have finished with your OKE environment, you can use Terraform to clean-up your OCI compartment. for example

% terraform destroy --auto-approve
...
Destroy complete! Resources: 28 destroyed.

Summary

In this blog post I have demonstrated how Terraform can be used to provision an OCI and OKE resources and automate a Portworx deployment.

[twitter-follow screen_name=’RonEkins’ show_count=’yes’]

Leave a Reply

Create a website or blog at WordPress.com

Up ↑

%d bloggers like this: