Introduction
I frequently have the need to stand-up Kubernetes lab environments for R&D, learning and to develop blogs and solutions, Oracle Cloud Infrastructure (OCI) provides a great wizard to enable the rapid deployment of Oracle Kubernetes Engine (OKE) clusters.
However, if you find yourself performing the same task often you should consider automating the process, Oracle Cloud provides the OCI command line interface which can be used, however Terraform by HasiCorp should be top of your list for consideration.
The Oracle Cloud Infrastructure (OCI) Terraform provider enables us to use Terraform to interact with Oracle Cloud Infrastructure (OCI) resources and data sources.
I have previously shared how we can deploy an OKE environment with Terraform, in this post I will show how we can automate the provisioning of an Oracle Cloud Kubernetes environment on OCI with block storage configured for use by Portworx using Terraform.
Getting Started
Start by visiting https://central.portworx.com/ and use the Portworx wizard to create a specification for either Portworx Essentials or Portworx Enterprise.
If you have never previously used the Portworx specification Wizard, you may want to read my post Portworx installation on OCI which provides a detailed walkthrough including the creation of a Portworx specification.
Once you have followed the wizard, copy and save the deployment specification URLs as they we will be required in our Terraform terraform.tfvars file.
The Terraform HashiCorp Configuration Language (HCL) script creates a Kubernetes configuration file in .kube/config (see kubeconfig.tf), to enable direct desktop access using kubectl add this location to your KUBECONFIG, for example:
export KUBECONFIG=${KUBECONFIG}:/Users/rekins/myTerraform/oke-px-terrafom/.kube/configGit Repo
Clone the Terraform code from my GitHub repo, change directory, localise the terraform.tfvars file providing OCI account, tenancy, and Portworx URLs.
- git clone https://github.com/raekins/oke-px-terraform.git
- cd oke-px-terraform
- cp terraform.tfvars.example terraform.tfvars
- vi terraform.tfvars
OKE Architecture
The Terraform HCL script builds a three node OKE cluster, with a compute instance in each of the three Availability Domains (ADs) and an attached dedicated block device for Portworx.
The Terraform HCL script does not just automate the creation of a Kubernetes cluster but also the automates the creation of other used OCI resources, including:
- Virtual Cloud Network (VCN)
- Subnets
- Security Lists
- Route Tables
- Intenet Gateway
- Service Gateway
- NAT Gateway
- Oracle Kubernetes Engine (OKE)
- Node Pool
- Kubernetes ConfigFile
- Block Volumes
- Attachment of block volumes to compute instances
- Bastion Host
- Portworx Operator
Terraform Deployment
Download, install Terraform, and confirm installation and version, for example
% terraform version Terraform v1.1.7 on darwin_amd64
Change directory to the cloned repo, and deploy using Terraform for example.
- terraform init
- terraform validate
- terraform plan
If all ok, now use terraform apply to deploy.
% terraform apply --auto-approve ... Apply complete! Resources: 28 added, 0 changed, 0 destroyed. Outputs: all-availability-domains-in-your-tenancy = tolist([ "iXvO:UK-LONDON-1-AD-1", "iXvO:UK-LONDON-1-AD-2", "iXvO:UK-LONDON-1-AD-3", ]) all-running-oke-instance-names = tolist([ "oke-cumadkmup7q-nyu576ijn5q-sma4gmd7eca-2", "oke-cumadkmup7q-nyu576ijn5q-sma4gmd7eca-1", "oke-cumadkmup7q-nyu576ijn5q-sma4gmd7eca-0", ]) containerengine-cluster-kubernetes_version = "v1.20.11" containerengine-cluster-name = "pxcluster" containerengine-cluster-public-endpoint = "144.21.57.50:6443" selected-core-services = tolist([ "All LHR Services In Oracle Services Network", ])
From the above, you can see that Terraform created a three node private OKE cluster across three Availability Domains (ADs), with a public K8 API endpoint.
Oracle Kubernetes Engine
To validate the deployment logon to OCI, navigate using the hamburger menu to Developer Services -> Kubernetes Clusters (OKE).
Select the cluster, then node pool and confirm Availability Domain Placements, Kubernetes Node Condition and Versions, Kubernetes labels, for example.
Kubernetes Nodes
We can also check the OKE environment using kubectl, for example kubectl get nodes.
% kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME 10.0.10.179 Ready node 7m32s v1.20.11 10.0.10.179 <none> Oracle Linux Server 7.8 4.14.35-1902.306.2.el7uek.x86_64 cri-o://1.20.2 10.0.10.190 Ready node 7m37s v1.20.11 10.0.10.190 <none> Oracle Linux Server 7.8 4.14.35-1902.306.2.el7uek.x86_64 cri-o://1.20.2 10.0.10.88 Ready node 6m52s v1.20.11 10.0.10.88 <none> Oracle Linux Server 7.8 4.14.35-1902.306.2.el7uek.x86_64 cri-o://1.20.2
Portworx Version
The Terraform HCL script also deploys Portworx, we can use pxctl -v to check version, or pxctl status to see overall status of the Portworx cluster, for example.
% PX_POD=$(kubectl get pods -l name=portworx -n kube-system -o jsonpath='{.items[0].metadata.name}')
% kubectl exec -it $PX_POD -n kube-system -- /opt/pwx/bin/pxctl -v
Defaulting container name to portworx.
Use 'kubectl describe pod/px-cluster-5ad151aa-784e-4103-9ab5-a3529b18c12f-5l9vc -n kube-system' to see all of the containers in this pod.
pxctl version 2.8.1.5-937bfdd
Kubernetes Storage Classes
The Portworx Operator creates a number of new Kubernetes storage classes, we can see these and the two OCI storage classes using kubectl get storageclass or kubectl get sc.
% kubectl get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE oci (default) oracle.com/oci Delete Immediate false 12m oci-bv blockvolume.csi.oraclecloud.com Delete WaitForFirstConsumer true 12m px-db kubernetes.io/portworx-volume Delete Immediate true 4m48s px-db-cloud-snapshot kubernetes.io/portworx-volume Delete Immediate true 4m48s px-db-cloud-snapshot-encrypted kubernetes.io/portworx-volume Delete Immediate true 4m48s px-db-encrypted kubernetes.io/portworx-volume Delete Immediate true 4m48s px-db-local-snapshot kubernetes.io/portworx-volume Delete Immediate true 4m48s px-db-local-snapshot-encrypted kubernetes.io/portworx-volume Delete Immediate true 4m48s px-replicated kubernetes.io/portworx-volume Delete Immediate true 4m48s px-replicated-encrypted kubernetes.io/portworx-volume Delete Immediate true 4m48s stork-snapshot-sc stork-snapshot Delete Immediate true 4m48s
Portworx StorageCluster
Using kubectl get storagecluster we can see the Portworx cluster status and version.
% kubectl get storagecluster -n kube-system NAME CLUSTER UUID STATUS VERSION AGE px-cluster-5ad151aa-784e-4103-9ab5-a3529b18c12f b27154ec-6ff4-498b-87ed-7fa8349d9f54 Online 2.8.1.5 46h
Portworx Storage Nodes
The Terraform code has deployed three Portworx Storage nodes, we can see these using kubectl get storage nodes for example.
% kubectl -n kube-system get storagenodes -l name=portworx NAME ID STATUS VERSION AGE 10.0.10.179 b4eca6dc-b598-49fe-8dc1-d4ff88537516 Online 2.8.1.5-937bfdd 52m 10.0.10.190 fcab60ca-d749-4762-a8df-bd8a3642422e Online 2.8.1.5-937bfdd 52m 10.0.10.88 3f0f7d6e-7c9a-4af5-8499-2f9914a58491 Online 2.8.1.5-937bfdd 52m
Clean-Up
Finally, once you have finished with your OKE environment, you can use Terraform to clean-up your OCI compartment. for example
% terraform destroy --auto-approve ... Destroy complete! Resources: 28 destroyed.
Summary
In this blog post I have demonstrated how Terraform can be used to provision an OCI and OKE resources and automate a Portworx deployment.
[twitter-follow screen_name=’RonEkins’ show_count=’yes’]