Resizing Container File Systems with Kubernetes PVCs and Portworx

I have previous blogged on how you can run an Oracle 19c on Kubernetes with Portworx, if you want to check out the code examples you can find them here.

In this post I am going to show how we can use a Portworx Storage Class and Persistent Volume Claims (PVCs) to resize a containers file system.

StorageClass

Before we start let’s confirm our StorageClass supports volume expansion, we can do this with kubectl get StorageClass

[root@master-1 ~]# kubectl get StorageClass/px-ora-sc  
NAME      PROVISIONER                   RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
px-ora-sc kubernetes.io/portworx-volume Delete        Immediate           true                 24h

If ALLOWVOLUMEEXPANSION is set to false this can be updated with kubectl edit StorageClass.

Persistent Volume Claims

My Oracle container is using three PVCs for data, setup and startup, for this blog I will resize the ora-data193 file system.

Using kubectl get pvc we check the name, size and storageclass of our persistent volume claims.

[root@master-1 ~]# kubectl get pvc -n oracle-namespace -o wide -l version=19.3.0.1
 NAME                         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE   VOLUMEMODE
ora-data193-oracle19c-0      Bound    pvc-f275fa8e-a488-4506-9032-fca99a7f3b54   10Gi       RWO            px-ora-sc      24h   Filesystem
ora-setup193-oracle19c-0     Bound    pvc-9af1cb3a-e28d-47af-a8d6-f9f16256320e   1Gi        RWO            px-ora-sc      24h   Filesystem
ora-startup193-oracle19c-0   Bound    pvc-4ba8d1db-e45a-4718-9750-773421a5d66b   1Gi        RWO            px-ora-sc      24h   Filesystem

Running Pods

And now let’s find out where our database container is running.

[root@master-1 ~]# kubectl get pods -n oracle-namespace -o wide
NAME          READY   STATUS    RESTARTS   AGE   IP           NODE       
oracle19c-0   1/1     Running   0          23h   10.244.3.3   node-1-7   

From the above I can see my oracle19c-0 container is running on node-1-7 and is using a 10GB PVC called pvc-f275fa8e-a488-4506-9032-fca99a7f3b54.

Now let’s have a look from Portworx

View from Portworx

From one of my worker nodes I can use the Portworx pxctl volume list command to get the volume name and size.

[root@node-1-7 ~]# pxctl volume list
ID NAME SIZE HA SHARED ENCRYPTED IO_PRIORITY STATUS SNAP-ENABLED
...
1149398223018332021 pvc-f275fa8e-a488-4506-9032-fca99a7f3b54 10 GiB 3 no no HIGH up - attached on 10.225.115.122 no

Volume Inspect

The pxctl volume inspect command can be used to provide further details.

[root@node-1-7 ~]# pxctl volume inspect pvc-f275fa8e-a488-4506-9032-fca99a7f3b54
Volume :  1149398223018332021
Name              :  pvc-f275fa8e-a488-4506-9032-fca99a7f3b54
Size              :  10 GiB
Format            :  ext4
HA                :  3
IO Priority       :  HIGH
Creation time     :  Nov 12 10:30:23 UTC 2020
Shared            :  no
Status            :  up
State             :  Attached: 5b6cc1b1-6ffe-470b-a1b1-2bdc0043791c (10.225.115.122)
Device Path       :  /dev/pxd/pxd1149398223018332021
Labels            :  app=database,io_profile=db,namespace=oracle-namespace,priority_io=high,pvc=ora-data193-oracle19c-0,repl=3,version=19.3.0.1
Reads             :  129106
Reads MS          :  1185566
Bytes Read        :  9997258752
Writes            :  706272
Writes MS         :  18843130
Bytes Written     :  46046056448
IOs in progress   :  0
Bytes used        :  8.1 GiB
Replica sets on nodes:
Set 0
  Node   : 10.225.115.116 (Pool d73e98c8-528e-41e3-b3d8-f87cbc57850e )
  Node   : 10.225.115.119 (Pool 75d745b2-965c-4126-8912-3b6e944bb1b3 )
  Node   : 10.225.115.122 (Pool 16860c64-8d6d-406c-84c8-6ab5ee7a1e05 )
Replication Status  :  Up
Volume consumers  : 
 - Name           : oracle19c-0 (13af91c5-784c-4366-a434-094ccdb34d7f) (Pod)
  Namespace      : oracle-namespace
  Running on     : node-1-7
  Controlled by  : oracle19c (StatefulSet)

Check Container Linux File System

Finally before we resize our PVC, let’s shell into our Oracle19c container and use df -h to check the file system.

[root@master-1 ~]# kubectl exec -it pod/oracle19c-0 -n oracle-namespace -- /bin/bash
[oracle@oracle19c-0 ~]$ df -h
Filesystem                       Size  Used Avail Use% Mounted on
...
/dev/pxd/pxd1149398223018332021  9.8G  8.1G  1.2G  88% /opt/oracle/oradata

Resize File System

From the above I can my Oracle database is using a 10GB PVC and it is currently 88% utilised, let’s increase the file system size to 15GB.

Edit Persistent Volume Claim (PVC)

To resize our Portworx Persistent Volume Claim, we can simply edit the pvc specification and update the size using kubectl edit pvc.

[root@master-1 ~]# kubectl edit pvc/ora-data193-oracle19c-0 -n oracle-namespace
..
spec:
   accessModes:
   - ReadWriteOnce
   resources:
     requests:
       storage: 15Gi
   storageClassName: px-ora-sc
   volumeMode: Filesystem
   volumeName: pvc-f275fa8e-a488-4506-9032-fca99a7f3b54
...

Let’s check the resize with kubectl get pvc

[root@master-1 ~]# kubectl get pvc/ora-data193-oracle19c-0 -n oracle-namespace
NAME                      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
ora-data193-oracle19c-0   Bound    pvc-f275fa8e-a488-4506-9032-fca99a7f3b54   15Gi       RWO            px-ora-sc      25h

The Kubernetes pvc resize is reported as event as we can see this with kubectl describe pvc.

[root@master-1 ~]# kubectl describe pvc/ora-data193-oracle19c-0 -n oracle-namespace
 Name:          ora-data193-oracle19c-0
 Namespace:     oracle-namespace
 StorageClass:  px-ora-sc
 Status:        Bound
 Volume:        pvc-f275fa8e-a488-4506-9032-fca99a7f3b54
 Labels:        app=database
                version=19.3.0.1
 Annotations:   pv.kubernetes.io/bind-completed: yes
                pv.kubernetes.io/bound-by-controller: yes
                volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/portworx-volume
                volume.kubernetes.io/storage-resizer: kubernetes.io/portworx-volume
 Finalizers:    [kubernetes.io/pvc-protection]
 Capacity:      15Gi
 Access Modes:  RWO
 VolumeMode:    Filesystem
 Mounted By:    oracle19c-0
 Events:
   Type    Reason                  Age    From           Message
   ----    ------                  ----   ----           -------
   Normal  VolumeResizeSuccessful  2m11s  volume_expand  ExpandVolume succeeded for volume oracle-namespace/ora-data193-oracle19c-0

Portworx Volume Check

Let’s repeat the pxctl volume list to check the size increase is being reported.

[root@node-1-7 ~]# pxctl volume list
ID NAME SIZE HA SHARED ENCRYPTED IO_PRIORITY STATUS SNAP-ENABLED
...
1149398223018332021 pvc-f275fa8e-a488-4506-9032-fca99a7f3b54 15 GiB 3 no no HIGH up - attached on 10.225.115.122 no

Linux File System Check

Let’s return to our Oracle container pod and again use df -h to check the file system.

[oracle@oracle19c-0 ~]$ df -h 
Filesystem                       Size  Used Avail Use% Mounted on
...
/dev/pxd/pxd1149398223018332021   15G  8.1G  5.9G  58% /opt/oracle/oradata

Summary

In this blog I have shown how we can use a Kubernetes Persistent Volume Claims and a Portworx StorageClass to resize an Linux container file system.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Create a website or blog at WordPress.com

Up ↑

%d bloggers like this: