In this post I will show how we can perform an on-line, manual Portworx storage pool expansion.
In my lab I have a 7 node Kubernetes cluster, with 7 storage nodes, for the purposes of this blog post I will resize storage node node-1-4 from 64GB to 250GB.
First, let’s have a look at the Kubernetes worker nodes using kubectl get nodes.
[root@master-1 ~]# kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME master-1 Ready control-plane,master 72d v1.21.3 10.225.115.150 <none> CentOS Linux 7 (Core) 3.10.0-1160.45.1.el7.x86_64 docker://1.13.1 node-1-1 Ready <none> 72d v1.21.3 10.225.115.151 <none> CentOS Linux 7 (Core) 3.10.0-1160.45.1.el7.x86_64 docker://1.13.1 node-1-2 Ready <none> 72d v1.21.3 10.225.115.154 <none> CentOS Linux 7 (Core) 3.10.0-1160.45.1.el7.x86_64 docker://1.13.1 node-1-3 Ready <none> 72d v1.21.3 10.225.115.155 <none> CentOS Linux 7 (Core) 3.10.0-1160.45.1.el7.x86_64 docker://1.13.1 node-1-4 Ready <none> 72d v1.21.3 10.225.115.158 <none> CentOS Linux 7 (Core) 3.10.0-1160.45.1.el7.x86_64 docker://1.13.1 node-1-5 Ready <none> 72d v1.21.3 10.225.115.157 <none> CentOS Linux 7 (Core) 3.10.0-1160.45.1.el7.x86_64 docker://1.13.1 node-1-6 Ready <none> 72d v1.21.3 10.225.115.152 <none> CentOS Linux 7 (Core) 3.10.0-1160.45.1.el7.x86_64 docker://1.13.1 node-1-7 Ready <none> 72d v1.21.3 10.225.115.153 <none> CentOS Linux 7 (Core) 3.10.0-1160.45.1.el7.x86_64 docker://1.13.1
And use pxctl status to list Portworx storage nodes.
[root@master-1 ~]# pxctl status Defaulted container "portworx" out of: portworx, csi-node-driver-registrar Status: PX is operational Telemetry: Disabled or Unhealthy License: PX-Enterprise extended eval,VM limited (expires in 56 days ; NOTICE: Number of nodes maximum reduces by 7 on 2022-01-18) Node ID: c453a2fe-ba4a-4e3b-960d-c03e6341de62 IP: 10.225.115.158 Local Storage Pool: 1 pool POOL IO_PRIORITY RAID_LEVEL USABLE USED STATUS ZONE REGION 0 HIGH raid0 64 GiB 28 GiB Online default default Local Storage Devices: 1 device Device Path Media Type Size Last-Scan 0:1 /dev/sdb STORAGE_MEDIUM_MAGNETIC 64 GiB 10 Nov 21 09:51 UTC total - 64 GiB Cache Devices: * No cache devices Cluster Summary Cluster ID: px-deploy-1 Cluster UUID: 5598b98f-f585-4fa9-a281-0b458557f225 Scheduler: kubernetes Nodes: 7 node(s) with storage (7 online) IP ID SchedulerNodeName Auth StorageNode Used Capacity Status StorageStatus Version Kernel OS 10.225.115.154 e2fa8f7d-2be0-4436-9306-8b6094dcbc82 node-1-2 Disabled Yes 28 GiB 64 GiB Online Up 2.8.1.2-c9b840b 3.10.0-1160.45.1.el7.x86_64 CentOS Linux 7 (Core) 10.225.115.157 cfa4bf3a-af39-48c5-b421-0c85e142caa6 node-1-5 Disabled Yes 5.9 GiB 64 GiB Online Up 2.8.1.2-c9b840b 3.10.0-1160.45.1.el7.x86_64 CentOS Linux 7 (Core) 10.225.115.158 c453a2fe-ba4a-4e3b-960d-c03e6341de62 node-1-4 Disabled Yes 28 GiB 64 GiB Online Up (This node) 2.8.1.2-c9b840b 3.10.0-1160.45.1.el7.x86_64 CentOS Linux 7 (Core) 10.225.115.152 bb658f53-92ab-46bc-a817-b7088e25989b node-1-6 Disabled Yes 7.6 GiB 64 GiB Online Up 2.8.1.2-c9b840b 3.10.0-1160.45.1.el7.x86_64 CentOS Linux 7 (Core) 10.225.115.155 639a8f40-5f39-4c82-a6be-715b44145783 node-1-3 Disabled Yes 28 GiB 64 GiB Online Up 2.8.1.2-c9b840b 3.10.0-1160.45.1.el7.x86_64 CentOS Linux 7 (Core) 10.225.115.153 3f633a0e-3b60-46a5-a4a3-c2a0aa79eec0 node-1-7 Disabled Yes 6.8 GiB 64 GiB Online Up 2.8.1.2-c9b840b 3.10.0-1160.45.1.el7.x86_64 CentOS Linux 7 (Core) 10.225.115.151 2f48a8ad-3190-4038-abe5-54aa279b4a00 node-1-1 Disabled Yes 5.0 GiB 64 GiB Online Up 2.8.1.2-c9b840b 3.10.0-1160.45.1.el7.x86_64 CentOS Linux 7 (Core) Warnings: WARNING: Persistent journald logging is not enabled on this node. Global Storage Pool Total Used : 110 GiB Total Capacity : 448 GiB
To manually resize the disk on node-1-4 we need to obtain the disk UUID, this can be done with pxctl cluster provision-status, for example.
[root@master-1 ~]# pxctl cluster provision-status --output-type wide Defaulted container "portworx" out of: portworx, csi-node-driver-registrar NODE NODE STATUS POOL POOL STATUS IO_PRIORITY SIZE AVAILABLUSED (MEAN-DIFF %) PROVISIONED (MEAN-DIFF %) ZONE REGION RACK e2fa8f7d-2be0-4436-9306-8b6094dcbc82 Up 0 ( 6674d3d7-a576-4811-884c-d98cf8b99f82 ) Online HIGH 64 GiB 36 GiB 28 GiB ( +20 % ) 25 GiB ( +19 % ) default default default 2f48a8ad-3190-4038-abe5-54aa279b4a00 Up 0 ( 348254cc-c127-420e-b145-57966888dad8 ) Online HIGH 64 GiB 59 GiB 5.0 GiB ( -17 % ) 2.0 GiB ( -17 % ) default default default 3f633a0e-3b60-46a5-a4a3-c2a0aa79eec0 Up 0 ( 7e422d13-1771-44a4-913d-a1ad32211d1c ) Online HIGH 64 GiB 57 GiB 6.8 GiB ( -14 % ) 5.0 GiB ( -13 % ) default default default 639a8f40-5f39-4c82-a6be-715b44145783 Up 0 ( 0ad5dfef-36b3-4a72-a48b-2aa401003ada ) Online HIGH 64 GiB 36 GiB 28 GiB ( +20 % ) 25 GiB ( +19 % ) default default default bb658f53-92ab-46bc-a817-b7088e25989b Up 0 ( 3e5b6b7a-dea0-4a48-983e-1b8a883678a6 ) Online HIGH 64 GiB 56 GiB 7.6 GiB ( -13 % ) 7.0 GiB ( -10 % ) default default default c453a2fe-ba4a-4e3b-960d-c03e6341de62 Up 0 ( fed3d62e-42db-4dbc-884b-dc02af721498 ) Online HIGH 64 GiB 36 GiB 28 GiB ( +20 % ) 25 GiB ( +19 % ) default default default cfa4bf3a-af39-48c5-b421-0c85e142caa6 Up 0 ( bafe5139-78a3-4c16-a9b7-a7d158c2cf71 ) Online HIGH 64 GiB 58 GiB 5.9 GiB ( -15 % ) 4.0 GiB ( -14 % ) default default default
Before the resize, we can see storage pool node-1-4 has 64GB of storage
[root@master-1 ~]# pxctl service pool show Defaulted container "portworx" out of: portworx, csi-node-driver-registrar PX drive configuration: Pool ID: 0 UUID: fed3d62e-42db-4dbc-884b-dc02af721498 IO Priority: HIGH Labels: kubernetes.io/os=linux,medium=STORAGE_MEDIUM_MAGNETIC,beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,iopriority=HIGH,kubernetes.io/arch=amd64,kubernetes.io/hostname=node-1-4 Size: 64 GiB Status: Online Has metadata: Yes Balanced: Yes Drives: 1: /dev/sdb, Total size 64 GiB, Online Cache Drives: No Cache drives found in this pool
Manual Pool Expansion
To manually expand the storage pool disk use pxctl service pool expand -u <uid string> -s <size>-o resize-disk command, for example
[root@master-1 ~]# pxctl service pool expand -u fed3d62e-42db-4dbc-884b-dc02af721498 -s 250 -o resize-disk Defaulted container "portworx" out of: portworx, csi-node-driver-registrar Request to expand pool: fed3d62e-42db-4dbc-884b-dc02af721498 to size: 250 using operation: resize-disk Pool resize triggered successfully.
During the resize operation we can use pxctl service pool show to check LastOperation Status
[root@master-1 ~]# pxctl service pool show Defaulted container "portworx" out of: portworx, csi-node-driver-registrar PX drive configuration: Pool ID: 0 UUID: fed3d62e-42db-4dbc-884b-dc02af721498 IO Priority: HIGH Labels: medium=STORAGE_MEDIUM_MAGNETIC,beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,iopriority=HIGH,kubernetes.io/arch=amd64,kubernetes.io/hostname=node-1-4,kubernetes.io/os=linux Size: 64 GiB Status: Online Has metadata: Yes Balanced: Yes Drives: 1: /dev/sdb, Total size 64 GiB, Online LastOperation OPERATION_RESIZE Status: OPERATION_PENDING Message: resize request from 68719476736 to 268435456000 has been received Cache Drives: No Cache drives found in this pool
After a few minutes repeat the check to see if disk resize operation has completed again using pxctl service pool show.
[root@master-1 ~]# pxctl service pool show Defaulted container "portworx" out of: portworx, csi-node-driver-registrar PX drive configuration: Pool ID: 0 UUID: fed3d62e-42db-4dbc-884b-dc02af721498 IO Priority: HIGH Labels: medium=STORAGE_MEDIUM_MAGNETIC,beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,iopriority=HIGH,kubernetes.io/arch=amd64,kubernetes.io/hostname=node-1-4,kubernetes.io/os=linux Size: 250 GiB Status: Online Has metadata: Yes Balanced: Yes Drives: 1: /dev/sdb, Total size 250 GiB, Online LastOperation OPERATION_RESIZE Status: OPERATION_SUCCESSFUL Message: pool expansion to 250 GiB completed successfully Cache Drives: No Cache drives found in this pool
The Portworx storage pool size has now been increased and we can confirm this with pxctl status command, for example.
[root@master-1 ~]# pxctl status Defaulted container "portworx" out of: portworx, csi-node-driver-registrar Status: PX is operational Telemetry: Disabled or Unhealthy License: PX-Enterprise extended eval,VM limited (expires in 56 days ; NOTICE: Number of nodes maximum reduces by 7 on 2022-01-18) Node ID: c453a2fe-ba4a-4e3b-960d-c03e6341de62 IP: 10.225.115.158 Local Storage Pool: 1 pool POOL IO_PRIORITY RAID_LEVEL USABLE USED STATUS ZONE REGION 0 HIGH raid0 250 GiB 28 GiB Online default default Local Storage Devices: 1 device Device Path Media Type Size Last-Scan 0:1 /dev/sdb STORAGE_MEDIUM_MAGNETIC 250 GiB 21 Jan 22 13:48 UTC total - 250 GiB Cache Devices: * No cache devices Cluster Summary Cluster ID: px-deploy-1 Cluster UUID: 5598b98f-f585-4fa9-a281-0b458557f225 Scheduler: kubernetes Nodes: 7 node(s) with storage (7 online) IP ID SchedulerNodeName Auth StorageNode Used Capacity Status StorageStatus Version Kernel OS 10.225.115.154 e2fa8f7d-2be0-4436-9306-8b6094dcbc82 node-1-2 Disabled Yes 28 GiB 64 GiB Online Up 2.8.1.2-c9b840b 3.10.0-1160.45.1.el7.x86_64 CentOS Linux 7 (Core) 10.225.115.157 cfa4bf3a-af39-48c5-b421-0c85e142caa6 node-1-5 Disabled Yes 5.9 GiB 64 GiB Online Up 2.8.1.2-c9b840b 3.10.0-1160.45.1.el7.x86_64 CentOS Linux 7 (Core) 10.225.115.158 c453a2fe-ba4a-4e3b-960d-c03e6341de62 node-1-4 Disabled Yes 28 GiB 250 GiB Online Up (This node) 2.8.1.2-c9b840b 3.10.0-1160.45.1.el7.x86_64 CentOS Linux 7 (Core) 10.225.115.152 bb658f53-92ab-46bc-a817-b7088e25989b node-1-6 Disabled Yes 7.6 GiB 64 GiB Online Up 2.8.1.2-c9b840b 3.10.0-1160.45.1.el7.x86_64 CentOS Linux 7 (Core) 10.225.115.155 639a8f40-5f39-4c82-a6be-715b44145783 node-1-3 Disabled Yes 28 GiB 64 GiB Online Up 2.8.1.2-c9b840b 3.10.0-1160.45.1.el7.x86_64 CentOS Linux 7 (Core) 10.225.115.153 3f633a0e-3b60-46a5-a4a3-c2a0aa79eec0 node-1-7 Disabled Yes 6.8 GiB 64 GiB Online Up 2.8.1.2-c9b840b 3.10.0-1160.45.1.el7.x86_64 CentOS Linux 7 (Core) 10.225.115.151 2f48a8ad-3190-4038-abe5-54aa279b4a00 node-1-1 Disabled Yes 5.0 GiB 64 GiB Online Up 2.8.1.2-c9b840b 3.10.0-1160.45.1.el7.x86_64 CentOS Linux 7 (Core) Warnings: WARNING: Persistent journald logging is not enabled on this node. Global Storage Pool Total Used : 110 GiB Total Capacity : 634 GiB
Summary
In this blog I have shared how we can perform an online Portworx storage pool expansion using the Portworx command line interface pxctl.
[twitter-follow screen_name=’RonEkins’ show_count=’yes’]
Leave a Reply