How to manually expand a Portworx Storage Pool

In this post I will show how we can perform an on-line, manual Portworx storage pool expansion.

In my lab I have a 7 node Kubernetes cluster, with 7 storage nodes, for the purposes of this blog post I will resize storage node node-1-4 from 64GB to 250GB.

First, let’s have a look at the Kubernetes worker nodes using kubectl get nodes.

[root@master-1 ~]# kubectl get nodes -o wide
NAME       STATUS   ROLES                  AGE   VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION                CONTAINER-RUNTIME
master-1   Ready    control-plane,master   72d   v1.21.3   10.225.115.150   <none>        CentOS Linux 7 (Core)   3.10.0-1160.45.1.el7.x86_64   docker://1.13.1
node-1-1   Ready    <none>                 72d   v1.21.3   10.225.115.151   <none>        CentOS Linux 7 (Core)   3.10.0-1160.45.1.el7.x86_64   docker://1.13.1
node-1-2   Ready    <none>                 72d   v1.21.3   10.225.115.154   <none>        CentOS Linux 7 (Core)   3.10.0-1160.45.1.el7.x86_64   docker://1.13.1
node-1-3   Ready    <none>                 72d   v1.21.3   10.225.115.155   <none>        CentOS Linux 7 (Core)   3.10.0-1160.45.1.el7.x86_64   docker://1.13.1
node-1-4   Ready    <none>                 72d   v1.21.3   10.225.115.158   <none>        CentOS Linux 7 (Core)   3.10.0-1160.45.1.el7.x86_64   docker://1.13.1
node-1-5   Ready    <none>                 72d   v1.21.3   10.225.115.157   <none>        CentOS Linux 7 (Core)   3.10.0-1160.45.1.el7.x86_64   docker://1.13.1
node-1-6   Ready    <none>                 72d   v1.21.3   10.225.115.152   <none>        CentOS Linux 7 (Core)   3.10.0-1160.45.1.el7.x86_64   docker://1.13.1
node-1-7   Ready    <none>                 72d   v1.21.3   10.225.115.153   <none>        CentOS Linux 7 (Core)   3.10.0-1160.45.1.el7.x86_64   docker://1.13.1

And use pxctl status to list Portworx storage nodes.

[root@master-1 ~]# pxctl status
Defaulted container "portworx" out of: portworx, csi-node-driver-registrar
Status: PX is operational
Telemetry: Disabled or Unhealthy
License: PX-Enterprise extended eval,VM limited (expires in 56 days ; NOTICE: Number of nodes maximum reduces by 7 on 2022-01-18)
Node ID: c453a2fe-ba4a-4e3b-960d-c03e6341de62
	IP: 10.225.115.158 
 	Local Storage Pool: 1 pool
	POOL	IO_PRIORITY	RAID_LEVEL	USABLE	USED	STATUS	ZONE	REGION
	0	HIGH		raid0		64 GiB	28 GiB	Online	default	default
	Local Storage Devices: 1 device
	Device	Path		Media Type		Size		Last-Scan
	0:1	/dev/sdb	STORAGE_MEDIUM_MAGNETIC	64 GiB		10 Nov 21 09:51 UTC
	total			-			64 GiB
	Cache Devices:
	 * No cache devices
Cluster Summary
	Cluster ID: px-deploy-1
	Cluster UUID: 5598b98f-f585-4fa9-a281-0b458557f225
	Scheduler: kubernetes
	Nodes: 7 node(s) with storage (7 online)
	IP		ID					SchedulerNodeName	Auth		StorageNode	Used	Capacity	Status	StorageStatus	Version		Kernel				OS
	10.225.115.154	e2fa8f7d-2be0-4436-9306-8b6094dcbc82	node-1-2		Disabled	Yes		28 GiB	64 GiB		Online	Up		2.8.1.2-c9b840b	3.10.0-1160.45.1.el7.x86_64	CentOS Linux 7 (Core)
	10.225.115.157	cfa4bf3a-af39-48c5-b421-0c85e142caa6	node-1-5		Disabled	Yes		5.9 GiB	64 GiB		Online	Up		2.8.1.2-c9b840b	3.10.0-1160.45.1.el7.x86_64	CentOS Linux 7 (Core)
	10.225.115.158	c453a2fe-ba4a-4e3b-960d-c03e6341de62	node-1-4		Disabled	Yes		28 GiB	64 GiB		Online	Up (This node)	2.8.1.2-c9b840b	3.10.0-1160.45.1.el7.x86_64	CentOS Linux 7 (Core)
	10.225.115.152	bb658f53-92ab-46bc-a817-b7088e25989b	node-1-6		Disabled	Yes		7.6 GiB	64 GiB		Online	Up		2.8.1.2-c9b840b	3.10.0-1160.45.1.el7.x86_64	CentOS Linux 7 (Core)
	10.225.115.155	639a8f40-5f39-4c82-a6be-715b44145783	node-1-3		Disabled	Yes		28 GiB	64 GiB		Online	Up		2.8.1.2-c9b840b	3.10.0-1160.45.1.el7.x86_64	CentOS Linux 7 (Core)
	10.225.115.153	3f633a0e-3b60-46a5-a4a3-c2a0aa79eec0	node-1-7		Disabled	Yes		6.8 GiB	64 GiB		Online	Up		2.8.1.2-c9b840b	3.10.0-1160.45.1.el7.x86_64	CentOS Linux 7 (Core)
	10.225.115.151	2f48a8ad-3190-4038-abe5-54aa279b4a00	node-1-1		Disabled	Yes		5.0 GiB	64 GiB		Online	Up		2.8.1.2-c9b840b	3.10.0-1160.45.1.el7.x86_64	CentOS Linux 7 (Core)
	Warnings: 
		 WARNING: Persistent journald logging is not enabled on this node.
Global Storage Pool
	Total Used    	:  110 GiB
	Total Capacity	:  448 GiB

To manually resize the disk on node-1-4 we need to obtain the disk UUID, this can be done with pxctl cluster provision-status, for example.

[root@master-1 ~]# pxctl cluster provision-status --output-type wide
Defaulted container "portworx" out of: portworx, csi-node-driver-registrar
NODE					NODE STATUS	POOL						POOL STATUS	IO_PRIORITY	SIZE	AVAILABLUSED (MEAN-DIFF %)	PROVISIONED (MEAN-DIFF %)	ZONE	REGION	RACK
e2fa8f7d-2be0-4436-9306-8b6094dcbc82	Up		0 ( 6674d3d7-a576-4811-884c-d98cf8b99f82 )	Online		HIGH		64 GiB	36 GiB	28 GiB ( +20 % )	25 GiB ( +19 % )		default	default	default
2f48a8ad-3190-4038-abe5-54aa279b4a00	Up		0 ( 348254cc-c127-420e-b145-57966888dad8 )	Online		HIGH		64 GiB	59 GiB	5.0 GiB ( -17 % )	2.0 GiB ( -17 % )		default	default	default
3f633a0e-3b60-46a5-a4a3-c2a0aa79eec0	Up		0 ( 7e422d13-1771-44a4-913d-a1ad32211d1c )	Online		HIGH		64 GiB	57 GiB	6.8 GiB ( -14 % )	5.0 GiB ( -13 % )		default	default	default
639a8f40-5f39-4c82-a6be-715b44145783	Up		0 ( 0ad5dfef-36b3-4a72-a48b-2aa401003ada )	Online		HIGH		64 GiB	36 GiB	28 GiB ( +20 % )	25 GiB ( +19 % )		default	default	default
bb658f53-92ab-46bc-a817-b7088e25989b	Up		0 ( 3e5b6b7a-dea0-4a48-983e-1b8a883678a6 )	Online		HIGH		64 GiB	56 GiB	7.6 GiB ( -13 % )	7.0 GiB ( -10 % )		default	default	default
c453a2fe-ba4a-4e3b-960d-c03e6341de62	Up		0 ( fed3d62e-42db-4dbc-884b-dc02af721498 )	Online		HIGH		64 GiB	36 GiB	28 GiB ( +20 % )	25 GiB ( +19 % )		default	default	default
cfa4bf3a-af39-48c5-b421-0c85e142caa6	Up		0 ( bafe5139-78a3-4c16-a9b7-a7d158c2cf71 )	Online		HIGH		64 GiB	58 GiB	5.9 GiB ( -15 % )	4.0 GiB ( -14 % )		default	default	default

Before the resize, we can see storage pool node-1-4 has 64GB of storage

[root@master-1 ~]# pxctl service pool show
Defaulted container "portworx" out of: portworx, csi-node-driver-registrar
PX drive configuration:
Pool ID: 0 
	UUID:  fed3d62e-42db-4dbc-884b-dc02af721498 
	IO Priority:  HIGH 
	Labels:  kubernetes.io/os=linux,medium=STORAGE_MEDIUM_MAGNETIC,beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,iopriority=HIGH,kubernetes.io/arch=amd64,kubernetes.io/hostname=node-1-4 
	Size: 64 GiB 
	Status: Online 
	Has metadata:  Yes 
	Balanced:  Yes 
	Drives:
	1: /dev/sdb, Total size 64 GiB, Online
	Cache Drives:
	No Cache drives found in this pool

Manual Pool Expansion

To manually expand the storage pool disk use pxctl service pool expand -u <uid string> -s <size>-o resize-disk command, for example

[root@master-1 ~]#  pxctl service pool expand -u fed3d62e-42db-4dbc-884b-dc02af721498 -s 250 -o resize-disk
Defaulted container "portworx" out of: portworx, csi-node-driver-registrar
Request to expand pool: fed3d62e-42db-4dbc-884b-dc02af721498 to size: 250 using operation: resize-disk
Pool resize triggered successfully.

During the resize operation we can use pxctl service pool show to check LastOperation Status

[root@master-1 ~]# pxctl service pool show
Defaulted container "portworx" out of: portworx, csi-node-driver-registrar
PX drive configuration:
Pool ID: 0 
	UUID:  fed3d62e-42db-4dbc-884b-dc02af721498 
	IO Priority:  HIGH 
	Labels:  medium=STORAGE_MEDIUM_MAGNETIC,beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,iopriority=HIGH,kubernetes.io/arch=amd64,kubernetes.io/hostname=node-1-4,kubernetes.io/os=linux 
	Size: 64 GiB 
	Status: Online 
	Has metadata:  Yes 
	Balanced:  Yes 
	Drives:
	1: /dev/sdb, Total size 64 GiB, Online
	LastOperation  OPERATION_RESIZE 
		Status:  OPERATION_PENDING 
		Message: resize request from 68719476736 to 268435456000 has been received
	Cache Drives:
	No Cache drives found in this pool

After a few minutes repeat the check to see if disk resize operation has completed again using pxctl service pool show.

[root@master-1 ~]# pxctl service pool show
Defaulted container "portworx" out of: portworx, csi-node-driver-registrar
PX drive configuration:
Pool ID: 0 
	UUID:  fed3d62e-42db-4dbc-884b-dc02af721498 
	IO Priority:  HIGH 
	Labels:  medium=STORAGE_MEDIUM_MAGNETIC,beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,iopriority=HIGH,kubernetes.io/arch=amd64,kubernetes.io/hostname=node-1-4,kubernetes.io/os=linux 
	Size: 250 GiB 
	Status: Online 
	Has metadata:  Yes 
	Balanced:  Yes 
	Drives:
	1: /dev/sdb, Total size 250 GiB, Online
	LastOperation  OPERATION_RESIZE 
		Status:  OPERATION_SUCCESSFUL 
		Message: pool expansion to 250 GiB completed successfully
	Cache Drives:
	No Cache drives found in this pool

The Portworx storage pool size has now been increased and we can confirm this with pxctl status command, for example.

[root@master-1 ~]# pxctl status
Defaulted container "portworx" out of: portworx, csi-node-driver-registrar
Status: PX is operational
Telemetry: Disabled or Unhealthy
License: PX-Enterprise extended eval,VM limited (expires in 56 days ; NOTICE: Number of nodes maximum reduces by 7 on 2022-01-18)
Node ID: c453a2fe-ba4a-4e3b-960d-c03e6341de62
	IP: 10.225.115.158 
 	Local Storage Pool: 1 pool
	POOL	IO_PRIORITY	RAID_LEVEL	USABLE	USED	STATUS	ZONE	REGION
	0	HIGH		raid0		250 GiB	28 GiB	Online	default	default
	Local Storage Devices: 1 device
	Device	Path		Media Type		Size		Last-Scan
	0:1	/dev/sdb	STORAGE_MEDIUM_MAGNETIC	250 GiB		21 Jan 22 13:48 UTC
	total			-			250 GiB
	Cache Devices:
	 * No cache devices
Cluster Summary
	Cluster ID: px-deploy-1
	Cluster UUID: 5598b98f-f585-4fa9-a281-0b458557f225
	Scheduler: kubernetes
	Nodes: 7 node(s) with storage (7 online)
	IP		ID					SchedulerNodeName	Auth		StorageNode	Used	Capacity	Status	StorageStatus	Version		Kernel				OS
	10.225.115.154	e2fa8f7d-2be0-4436-9306-8b6094dcbc82	node-1-2		Disabled	Yes		28 GiB	64 GiB		Online	Up		2.8.1.2-c9b840b	3.10.0-1160.45.1.el7.x86_64	CentOS Linux 7 (Core)
	10.225.115.157	cfa4bf3a-af39-48c5-b421-0c85e142caa6	node-1-5		Disabled	Yes		5.9 GiB	64 GiB		Online	Up		2.8.1.2-c9b840b	3.10.0-1160.45.1.el7.x86_64	CentOS Linux 7 (Core)
	10.225.115.158	c453a2fe-ba4a-4e3b-960d-c03e6341de62	node-1-4		Disabled	Yes		28 GiB	250 GiB		Online	Up (This node)	2.8.1.2-c9b840b	3.10.0-1160.45.1.el7.x86_64	CentOS Linux 7 (Core)
	10.225.115.152	bb658f53-92ab-46bc-a817-b7088e25989b	node-1-6		Disabled	Yes		7.6 GiB	64 GiB		Online	Up		2.8.1.2-c9b840b	3.10.0-1160.45.1.el7.x86_64	CentOS Linux 7 (Core)
	10.225.115.155	639a8f40-5f39-4c82-a6be-715b44145783	node-1-3		Disabled	Yes		28 GiB	64 GiB		Online	Up		2.8.1.2-c9b840b	3.10.0-1160.45.1.el7.x86_64	CentOS Linux 7 (Core)
	10.225.115.153	3f633a0e-3b60-46a5-a4a3-c2a0aa79eec0	node-1-7		Disabled	Yes		6.8 GiB	64 GiB		Online	Up		2.8.1.2-c9b840b	3.10.0-1160.45.1.el7.x86_64	CentOS Linux 7 (Core)
	10.225.115.151	2f48a8ad-3190-4038-abe5-54aa279b4a00	node-1-1		Disabled	Yes		5.0 GiB	64 GiB		Online	Up		2.8.1.2-c9b840b	3.10.0-1160.45.1.el7.x86_64	CentOS Linux 7 (Core)
	Warnings: 
		 WARNING: Persistent journald logging is not enabled on this node.
Global Storage Pool
	Total Used    	:  110 GiB
	Total Capacity	:  634 GiB

Summary

In this blog I have shared how we can perform an online Portworx storage pool expansion using the Portworx command line interface pxctl.

[twitter-follow screen_name=’RonEkins’ show_count=’yes’]

Leave a Reply

Create a website or blog at WordPress.com

Up ↑

%d bloggers like this: