Kubernetes labels provide an important role in the management of large complex Kubernetes clusters.
Kubernetes implements labels as key value pairs which are attached to objects, such as nodes and pods to store identifying and meaningful attributes, which can then be used to manage the running Kubernetes platform.
For the purpose of the post I will be using an Oracle Cloud Infrastructure (OCI) Oracle Container Engine for Kubernetes (OKE) Kubernetes cluster, and 2 labels which I will be using to install Portworx in a future blog.
Working with Kubernetes Labels
There are a number of well-known Kubernetes labels which are common across Cloud vendors, for example kubernetes.io/os and kuberenetes.io/arch.
rekins@rekins--MacBookPro15 PX % kubectl get nodes -L kubernetes.io/os,kubernetes.io/arch NAME STATUS ROLES AGE VERSION OS ARCH 10.0.10.133 Ready node 5h16m v1.24.1 linux amd64 10.0.10.167 Ready node 5h16m v1.24.1 linux amd64 10.0.10.182 Ready node 5h16m v1.24.1 linux amd64 10.0.10.205 Ready node 5h16m v1.24.1 linux amd64 10.0.10.226 Ready node 5h16m v1.24.1 linux amd64 10.0.10.39 Ready node 5h16m v1.24.1 linux amd64
However, there are also many Kubernetes labels which are Cloud vendor specific and will return no data if queried on another Cloud, this can lead to operational and automation challenges.
Using the OKE console, I built a 6 node Kubernetes 1.24 Cluster in the UK-LONDON-1 Region, across the 3 Availability Domains (ADs).
Note: not every OKE Region has 3 ADs so you may need to confirm local availability.
I can determine where each worker nodes is running by using the Kubernetes labels OKE populated labels during the build.
- Region – topology.kubernetes.io/region
- Availability Domain – topology.kubernetes.io/zone
- Fault Domain – oci.oraclecloud.com/fault-domain
For example using kubectl get nodes -L –label-columns and above labels.
rekins@rekins--MacBookPro15 PX % kubectl get nodes -L topology.kubernetes.io/region,topology.kubernetes.io/zone,oci.oraclecloud.com/fault-domain NAME STATUS ROLES AGE VERSION REGION ZONE FAULT-DOMAIN 10.0.10.133 Ready node 5h24m v1.24.1 uk-london-1 UK-LONDON-1-AD-2 FAULT-DOMAIN-1 10.0.10.167 Ready node 5h24m v1.24.1 uk-london-1 UK-LONDON-1-AD-2 FAULT-DOMAIN-1 10.0.10.182 Ready node 5h24m v1.24.1 uk-london-1 UK-LONDON-1-AD-3 FAULT-DOMAIN-2 10.0.10.205 Ready node 5h24m v1.24.1 uk-london-1 UK-LONDON-1-AD-1 FAULT-DOMAIN-3 10.0.10.226 Ready node 5h24m v1.24.1 uk-london-1 UK-LONDON-1-AD-3 FAULT-DOMAIN-2 10.0.10.39 Ready node 5h24m v1.24.1 uk-london-1 UK-LONDON-1-AD-1 FAULT-DOMAIN-3
Add Label(s)
We can add a single label to an object using kubectl label, for example to label a node.
rekins@rekins--MacBookPro15 PX % kubectl label node 10.0.10.133 px/enabled=true node/10.0.10.133 labeled
For multiple labels provide a space delimited list, for example
rekins@rekins--MacBookPro15 PX % kubectl label node 10.0.10.167 px/enabled=true px/metadata-node=true node/10.0.10.167 labeled
Show Label(s)
We can check the labels have been applied using –show-labels
rekins@rekins--MacBookPro15 PX % kubectl get node/10.0.10.133 --show-labels NAME STATUS ROLES AGE VERSION LABELS 10.0.10.133 Ready node 59m v1.24.1 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=VM.Standard.E3.Flex,beta.kubernetes.io/os=linux,displayName=oke-csqdad5zmma-nxao222j34a-sq4lna5wcma-0,failure-domain.beta.kubernetes.io/region=uk-london-1,failure-domain.beta.kubernetes.io/zone=UK-LONDON-1-AD-2,hostname=oke-csqdad5zmma-nxao222j34a-sq4lna5wcma-0,internal_addr=10.0.10.133,kubernetes.io/arch=amd64,kubernetes.io/hostname=10.0.10.133,kubernetes.io/os=linux,last-migration-failure=get_kubesvc_failure,name=portworx,node-role.kubernetes.io/node=,node.info.ds_proxymux_client=true,node.info/compartment.id_prefix=ocid1.compartment.oc1,node.info/compartment.id_suffix=aaaaaaaa74sai3gxw4wikfatzj7t7rzniyhxuhpuvxsb4hahlbc5wwcbbcaa,node.info/compartment.name=PX-compartment,node.info/kubeletVersion=v1.24,node.kubernetes.io/instance-type=VM.Standard.E3.Flex,oci.oraclecloud.com/fault-domain=FAULT-DOMAIN-1,oke.oraclecloud.com/node.info.private_subnet=false,oke.oraclecloud.com/node.info.private_worker=true,oke.oraclecloud.com/tenant_agent.version=1.48.5-3e9306de62-942,px/enabled=true,topology.kubernetes.io/region=uk-london-1,topology.kubernetes.io/zone=UK-LONDON-1-AD-2
Or if you prefer with the -o / –output json
rekins@rekins--MacBookPro15 PX % kubectl get node 10.0.10.167 -o json
{
"apiVersion": "v1",
"kind": "Node",
"metadata": {
...
"labels": {
"beta.kubernetes.io/arch": "amd64",
"beta.kubernetes.io/instance-type": "VM.Standard.E3.Flex",
"beta.kubernetes.io/os": "linux",
"displayName": "oke-csqdad5zmma-nxao222j34a-sq4lna5wcma-3",
"failure-domain.beta.kubernetes.io/region": "uk-london-1",
"failure-domain.beta.kubernetes.io/zone": "UK-LONDON-1-AD-2",
"hostname": "oke-csqdad5zmma-nxao222j34a-sq4lna5wcma-3",
"internal_addr": "10.0.10.167",
"kubernetes.io/arch": "amd64",
"kubernetes.io/hostname": "10.0.10.167",
"kubernetes.io/os": "linux",
"last-migration-failure": "get_kubesvc_failure",
"name": "portworx",
"node-role.kubernetes.io/node": "",
"node.info.ds_proxymux_client": "true",
"node.info/compartment.id_prefix": "ocid1.compartment.oc1",
"node.info/compartment.id_suffix": "aaaaaaaa74sai3gxw4wikfatzj7t7rzniyhxuhpuvxsb4hahlbc5wwcbbcaa",
"node.info/compartment.name": "PX-compartment",
"node.info/kubeletVersion": "v1.24",
"node.kubernetes.io/instance-type": "VM.Standard.E3.Flex",
"oci.oraclecloud.com/fault-domain": "FAULT-DOMAIN-1",
"oke.oraclecloud.com/node.info.private_subnet": "false",
"oke.oraclecloud.com/node.info.private_worker": "true",
"oke.oraclecloud.com/tenant_agent.version": "1.48.5-3e9306de62-942",
"px/enabled": "true",
"px/metadata-node": "true",
"topology.kubernetes.io/region": "uk-london-1",
"topology.kubernetes.io/zone": "UK-LONDON-1-AD-2"
},
...
Select Label(s)
Use -l or –selector= to filter objects to specific value, for example
rekins@rekins--MacBookPro15 PX % kubectl get nodes -l px/enabled=true NAME STATUS ROLES AGE VERSION 10.0.10.133 Ready node 109m v1.24.1 10.0.10.167 Ready node 109m v1.24.1
With multiple labels.
rekins@rekins--MacBookPro15 PX % kubectl get nodes -l px/enabled=true,px/metadata-node=true NAME STATUS ROLES AGE VERSION 10.0.10.167 Ready node 111m v1.24.1
Or return objects that are not equal using != for example.
rekins@rekins--MacBookPro15 PX % kubectl get nodes --selector px/enabled!=true NAME STATUS ROLES AGE VERSION 10.0.10.182 Ready node 96m v1.24.1 10.0.10.205 Ready node 96m v1.24.1 10.0.10.226 Ready node 96m v1.24.1 10.0.10.39 Ready node 96m v1.24.1
Update Label
If we try to re-add or update an existing label we can expect to see an error message informing us that the key value pair already exists and we need to use –overwrite to update.
rekins@rekins--MacBookPro15 PX % kubectl label node 10.0.10.167 px/enabled=flase error: 'px/enabled' already has a value (true), and --overwrite is false
This time with the –overwrite option.
rekins@rekins--MacBookPro15 PX % kubectl label node 10.0.10.167 px/enabled=false --overwrite node/10.0.10.167 labeled
Remove Label(s)
To remove a label we append a – to the key, thus
rekins@rekins--MacBookPro15 PX % kubectl label node 10.0.10.133 px/enabled- node/10.0.10.133 unlabeled
Bulk Updates
As we have seen from above add, updating, and deleting labels is very simple, however what if we have many objects ?
The bash script below labels all nodes within my Kubernetes Cluster with px/enabled=true and px/metadata-node
for node in $(kubectl get nodes --output=jsonpath={.items..metadata.name});
do
echo Label $node for PX && kubectl label node $node px/enabled=true px/metadata-node=true --overwrite;
done
Looking at the script, the first line returns the node name from the kubectl get nodes command.
rekins@rekins--MacBookPro15 PX % kubectl get nodes --output=jsonpath={.items..metadata.name}
10.0.10.133 10.0.10.167 10.0.10.182 10.0.10.205 10.0.10.226 10.0.10.39
Line 3 processes each node within the loop labeling each node with kubectl label node
We can confirm all my worker nodes have been labeled correctly using kubectl get nodes -l
rekins@rekins--MacBookPro15 PX % kubectl get nodes -l px/enabled=true,px/metadata-node=true NAME STATUS ROLES AGE VERSION 10.0.10.133 Ready node 3h58m v1.24.1 10.0.10.167 Ready node 3h58m v1.24.1 10.0.10.182 Ready node 3h58m v1.24.1 10.0.10.205 Ready node 3h58m v1.24.1 10.0.10.226 Ready node 3h57m v1.24.1 10.0.10.39 Ready node 3h58m v1.24.1
Summary
In the short post I have shared how to add, update and delete Kubernetes labels, I have demonstrated how we can use labels to filter results and also perform bulk updates.