Background
As a long time fan of the Oracle ASM Filer Driver, one of the major benefits for Oracle ASM Filter Driver (AFD) is AFD Filtering.
AFD can ensure writes are only performed by trusted and registered Oracle processes, blocking potentially damaging writes that would have otherwise resulted in data corruption.
Unfortunately, a recent Linux kernel change has impacted the AFD and the latest recommendation from Oracle is to disable AFD Filtering until a fix is available.
From Oracle Database 19.24 filtering is disabled by default, but for earlier versions Oracle has provided Patch 35618858: DISABLE FILTERING BY DEFAULT FOR 5.15 AND ABOVE KERNELS.
Available Oracle Release Update (RU) Patches
- Oracle Clusterware 19 Release (19.23.0.0.240416ACFSRU)
- Oracle Clusterware 19 Release (19.24.0.0.0ACFSRU)
- Oracle Clusterware 19 Release (19.25.0.0.0ACFSRU)
In this blog post I will explore the impact of disabling AFD Filtering on space reclamation on all-flash storage arrays.
Kernel Version
I have recently upgraded my Oracle RAC servers to Oracle Linux 8.10 using a kernel 5.15, we can check your kernel version using uname -r
[oracle@z-rac1 ~]$ cat /etc/oracle-release
Oracle Linux Server release 8.10
[oracle@z-rac1 ~]$ uname -r
5.15.0-208.159.3.el8uek.x86_64
Oracle ASMFD State
We can use the asmcmd afd_state command to get the ASM Filter Driver status, for example
[oracle@z-rac1 ~]$ asmcmd afd_state
ASMCMD-9526: The AFD state is 'LOADED' and filtering is 'ENABLED' on host 'z-rac1.uklab.purestorage.com'
In a RAC cluster use asmcmd afd_state –all option to see the status of all nodes, all nodes should have the same Filtering Status.
[oracle@z-rac1 ~]$ asmcmd afd_state --all
node 'z-rac1': State of AFD: LOADED and ASMFD Filtering Status: ENABLED
node 'z-rac2': State of AFD: LOADED and ASMFD Filtering Status: DISABLED
Disable ASMFD Filtering
Disable filtering using asmcmd afd_filter -d command
[oracle@z-rac1 ~]$ asmcmd afd_filter -d
[oracle@z-rac1 ~]$ asmcmd afd_state
ASMCMD-9526: The AFD state is 'LOADED' and filtering is 'DISABLED' on host 'z-rac1.uklab.purestorage.com'
Enable ASMFD Filtering
If required you can re-enable filtering using asmcmd afd_filter -e command, for example:
[oracle@z-rac1 ~]$ asmcmd afd_filter -e
[oracle@z-rac1 ~]$ asmcmd afd_state
ASMCMD-9526: The AFD state is 'LOADED' and filtering is 'ENABLED' on host 'z-rac1.uklab.purestorage.com'
Storage Space Reclamation
Now this raises a big question, if I disable the ASM Filter Driver, do I loose the ability to send SCSI UNMAP commands back to an all-flash storage array to release unused space.
OK, I have a 2 node RAC cluster with ASMFD Filtering DISABLED.
[oracle@z-rac1 ~]$ asmcmd afd_state --all
node 'z-rac1': State of AFD: LOADED and ASMFD Filtering Status: DISABLED
node 'z-rac2': State of AFD: LOADED and ASMFD Filtering Status: DISABLED
Create Oracle Tablespace
To demonstrate this let’s create a 4TB tablespace and wait for Oracle to initialise the datafile.
Let’s confirm there is no space being used using my v$asm_disk query
ASMFD Disks
Disk Disk Disk Total Used Free
Nbr Name Label Header Mount State Path GB GB GB
---- ---------- ---------- ------- ------- ------- ---------- -------- -------- --------
0 DATA08 DATA08 MEMBER CACHED NORMAL AFD:DATA08 12,288 0 12,288
1 DATA07 DATA07 MEMBER CACHED NORMAL AFD:DATA07 12,288 0 12,288
2 DATA05 DATA05 MEMBER CACHED NORMAL AFD:DATA05 12,288 0 12,288
3 DATA06 DATA06 MEMBER CACHED NORMAL AFD:DATA06 12,288 0 12,288
And now create the 4TB bigfile tablespace.
SQL> create BIGFILE tablespace test datafile '+AFDATA' size 4T;
Tablespace created.
Using the v$asm_disk query I can see my 4 ASMFD volumes have used in total 4TB as expected.
ASMFD Disks
Disk Disk Disk Total Used Free
Nbr Name Label Header Mount State Path GB GB GB
---- ---------- ---------- ------- ------- ------- ---------- -------- -------- --------
0 DATA08 DATA08 MEMBER CACHED NORMAL AFD:DATA08 12,288 1,024 11,264
1 DATA07 DATA07 MEMBER CACHED NORMAL AFD:DATA07 12,288 1,024 11,264
2 DATA05 DATA05 MEMBER CACHED NORMAL AFD:DATA05 12,288 1,024 11,264
3 DATA06 DATA06 MEMBER CACHED NORMAL AFD:DATA06 12,288 1,024 11,264
Using the asm command line utility asmcmd with the list attribute option I see than Thin Provisioning is enabled on my AFDATA diskgroup.
[oracle@z-rac1 ~]$ asmcmd lsattr -G AFDATA -l | grep thin
thin_provisioned TRUE
Using the FlasArray CLI and performing ‘purevol list <volume name> –-space –-total’ command we can confirm the space used by the tablespace in AFD diskgroup. For example:
purevol list z-racpod::afd_data* –space –total
pureuser@z-x90-a> purevol list z-racpod::afd_data* --space --total
Name Size Virtual Thin Provisioning Data Reduction Total Reduction Unique Snapshots Total
z-racpod::afd_data05 12T 1.00T 92% >100 to 1 >100 to 1 4.95G 0.00 4.95G
z-racpod::afd_data06 12T 1.00T 92% >100 to 1 >100 to 1 4.95G 0.00 4.95G
z-racpod::afd_data07 12T 1.00T 92% >100 to 1 >100 to 1 4.95G 0.00 4.95G
z-racpod::afd_data08 12T 1.00T 92% >100 to 1 >100 to 1 4.95G 0.00 4.95G
(total) 48T 4.00T 92% >100 to 1 >100 to 1 19.79G 0.00 19.79G
Drop Tablespace
Let’s now drop the Oracle tablespace using DROP TABLESPACE <tablespace_name> INCLUDING CONTENTS AND DATAFILES; and see the impact on the storage platform and Oracle database.
SQL> DROP TABLESPACE test INCLUDING CONTENTS AND DATAFILES;
Tablespace dropped.
If we re-run the v$asm_view query Oracle is now reporting the 4TB test tablespace has been released.
ASMFD Disks
Disk Disk Disk Total Used Free
Nbr Name Label Header Mount State Path GB GB GB
---- ---------- ---------- ------- ------- ------- ---------- -------- -------- --------
0 DATA08 DATA08 MEMBER CACHED NORMAL AFD:DATA08 12,288 0 12,288
1 DATA07 DATA07 MEMBER CACHED NORMAL AFD:DATA07 12,288 0 12,288
2 DATA05 DATA05 MEMBER CACHED NORMAL AFD:DATA05 12,288 0 12,288
3 DATA06 DATA06 MEMBER CACHED NORMAL AFD:DATA06 12,288 0 12,288
Returning to the Pure Storage FlashArray, we can see that we are still consuming space even though the Oracle tablespace has been deleted.
pureuser@z-x90-a> purevol list z-racpod::afd_data* --space --total
Name Size Virtual Thin Provisioning Data Reduction Total Reduction Unique Snapshots Total
z-racpod::afd_data05 12T 1.00T 92% >100 to 1 >100 to 1 4.95G 0.00 4.95G
z-racpod::afd_data06 12T 1.00T 92% >100 to 1 >100 to 1 4.95G 0.00 4.95G
z-racpod::afd_data07 12T 1.00T 92% >100 to 1 >100 to 1 4.95G 0.00 4.95G
z-racpod::afd_data08 12T 1.00T 92% >100 to 1 >100 to 1 4.95G 0.00 4.95G
(total) 48T 4.00T 92% >100 to 1 >100 to 1 19.79G 0.00 19.79G
Oracle ASMFD rebalance compact Phase
When using Oracle ASM Filter Driver we can manually trigger a SCSI UNMAP command to release space back to the storage array using the ALTER DISKGROUP <NAME> REBALANCE WITH BALANCE COMPACT.
I’ve included the optional WAIT option, to only return once complete.
SQL> ALTER DISKGROUP AFDATA REBALANCE WITH BALANCE COMPACT WAIT;
Diskgroup altered.
If we now return to the FlasArray CLI and repeat the ‘purevol list <volume name> –space –total’ command we can see the impact of the ALTER DISKGROUP <name> REBALANCE WITH COMPACT.
We can see that even though AFD Filtering has been disabled, ASMFD is still able to send the SCSI UNMAP request to the storage array to release the space.
pureuser@z-x90-a> purevol list z-racpod::afd_data* --space --total
z-racpod::afd_data05 12T 1.00T 92% >100 to 1 >100 to 1 4.95G 0.00 4.95G
z-racpod::afd_data06 12T 1.00T 92% >100 to 1 >100 to 1 4.95G 0.00 4.95G
z-racpod::afd_data07 12T 1.00T 92% >100 to 1 >100 to 1 4.95G 0.00 4.95G
z-racpod::afd_data08 12T 1.00T 92% >100 to 1 >100 to 1 4.95G 0.00 4.95G
(total) 48T 4.00T 92% >100 to 1 >100 to 1 19.79G 0.00 19.79G
pureuser@z-x90-a> purevol list z-racpod::afd_data* --space --total
Name Size Virtual Thin Provisioning Data Reduction Total Reduction Unique Snapshots Total
z-racpod::afd_data05 12T 92.00M 100% 6.9 to 1 >100 to 1 9.29M 0.00 9.29M
z-racpod::afd_data06 12T 92.00M 100% 6.6 to 1 >100 to 1 10.47M 0.00 10.47M
z-racpod::afd_data07 12T 92.00M 100% 6.7 to 1 >100 to 1 9.82M 0.00 9.82M
z-racpod::afd_data08 12T 92.00M 100% 7.0 to 1 >100 to 1 8.63M 0.00 8.63M
(total) 48T 368.00M 100% 6.8 to 1 >100 to 1 38.21M 0.00 38.21M
Summary
In this blog post I have confirmed that disabling ASMFD Filtering has no impact on space reclamation.
The Oracle AFD development team is actively working on a new filtering and protection mechanism on the above mentioned and newer kernels. When the new filtering solution becomes available, I will give it a try and share my findings.
If you want to learn more about using the ASM Filter Driver to reclaim storage space, check out this blog post.