Oracle ASM Filter Driver (AFD) Filtering and 5.15 UEK7 kernels

Background

As a long time fan of the Oracle ASM Filer Driver, one of the major benefits for Oracle ASM Filter Driver (AFD) is AFD Filtering.

AFD can ensure writes are only performed by trusted and registered Oracle processes, blocking potentially damaging writes that would have otherwise resulted in data corruption.

Unfortunately, a recent Linux kernel change has impacted the AFD and the latest recommendation from Oracle is to disable AFD Filtering until a fix is available.

From Oracle Database 19.24 filtering is disabled by default, but for earlier versions Oracle has provided Patch 35618858: DISABLE FILTERING BY DEFAULT FOR 5.15 AND ABOVE KERNELS.

Available Oracle Release Update (RU) Patches

• Oracle Clusterware 19 Release (19.23.0.0.240416ACFSRU)
• Oracle Clusterware 19 Release (19.24.0.0.0ACFSRU)
• Oracle Clusterware 19 Release (19.25.0.0.0ACFSRU)

In this blog post I will explore the impact of disabling AFD Filtering on space reclamation on all-flash storage arrays.

Kernel Version

I have recently upgraded my Oracle RAC servers to Oracle Linux 8.10 using a kernel 5.15, we can check your kernel version using uname -r

[oracle@z-rac1 ~]$ cat /etc/oracle-release
Oracle Linux Server release 8.10
[oracle@z-rac1 ~]$ uname -r
5.15.0-208.159.3.el8uek.x86_64

Oracle ASMFD State

We can use the asmcmd afd_state command to get the ASM Filter Driver status, for example

[oracle@z-rac1 ~]$ asmcmd afd_state
ASMCMD-9526: The AFD state is 'LOADED' and filtering is 'ENABLED' on host 'z-rac1.uklab.purestorage.com'

In a RAC cluster use asmcmd afd_state –all option to see the status of all nodes, all nodes should have the same Filtering Status.

[oracle@z-rac1 ~]$ asmcmd afd_state --all
node 'z-rac1': State of AFD: LOADED and ASMFD Filtering Status: ENABLED
node 'z-rac2': State of AFD: LOADED and ASMFD Filtering Status: DISABLED

Disable ASMFD Filtering

Disable filtering using asmcmd afd_filter -d command

[oracle@z-rac1 ~]$ asmcmd afd_filter -d
[oracle@z-rac1 ~]$ asmcmd afd_state
ASMCMD-9526: The AFD state is 'LOADED' and filtering is 'DISABLED' on host 'z-rac1.uklab.purestorage.com'

Enable ASMFD Filtering

If required you can re-enable filtering using asmcmd afd_filter -e command, for example:

[oracle@z-rac1 ~]$ asmcmd afd_filter -e
[oracle@z-rac1 ~]$ asmcmd afd_state
ASMCMD-9526: The AFD state is 'LOADED' and filtering is 'ENABLED' on host 'z-rac1.uklab.purestorage.com'

Storage Space Reclamation

Now this raises a big question, if I disable the ASM Filter Driver, do I loose the ability to send SCSI UNMAP commands back to an all-flash storage array to release unused space.

OK, I have a 2 node RAC cluster with ASMFD Filtering DISABLED.

[oracle@z-rac1 ~]$ asmcmd afd_state --all
node 'z-rac1': State of AFD: LOADED and ASMFD Filtering Status: DISABLED
node 'z-rac2': State of AFD: LOADED and ASMFD Filtering Status: DISABLED

Create Oracle Tablespace

To demonstrate this let’s create a 4TB tablespace and wait for Oracle to initialise the datafile.

Let’s confirm there is no space being used using my v$asm_disk query

ASMFD Disks

Disk Disk	Disk						 Total	   Used     Free
 Nbr Name	Label	   Header  Mount   State   Path 	    GB	     GB       GB
---- ---------- ---------- ------- ------- ------- ---------- -------- -------- --------
   0 DATA08	DATA08	   MEMBER  CACHED  NORMAL  AFD:DATA08	12,288	      0   12,288
   1 DATA07	DATA07	   MEMBER  CACHED  NORMAL  AFD:DATA07	12,288	      0   12,288
   2 DATA05	DATA05	   MEMBER  CACHED  NORMAL  AFD:DATA05	12,288	      0   12,288
   3 DATA06	DATA06	   MEMBER  CACHED  NORMAL  AFD:DATA06	12,288	      0   12,288

And now create the 4TB bigfile tablespace.

SQL> create BIGFILE tablespace test datafile '+AFDATA' size 4T;
Tablespace created.

Using the v$asm_disk query I can see my 4 ASMFD volumes have used in total 4TB as expected.

ASMFD Disks

Disk Disk	Disk						 Total	   Used     Free
 Nbr Name	Label	   Header  Mount   State   Path 	    GB	     GB       GB
---- ---------- ---------- ------- ------- ------- ---------- -------- -------- --------
   0 DATA08	DATA08	   MEMBER  CACHED  NORMAL  AFD:DATA08	12,288	  1,024   11,264
   1 DATA07	DATA07	   MEMBER  CACHED  NORMAL  AFD:DATA07	12,288	  1,024   11,264
   2 DATA05	DATA05	   MEMBER  CACHED  NORMAL  AFD:DATA05	12,288	  1,024   11,264
   3 DATA06	DATA06	   MEMBER  CACHED  NORMAL  AFD:DATA06	12,288	  1,024   11,264

Using the asm command line utility asmcmd with the list attribute option I see than Thin Provisioning is enabled on my AFDATA diskgroup.

[oracle@z-rac1 ~]$ asmcmd lsattr -G AFDATA -l | grep thin
thin_provisioned               TRUE

Using the FlasArray CLI and performing ‘purevol list <volume name> –-space –-total’ command we can confirm the space used by the tablespace in AFD diskgroup. For example:

purevol list z-racpod::afd_data* –space –total

pureuser@z-x90-a> purevol list z-racpod::afd_data* --space --total
Name                  Size  Virtual   Thin Provisioning  Data Reduction  Total Reduction  Unique   Snapshots  Total
z-racpod::afd_data05  12T   1.00T    92%                >100 to 1       >100 to 1        4.95G   0.00       4.95G
z-racpod::afd_data06  12T   1.00T    92%                >100 to 1       >100 to 1        4.95G   0.00       4.95G
z-racpod::afd_data07  12T   1.00T    92%                >100 to 1       >100 to 1        4.95G   0.00       4.95G
z-racpod::afd_data08  12T   1.00T    92%                >100 to 1       >100 to 1        4.95G   0.00       4.95G
(total)               48T   4.00T    92%                >100 to 1       >100 to 1        19.79G  0.00       19.79G

Drop Tablespace

Let’s now drop the Oracle tablespace using DROP TABLESPACE <tablespace_name> INCLUDING CONTENTS AND DATAFILES; and see the impact on the storage platform and Oracle database.

SQL> DROP TABLESPACE test INCLUDING CONTENTS AND DATAFILES;
Tablespace dropped.

If we re-run the v$asm_view query Oracle is now reporting the 4TB test tablespace has been released.

ASMFD Disks

Disk Disk	Disk						 Total	   Used     Free
 Nbr Name	Label	   Header  Mount   State   Path 	    GB	     GB       GB
---- ---------- ---------- ------- ------- ------- ---------- -------- -------- --------
   0 DATA08	DATA08	   MEMBER  CACHED  NORMAL  AFD:DATA08	12,288	      0   12,288
   1 DATA07	DATA07	   MEMBER  CACHED  NORMAL  AFD:DATA07	12,288	      0   12,288
   2 DATA05	DATA05	   MEMBER  CACHED  NORMAL  AFD:DATA05	12,288	      0   12,288
   3 DATA06	DATA06	   MEMBER  CACHED  NORMAL  AFD:DATA06	12,288	      0   12,288

Returning to the Pure Storage FlashArray, we can see that we are still consuming space even though the Oracle tablespace has been deleted.

pureuser@z-x90-a> purevol list z-racpod::afd_data* --space --total
Name                  Size  Virtual   Thin Provisioning  Data Reduction  Total Reduction  Unique   Snapshots  Total
z-racpod::afd_data05  12T   1.00T    92%                >100 to 1       >100 to 1        4.95G   0.00       4.95G
z-racpod::afd_data06  12T   1.00T    92%                >100 to 1       >100 to 1        4.95G   0.00       4.95G
z-racpod::afd_data07  12T   1.00T    92%                >100 to 1       >100 to 1        4.95G   0.00       4.95G
z-racpod::afd_data08  12T   1.00T    92%                >100 to 1       >100 to 1        4.95G   0.00       4.95G
(total)               48T   4.00T    92%                >100 to 1       >100 to 1        19.79G  0.00       19.79G

Oracle ASMFD rebalance compact Phase

When using Oracle ASM Filter Driver we can manually trigger a SCSI UNMAP command to release space back to the storage array using the ALTER DISKGROUP <NAME> REBALANCE WITH BALANCE COMPACT.

I’ve included the optional WAIT option, to only return once complete.

SQL> ALTER DISKGROUP AFDATA REBALANCE WITH BALANCE COMPACT WAIT;
Diskgroup altered.

If we now return to the FlasArray CLI and repeat the ‘purevol list <volume name> –space –total’ command we can see the impact of the ALTER DISKGROUP <name> REBALANCE WITH COMPACT.

We can see that even though AFD Filtering has been disabled, ASMFD is still able to send the SCSI UNMAP request to the storage array to release the space.

pureuser@z-x90-a> purevol list z-racpod::afd_data* --space --total
z-racpod::afd_data05  12T   1.00T    92%                >100 to 1       >100 to 1        4.95G   0.00       4.95G
z-racpod::afd_data06  12T   1.00T    92%                >100 to 1       >100 to 1        4.95G   0.00       4.95G
z-racpod::afd_data07  12T   1.00T    92%                >100 to 1       >100 to 1        4.95G   0.00       4.95G
z-racpod::afd_data08  12T   1.00T    92%                >100 to 1       >100 to 1        4.95G   0.00       4.95G
(total)               48T   4.00T    92%                >100 to 1       >100 to 1        19.79G  0.00       19.79G


pureuser@z-x90-a> purevol list z-racpod::afd_data* --space --total
Name                  Size  Virtual  Thin Provisioning  Data Reduction  Total Reduction  Unique  Snapshots  Total
z-racpod::afd_data05  12T   92.00M   100%               6.9 to 1        >100 to 1        9.29M   0.00       9.29M
z-racpod::afd_data06  12T   92.00M   100%               6.6 to 1        >100 to 1        10.47M  0.00       10.47M
z-racpod::afd_data07  12T   92.00M   100%               6.7 to 1        >100 to 1        9.82M   0.00       9.82M
z-racpod::afd_data08  12T   92.00M   100%               7.0 to 1        >100 to 1        8.63M   0.00       8.63M
(total)               48T   368.00M  100%               6.8 to 1        >100 to 1        38.21M  0.00       38.21M

Summary

In this blog post I have confirmed that disabling ASMFD Filtering has no impact on space reclamation.

The Oracle AFD development team is actively working on a new filtering and protection mechanism on the above mentioned and newer kernels. When the new filtering solution becomes available, I will give it a try and share my findings.

If you want to learn more about using the ASM Filter Driver to reclaim storage space, check out this blog post.