Background
Ok, what is the Oracle ASM Filter Driver ?
The Oracle ASM Filter Driver (Oracle ASMFD) is an Oracle provided Linux kernel module that resides in the IO path of the Oracle ASM disks.
Oracle ASMFD provides a number of advantages over the use of ASMLIB and UDEV managed volumes, these include: storage protection (a topic for a future post), performance improvements and space reclamation for flash storage which us the focus of this blog post.
How to check Oracle ASMFD installation and configuration
Before we start, let’s verify the ASM Filer Driver (AFD) Kernel module has been loaded.
Use lsmod to list loaded Kernel modules and look for oracleafd, for example:
[oracle@z-oracle ~]$ lsmod | grep afd
oracleafd 217088 1
To confirm ASM Filter Driver has already been installed, loaded, is supported and versions, use afddriverstate installed | loaded | supported | version, for example:
[oracle@z-oracle ~]$ afddriverstate installed
AFD-9203: AFD device driver installed status: 'true'
[oracle@z-oracle ~]$ afddriverstate loaded
AFD-9205: AFD device driver loaded status: 'true'
[oracle@z-oracle ~]$ afddriverstate supported
AFD-9200: Supported
[oracle@z-oracle ~]$ afddriverstate version
AFD-9325: Driver OS kernel version = 4.1.12-112.16.4.el7uek.x86_64.
AFD-9326: Driver build number = 201219.
AFD-9212: Driver build version = 19.0.0.0.0.
AFD-9547: Driver available build number = 201219.
AFD-9548: Driver available build version = 19.0.0.0.0.
To check the ASMFD filtering status use the asmcmd afd_state command, for example:
oracle@z-oracle ~]$ asmcmd afd_state
ASMCMD-9526: The AFD state is 'LOADED' and filtering is 'ENABLED' on host 'z-oracle.uklab.purestorage.com'
Confirm AFD Disks
We can review the Oracle ASM Filter Driver (AFD) disks by querying the v$asm_disk view, using asmcmd and KFOD.
Using v$asm_disk
Use the v$asm_disk view and library to identify AFD disks.
set feed off
set pages 100
set lines 250
col group_number heading 'Grp' format 99
col disk_number heading 'Disk|Nbr' format 99
col name heading 'Disk|Name' format a10
col state heading 'State' format a10
col type heading 'Type' format a10
col label heading 'Disk|Label' format a15
col header_status heading 'Header' format a10
col mount_status heading 'Mount' format a10
col path heading 'Path' format a20
col library heading 'Library' format a44
col tgb format 999,999 heading 'Total|GB'
col fgb format 999,999 heading 'Free|GB'
col ugb format 999,999 heading 'Used|GB'
Prompt ASMFD Disks
SELECT d.disk_number, d.name, d.label, d.header_status, d.mount_status, d.state, d.path,
d.total_mb/1024 as tgb, ((d.total_mb-d.free_mb)/1024) as ugb, d.free_mb/1024 as fgb
FROM v$asm_disk d
WHERE d.library like 'AFD%'
order by d.disk_number;
For example:
Disk Disk Disk Total Used Free
Nbr Name Label Header Mount State Path GB GB GB
---- ---------- --------------- ---------- ---------- ---------- -------------------- -------- -------- --------
0 AFDATA1 AFDATA1 MEMBER CACHED NORMAL AFD:AFDATA1 1,024 0 1,024
1 AFDATA2 AFDATA2 MEMBER CACHED NORMAL AFD:AFDATA2 1,024 0 1,024
2 AFDATA3 AFDATA3 MEMBER CACHED NORMAL AFD:AFDATA3 1,024 0 1,024
3 AFDATA4 AFDATA4 MEMBER CACHED NORMAL AFD:AFDATA4 1,024 0 1,024
Using asmcmd
Alternately, list the Oracle AFD disks asmcmd afd_lsdsk command to list ASM Filter Driver disks, thus:
[oracle@z-oracle sql_scripts]$ asmcmd afd_lsdsk
--------------------------------------------------------------------------------
Label Filtering Path
================================================================================
AFDATA1 ENABLED /dev/mapper/afd_oradata5
AFDATA2 ENABLED /dev/mapper/afd_oradata6
AFDATA3 ENABLED /dev/mapper/afd_oradata7
AFDATA4 ENABLED /dev/mapper/afd_oradata8
Using KFOD
The Oracle KFOD utility can also be used to identify AFD disks, using the following syntax:
kfod OP=DISKS status=TRUE disks=ASM label=TRUE dscvgroup=TRUE asm_diskstring=AFD:*
[oracle@z-oracle ~]$ kfod OP=DISKS status=TRUE disks=ASM label=TRUE dscvgroup=TRUE asm_diskstring=AFD:*
--------------------------------------------------------------------------------
Disk Size Header Path Disk Group Site Failgroup Disk-Label User Group
================================================================================
1: 1048576 MB MEMBER AFD:AFDATA1 AFDATA # # AFDATA1
2: 1048576 MB MEMBER AFD:AFDATA2 AFDATA # # AFDATA2
3: 1048576 MB MEMBER AFD:AFDATA3 AFDATA # # AFDATA3
4: 1048576 MB MEMBER AFD:AFDATA4 AFDATA # # AFDATA4
If you want to learn more about KFOD, you may want to check out this blog post.
Oracle ASM Thin Provisioned Attribute
The ASM Thin Provisioned attribute enables or disables the functionality to discard unused storage space after a diskgroup rebalance is completed.
Note: the ASM Thin Provisioned attribute only works with the ASM Filter Driver, setting it on non-ASMFD diskgroups has no effect.
The attribute value can be true to enable or false to disable the functionality. The default value is false.
Modern all-flash storage platforms including the Pure FlashArray have the capability to reuse the discarded storage space for a more efficient overall physical storage utilisation.
Oracle ASM informs the storage array which space is no longer used and can be repurposed. If a rebalance is not running, a rebalance can be triggered manually with the ALTER DISKGROUP … REBALANCE SQL statement.
How to check ASM ‘thin_provisioned’ attribute
To check & change the ASMFD thin_provisioned status, use the SQL code below:
set linesize 250
column diskgroup heading 'Diskgroup' format a15
column name heading 'Attribute Name' format a25
column value heading 'Value' format a15
column read_only heading 'Read Only' format a15
SELECT
SUBSTR(dg.name,1,12) AS diskgroup, SUBSTR(a.name,1,24) AS name,
SUBSTR(a.value,1,24) AS value, read_only
FROM
V$ASM_DISKGROUP dg,
V$ASM_ATTRIBUTE a
WHERE
dg.group_number = a.group_number AND
a.name NOT LIKE '%template%' AND
a.name LIKE '%thin%'
/
exit
Diskgroup Attribute Name Value Read Only
--------------- ------------------------- --------------- ---------------
CONTROL_REDO thin_provisioned FALSE N
DATA thin_provisioned FALSE N
FRA thin_provisioned FALSE N
AFDATA thin_provisioned FALSE N
We can easily change the ‘thin_provisioned’ attribute using ASMCA or using ‘alter disk group <DISK_GROUP> set attribute ‘thin_provisioned’=’TRUE’;’
SQL> alter diskgroup AFDATA set attribute 'thin_provisioned'='TRUE';
Diskgroup altered.
And now if we repeat the query.
Diskgroup Attribute Name Value Read Only
--------------- ------------------------- --------------- ---------------
CONTROL_REDO thin_provisioned FALSE N
DATA thin_provisioned FALSE N
FRA thin_provisioned FALSE N
AFDATA thin_provisioned TRUE N
We can also use the asmcmd lsattr (list attribute) option, as below
[oracle@z-oracle ~]$ asmcmd lsattr -G AFDATA -l | grep thin
thin_provisioned TRUE
Oracle Create Tablespace
To demonstrate this let’s create a 3.9TB tablespace and wait for Oracle to initialise the datafile.
SQL> create BIGFILE tablespace test datafile '+AFDATA' size 3900G;
Tablespace created.
Repeating the v$asm_disk query I can see my 4 ASMFD volumes have used in total 3.9TB as expected.
ASMFD Disks
Disk Disk Disk Total Used Free
Nbr Name Label Header Mount State Path GB GB GB
---- ---------- --------------- ---------- ---------- ---------- -------------------- -------- -------- --------
0 AFDATA1 AFDATA1 MEMBER CACHED NORMAL AFD:AFDATA1 1,024 979 45
1 AFDATA2 AFDATA2 MEMBER CACHED NORMAL AFD:AFDATA2 1,024 979 45
2 AFDATA3 AFDATA3 MEMBER CACHED NORMAL AFD:AFDATA3 1,024 979 45
3 AFDATA4 AFDATA4 MEMBER CACHED NORMAL AFD:AFDATA4 1,024 979 45
Using the FlasArray CLI and performing ‘purevol list <volume name> –space –total’ command we can confirm the space used by the new tablespace in AFD diskgroup. For example:
purevol list z-oracle-afd* –space –total
pureuser@z-fastaines-c> purevol list z-oracle-afd* --space --total
Name Size Virtual Thin Provisioning Data Reduction Total Reduction Unique Snapshots Total
z-oracle-afd_oradata5 1T 979.18G 4% >100 to 1 >100 to 1 4.26G 0.00 4.26G
z-oracle-afd_oradata6 1T 979.17G 4% >100 to 1 >100 to 1 4.18G 0.00 4.18G
z-oracle-afd_oradata7 1T 979.17G 4% >100 to 1 >100 to 1 4.19G 0.00 4.19G
z-oracle-afd_oradata8 1T 979.10G 4% >100 to 1 >100 to 1 4.16G 0.00 4.16G
(total) 4T 3.82T 4% >100 to 1 >100 to 1 16.79G 0.00 16.79G
Oracle Drop Tablespace
Let’s now drop the bigfile tablespace using DROP TABLESPACE <tablespace_name> INCLUDING CONTENTS AND DATAFILES; and see the impact on the storage platform and Oracle.
SQL> DROP TABLESPACE test INCLUDING CONTENTS AND DATAFILES;
Tablespace dropped.
If we re-run the v$asm_view quay Oracle is now reporting the 3.9TB test tablespace has gone.
ASMFD Disks
Disk Disk Disk Total Used Free
Nbr Name Label Header Mount State Path GB GB GB
---- ---------- --------------- ---------- ---------- ---------- -------------------- -------- -------- --------
0 AFDATA1 AFDATA1 MEMBER CACHED NORMAL AFD:AFDATA1 1,024 4 1,020
1 AFDATA2 AFDATA2 MEMBER CACHED NORMAL AFD:AFDATA2 1,024 4 1,020
2 AFDATA3 AFDATA3 MEMBER CACHED NORMAL AFD:AFDATA3 1,024 4 1,020
3 AFDATA4 AFDATA4 MEMBER CACHED NORMAL AFD:AFDATA4 1,024 4 1,020
However, returning to the Pure Storage FlashArray, we can see that we are still consuming space even though the Oracle tablespace has been deleted.
pureuser@z-fastaines-c> purevol list z-oracle-afd* --space --total
Name Size Virtual Thin Provisioning Data Reduction Total Reduction Unique Snapshots Total
z-oracle-afd_oradata5 1T 979.18G 4% >100 to 1 >100 to 1 4.26G 0.00 4.26G
z-oracle-afd_oradata6 1T 979.17G 4% >100 to 1 >100 to 1 4.18G 0.00 4.18G
z-oracle-afd_oradata7 1T 979.17G 4% >100 to 1 >100 to 1 4.19G 0.00 4.19G
z-oracle-afd_oradata8 1T 979.10G 4% >100 to 1 >100 to 1 4.16G 0.00 4.16G
(total) 4T 3.82T 4% >100 to 1 >100 to 1 16.78G 0.00 16.78G
Oracle ASM rebalance compact Phase
From Oracle MOS Note: What is ASM rebalance compact Phase and how it can be disabled (Doc ID 1902001.1)
The compact phase is part of rebalance operation, it moves the data as close as possible to the outer tracks of the disks (the lower numbered offsets).
In the disk world we try to place data on the outer tracks as these have lower seek times, performing faster than inner tracks because the outer tracks have more data packed in and more sectors per track.
Oracle 12c removed the instance level _disable_rebalance_compact parameter, this has been replaced with the diskgroup level attribute _rebalance_compact can be used, for example:
SQL> ALTER DISKGROUP DATA SET ATTRIBUTE '_rebalance_compact'='FALSE';
Diskgroup altered.
Note: As with all Oracle underscore parameters, they should only be used in Production under the direction of Oracle Support
Since Oracle 18c the phase option includes the keywords WITH or WITHOUT, removing the need to use the underscore parameters, check your version documentation for limitations.
SQL> ALTER DISKGROUP DATA REBALANCE WITHOUT COMPACT;
Diskgroup altered.
Oracle ASMFD and rebalance compact phase
When using Oracle ASM Filter Driver we can manually trigger a SCSI UNMAP command to release space back to the storage array using the ALTER DISKGROUP <NAME> REBALANCE WITH BALANCE COMPACT.
Note: Consult your storage vendor to confirm support of SCSI UNMAP commands.
I’ve included the optional WAIT option, to only return once complete, and a POWER value of 1024 to finish as fast as possible.
SQL> ALTER DISKGROUP AFDATA REBALANCE WITH BALANCE COMPACT POWER 1024 WAIT;
Diskgroup altered.
If we now return to the FlasArray CLI and repeat the ‘purevol list <volume name> –space –total’ command we can confirm that Oracle has sent an UNMAP request to the storage array and the space has been released.
pureuser@z-fastaines-c> purevol list z-oracle-afd* --space --total
Name Size Virtual Thin Provisioning Data Reduction Total Reduction Unique Snapshots Total
z-oracle-afd_oradata5 1T 4.17G 100% 17.5 to 1 >100 to 1 111.71M 0.00 111.71M
z-oracle-afd_oradata6 1T 4.17G 100% 17.3 to 1 >100 to 1 115.96M 0.00 115.96M
z-oracle-afd_oradata7 1T 4.17G 100% 16.5 to 1 >100 to 1 126.72M 0.00 126.72M
z-oracle-afd_oradata8 1T 4.11G 100% 17.9 to 1 >100 to 1 102.11M 0.00 102.11M
(total) 4T 16.62G 100% 17.3 to 1 >100 to 1 456.50M 0.00 456.50M
We can also confirm the space has been released using the Pure Storage FlashArray WebUI and navigating to Storage -> Volumes, for example:
Summary
If you are using an all-flash-storage array and Oracle ASM to manage your block storage, I suggest you spend some time reviewing and testing the Oracle ASM Filter Driver (ASMFD), the performance, security, and simplified space management provide compelling reason why to to migrate to ASMFD.
If you want to learn more about the installation, configuration, migration and management of ASM Filter Driver you consult the Administering Oracle ASM Filter Driver section within the Oracle Database 19c Administrator’s Guide.