Oracle Database 19c on Oracle Direct NFS (dNFS) using FA File

Background

With Pure Storage Purity 6.1 we saw the arrival of file support to the family of Pure Storage FlashArrays, providing a unified block and file storage platform.

In this blog post I will share how to get an Oracle database 19c up and running on a Pure Storage FlashArray using Oracle Direct NFS (dNFS).

Oracle Direct NFS (dNFS)

Oracle dNFS is a mature Oracle Database feature, and has been available since Oracle Database 11g Release 1. Oracle dNFS addresses many of the challenges of running Oracle on NFS providing many benefits these include:

• Eliminating operating system write-ordering locks
• Improving HA (High Availability) of NAS storage 
• Bypassing Operating system caches improving database performance on NAS storage.
• Supports up to 4 parallel network paths to storage
• Automatically Load Balances across all network paths
• Optimises scalability of NAS storage 
• Displays linear scaling over both 1GbE, 10GbE, 40GbE, 100GbE paths
• Vastly reduces system CPU utilisation 
• Reduces system CPU by bypassing file system stack 
• Also available for Microsoft Windows 

Before configuring Oracle dNFS, I always recommended confirming that the Linux NFS exports can be mounted and used ok, ruling-out any storage array, network, operating system and access permission issues.

For RAC consult: Oracle MOS Doc ID 359515.1 Mount Options for Oracle files for RAC databases and Clusterware when used with NFS on NAS devices.

Getting Started

Logon to the FlashArray, click Storage from the left-hand panel and select File Systems.

Create File System

Navigate to the File Systems panel and click +

Provide File System Name, for example oracle.

Create Directory

Click on the newly created File System oracle.

Navigate to the Directories panel and click +

Provide Directory Name and Path for example cdb2-u02 & /cdb2-u02.

And repeat for any other required filesystems, for example cdb2-u03

Add Export Policies

I have pre-created a policy called nfs-no-root-squash with the following settings:

• Rules
  • Clients: Only the Oracle Database nodes should be added as clients 
  • Access: no-root-squash 
  • permission: rw
  • Version: NFSv3
• Details
  • Type: NFS
  • Enabled: True
  • User Mapping: false
  • Version: NFSv3

Select the newly created Directory and from the Policies panel click the 3 vertical pips and select Add Export Policies.

That’s the FlashArray complete, now over to the Oracle database server.

For detailed information review: Oracle MOS Doc ID 762374.1: Step by Step – Configure Direct NFS Client (DNFS) on Linux

Oracle Database 19c

If your Oracle Home has been previously linked with dnfs, you may be surprised to see that Oracle dNFS is already in use, query the v$dnfs_servers view to see if dNFS is in use, for example:

select svrname, dirname FROM v$dnfs_servers;

Server					 Export Name
---------------------------------------- -----------
z-x70-a-fastfile			 /cdb2-u02
z-x70-a-fastfile			 /cdb2-u03

In the example above Oracle has automatically configured dNFS using the NFS mount information in the /etc/fstab file.

If not enabled, you will need to shutdown your database and link Oracle.

Change the directory to $ORACLE_HOME/rdbms/lib and perform the below.

cd $ORACLE_HOME/rdbms/lib
make -f ins_rdbms.mk dnfs_on

Example output:

[oracle@z-rac2 lib]$ make -f ins_rdbms.mk dnfs_on
rm -f /u01/app/oracle/product/19.0.0/dbhome_1/rdbms/lib/odm/libnfsodm19.so; \
cp /u01/app/oracle/product/19.0.0/dbhome_1/lib/libnfsodm19.so /u01/app/oracle/product/19.0.0/dbhome_1/rdbms/lib/odm/libnfsodm19.so

Create an oranfstab File

Change the directory to $ORACLE_HOME/dbs and create/edit a file called orafstab.

The parameters used in my oranfstab are described below:

server
The NFS server name, this can be anything and is used for information and does not need to be resolvable by DNS.

local
Up to four paths on the database host, specified by IP address or by name.

path
Up to four network paths to the NFS server, specified either by IP address, or by name.

export
The exported path from the NFS server.

mount
The corresponding local mount point for the exported volume.

nfs_version
Specifies the NFS protocol version used by Direct NFS Client. Possible values are NFSv3, NFSv4, NFSv4.1, and pNFS.

The default version is NFSv3. If you select NFSv4.x, then you must configure the value in oranfstab for nfs_version. Specify nfs_version as pNFS, if you want to use Direct NFS with Parallel NFS.

Below is an extract for my FA Files NFS exports on my database lab server.

#x70a
server: x70a
local: 192.168.90.26: path:192.168.90.40
nfs_version: nfsv3
export: /cdb2-u02 mount: /u02
export: /cdb2-u03 mount: /u03

Now bounce your Oracle database, and confirm the Oracle database alert log file includes an Oracle Direct NFS ODM Library entry.

Oracle instance running with ODM: Oracle Direct NFS ODM Library Version 6.0

If the $ORACLE_HOME/dbs/oranfstab file has been updated correctly, we should now see the server name defined within the oranfstab file rather than /etc/fstab name, for example:

Server					 Export Name
---------------------------------------- -----------
x70a					 /cdb2-u02
x70a					 /cdb2-u03

Calibrate IO

Introduced in Oracle 11g the Oracle provided DBMS_RESOURCE_MANAGER.CALIBRATE_IO procedure can be used to provide an indication of the storage subsystem performance.

You need to run calibrate_io from with the container database as it is not an allowed operation from within a pluggable database.

SET LINESIZE 200
SET SERVEROUTPUT ON
DECLARE
  lat INTEGER;
  iops INTEGER;
  mbps INTEGER;
BEGIN
-- DBMS_RESOURCE_MANAGER.CALIBRATE_IO (<DISKS>, <MAX_LATENCY>, iops, mbps, lat);
   DBMS_RESOURCE_MANAGER.CALIBRATE_IO (1000, 10, iops, mbps, lat);
   DBMS_OUTPUT.PUT_LINE ('max_iops = ' || iops);
   DBMS_OUTPUT.PUT_LINE ('latency = ' || lat);
   DBMS_OUTPUT.PUT_LINE ('max_mbps = ' || mbps);
end;
/

Confirm timed_statistics are enabled and run the example SQL above.

SELECT NAME,VALUE FROM V$PARAMETER WHERE NAME='timed_statistics'

For example:

SQL> @calibrate_io

Database Details
===============================================
timed_statistics should be TRUE
filesystemio_options should be SETALL

Name			  Value
------------------------- ----------
timed_statistics	  TRUE

Name			  Value
------------------------- ----------
filesystemio_options	  SETALL

max_iops = 229657
latency  = .443
max_mbps = 5565
max_iops = 229657
latency = 0
max_mbps = 5565

PL/SQL procedure successfully completed.


Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.21.0.0.0

Calibration is extremely disruptive to the database performance. It is strongly recommended to run calibration only when database users can tolerate severe deterioration to database performance. For optimal calibration results, no other database workloads should be running.

Oracle dNFS Views

Oracle 19c provides the following database views to assist with dNFS management.

• v$dnfs_channels – Displays information about the Oracle process connections (channels) open to NFS servers.
• v$dnfs_files – Displays information about the Oracle process files open through Direct NFS.
• v$dnfs_servers – Displays information about the Direct NFS servers accessed by Direct NFS.
• v$dnfs_stats – Displays information about the Oracle process NFS operation statistics issued by Direct NFS.

Oracle also provide a database package to monitor dNFS performance and throughput, I have previously written a blog on how to configure and use MONDNFS, which you can read here.

Summary

In this blog post I have shared how to configure a Pure Storage FlashArray for NFS, and an Oracle 19c database for Oracle Direct NFS (dNFS), and then confirmed usage, by reviewing the Oracle database alert.log files and using dNFS database views.