Getting Up and Running with Oracle 19c on NFSv3

@RonEkins

2 years ago

Background

As many of you may already know, I have been a long time supporter of running Oracle on NFS, having deployed many Oracle E-Business suite, single instance and RAC databases on NFS over years.

However, for many Oracle DBAs NFS (Network File Systems) is still something of an unknown entity, in this post I hope to demystify Oracle on NFS.

What is NFS (Network File System)

NFS is a distributed file system protocol that provides clients access to shared resources like files and directories in file systems located on a server. Clients can remotely read files and write to files on the server without downloading the files locally. NFS enables clients and applications to share data and makes it easier for administrators to centrally manage, share, and back up data.

NFS is still a widely used protocol, there are several versions of NFS, however the most commonly used versions today are NFSv3 and NFSv4.1.

Pure Storage FlashArray Files

With Pure Storage Purity 6.1 we saw the arrival of File support to the Pure Storage family of FlashArrays.

So in this blog post I will share how we can run an Oracle 19c database on Oracle Linux 8 using NFSv3 from a Pure Storage FlashArray.

Getting Started

Logon to the FlashArray, click Storage from the left-hand panel and select File Systems.

Create File System

Navigate to the File Systems panel and click +

Provide File System Name, for example oracle.

Create Directory

Click on the newly created File System oracle.

Navigate to the Directories panel and click +

Provide Directory Name and Path for example cdb2-u02 & /cdb2-u02.

And repeat for any other required filesystems, for example cdb2-u03

Add Export Policies

I have pre-created a policy called nfs-no-root-squash with the following settings:

Rules
- Clients: Only the Oracle Database nodes should be added as clients
- Access: no-root-squash
- permission: rw
- Version: NFSv3
Details
- Type: NFS
- Enabled: True
- User Mapping: false
- Version: NFSv3

Select the newly created Directory and from the Policies panel click the 3 vertical pips and select Add Export Policies.

That’s the FlashArray complete, now over to the Oracle database server.

Oracle Optimal Flexible Architecture (OFA)

With Oracle ASM being the de-facto way of deploying Oracle databases, it’s easy to forget about the reasons and benefits of using the Oracle Optimal Flexible Architecture (OFA).

OFA is a set of configuration guidelines created to ensure well-organised Oracle installations, which ultimatly simplifies administration, support and maintenance, if you want to learn more about OFA check-out the Oracle Database Installation Guide for Linux.

Below is an example OFA compliant layout, which I will be using for the blog post:

Description	Example
Oracle Base	/u01/app/oracle
Oracle Home	$ORACLE_BASE/product/19.0.0/dbhome_1
Oracle data files directories	/u02/app/oracle/oradata
Oracle recovery files	/u03/app/oracle/fast_recovery_area/

OFA Layout

Database Server Operating System

The sunrpc.tcp_slot_table_entries Kernel parameter sets the number of (TCP) RPC entries to pre-allocate for in-flight RPC requests (essentially the minimum).

As the root user check the value of sunrpc.tcp_slot_table_entries and increase to 128 if required.

[root@z-rac2 ~]# sysctl -a | grep tcp.*.slot_table
sunrpc.tcp_max_slot_table_entries = 65536
sunrpc.tcp_slot_table_entries = 2

To have the system immediately apply the updated value in /etc/sysctl.conf file, run sysctl -p

[root@z-rac2 ~]# echo "sunrpc.tcp_max_slot_table_entries = 128" >> /etc/sysctl.conf
[root@z-rac2 ~]# sysctl -p
fs.file-max = 6815744
kernel.sem = 250 32000 100 128
kernel.shmmni = 4096
kernel.shmall = 1073741824
kernel.shmmax = 4398046511104
kernel.panic_on_oops = 1
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576
net.ipv4.conf.all.rp_filter = 2
net.ipv4.conf.default.rp_filter = 2
fs.aio-max-nr = 1048576
net.ipv4.ip_local_port_range = 9000 65500
sunrpc.tcp_max_slot_table_entries = 128

To make the sunrpc.tcp_max_slot_table_entries persistent update the /etc/modprobe.d/sunrpc.conf file to include:

options sunrpc tcp_max_slot_table_entries=128

After a reboot check the value using the below.

[root@z-rac2 ~]# cat /proc/sys/sunrpc/tcp_max_slot_table_entries
128

Now, create any required mount points, for example /u02 & /u03, update the /etc/fstab file with the new mounts and mount the filesystems.

Extract from my /etc/fstab file

#x70
z-x70-a-fastfile:/cdb2-u02              /u02		nfs  rw,bg,hard,nointr,tcp,actimeo=0,vers=3,timeo=600 0 0

z-x70-a-fastfile:/cdb2-u03              /u03		nfs  rw,bg,hard,nointr,tcp,actimeo=0,vers=3,timeo=600 0 0

We can review and confirm the mount options using mount -t nfs, from the below we can see the FlashArray Files supports a read and write transfer size of 1MB.

[root@z-rac2 ~]# mount -t nfs | grep /u02; mount -t nfs | grep /u03
z-x70-a-fastfile:/cdb2-u02 on /u02 type nfs (rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,acregmin=0,acregmax=0,acdirmin=0,acdirmax=0,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.90.40,mountvers=3,mountport=2049,mountproto=tcp,local_lock=none,addr=192.168.90.40)
z-x70-a-fastfile:/cdb2-u03 on /u03 type nfs (rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,acregmin=0,acregmax=0,acdirmin=0,acdirmax=0,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.90.40,mountvers=3,mountport=2049,mountproto=tcp,local_lock=none,addr=192.168.90.40)

We can now create any required directories, for example

[root@z-rac2 ~]# mkdir -p /u02/app/oracle/oradata/
[root@z-rac2 ~]# chown -R oracle:oinstall /u02

[root@z-rac2 ~]# mkdir -p /u03/app/oracle/fast_recovery_area/
[root@z-rac2 ~]# chown -R oracle:oinstall /u03

For the latest recommended OS specific Oracle NFS mount options consult Mount Options for Oracle files for RAC databases and Clusterware when used with NFS on NAS devices (Doc ID 359515.1)

Oracle Database Creation

Now we have our NFS exports mounted, we are ready to create a new database, for this blog post I will use the Oracle Database Configuration Assistant (DBCA) tool in silent mode.

dbca -silent -createDatabase \
 -templateName General_Purpose.dbc \
 -gdbname cdb2 -sid cdb2 -responseFile NO_VALUE \
 -characterSet AL32UTF8 \
 -sysPassword OraPasswd1 \
 -systemPassword OraPasswd1 \
 -createAsContainerDatabase true \
 -numberOfPDBs 1 \
 -pdbName pdb2 \
 -pdbAdminPassword OraPasswd1 \
 -databaseType MULTIPURPOSE \
 -memoryMgmtType auto_sga \
 -totalMemory 4096 \
 -storageType FS \
 -datafileDestination "/u02/app/oracle/oradata/" \
 -recoveryAreaDestination "/u03/app/oracle/fast_recovery_area/" \
 -recoveryAreaSize 300000 \
 -enableArchive true \
 -useOMF true \
 -redoLogFileSize 4096 \
 -emConfiguration NONE \
 -ignorePreReqs

Oracle File System Options

For Oracle database on NFS I generally recommend setting filesystemio_options to setall.

The Oracle database filesystemio_options parameter has no effect in Oracle dNFS and ASM environments. The use of dNFS or ASM automatically results in the use of both asynchronous and direct I/O.

However, since we can always fall back to the OS NFS client in case of mis-configuration, it’s good practice to set filesystemio_options to setall, as a precaution, even if we don’t plan to use NFS.

alter system set filesystemio_options=SETALL scope=spfile;

Note filesystemio_options is not a dynamic parameter so a database bounce is required for it to take effect.

Oracle dNFS

If your Oracle Home has been previously linked with dnfs, you may be surprised to see you are already using Oracle dNFS, use the v$dnfs_servers view to check dNFS is in use, for example.

SQL> select svrname, dirname, mntport, nfsport, wtmax, rtmax FROM v$dnfs_servers;
no rows selected

Calibrate IO

Introduced in Oracle 11g the Oracle provided DBMS_RESOURCE_MANAGER.CALIBRATE_IO procedure can be used to provide an indication of the storage subsystem performance.

Calibration is extremely disruptive to the database performance. It is strongly recommended to run calibration only when database users can tolerate severe deterioration to database performance. For optimal calibration results, no other database workloads should be running.

SET LINESIZE 200
SET SERVEROUTPUT ON
DECLARE
  lat INTEGER;
  iops INTEGER;
  mbps INTEGER;
BEGIN
-- DBMS_RESOURCE_MANAGER.CALIBRATE_IO (<DISKS>, <MAX_LATENCY>, iops, mbps, lat);
   DBMS_RESOURCE_MANAGER.CALIBRATE_IO (1000, 10, iops, mbps, lat);
   DBMS_OUTPUT.PUT_LINE ('max_iops = ' || iops);
   DBMS_OUTPUT.PUT_LINE ('latency = ' || lat);
   DBMS_OUTPUT.PUT_LINE ('max_mbps = ' || mbps);
end;
/

Confirm timed_statistics are enabled and run the example SQL above.

SELECT NAME,VALUE FROM V$PARAMETER WHERE NAME='timed_statistics'

For example,

SQL> @calibrate_io

Database Details
===============================================
timed_statistics should be TRUE
filesystemio_options should be SETALL

Name			  Value
------------------------- ----------
timed_statistics	  TRUE


Name			  Value
------------------------- ----------
filesystemio_options	  SETALL

max_iops = 86977
latency  = .997
max_mbps = 2112
max_iops = 86977
latency = 1
max_mbps = 2112

PL/SQL procedure successfully completed.

Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.21.0.0.0

Linux nconnect

The Linux nconnect mount option allows you to specify the number of connections (network flows) that should be established between the NFS client and NFS endpoint up to a limit of 16.

Traditionally, an NFS client uses a single connection between itself and the endpoint. By increasing the number of network flows, the upper limits of I/O and throughput are increased significantly.

So let’s update the /etc/fstab to include the nconnect=16.

#x70
z-x70-a-fastfile:/cdb2-u02              /u02		nfs  rw,bg,hard,nointr,nconnect=16,tcp,actimeo=0,vers=3,timeo=600 0 0

z-x70-a-fastfile:/cdb2-u03              /u03		nfs  rw,bg,hard,nointr,nconnect=16,tcp,actimeo=0,vers=3,timeo=600 0 0

After remounting the NFS exports we can now see the nconnect option.

[root@z-rac2 ~]# mount -t nfs | grep /u02; mount -t nfs | grep /u03
z-x70-a-fastfile:/cdb2-u02 on /u02 type nfs (rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,acregmin=0,acregmax=0,acdirmin=0,acdirmax=0,hard,proto=tcp,nconnect=16,timeo=600,retrans=2,sec=sys,mountaddr=192.168.90.40,mountvers=3,mountport=2049,mountproto=tcp,local_lock=none,addr=192.168.90.40)
z-x70-a-fastfile:/cdb2-u03 on /u03 type nfs (rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,acregmin=0,acregmax=0,acdirmin=0,acdirmax=0,hard,proto=tcp,nconnect=16,timeo=600,retrans=2,sec=sys,mountaddr=192.168.90.40,mountvers=3,mountport=2049,mountproto=tcp,local_lock=none,addr=192.168.90.40)

nconnect feature is supported on linux clients with a Kernel version higher than 5.3.

During my testing I and did not experience any issues, or see any errors reported in the Oracle alert log file.

In my lab nconnect=16 provided the most signifiant improvements in throughput, as always I would suggest you perform your own testing to validate the results.

SQL> @calibrate_io

Database Details
===============================================
timed_statistics should be TRUE
filesystemio_options should be SETALL

Name			  Value
------------------------- ----------
timed_statistics	  TRUE


Name			  Value
------------------------- ----------
filesystemio_options	  SETALL

max_iops = 125524
latency  = .912
max_mbps = 5563
max_iops = 125524
latency = 1
max_mbps = 5563

PL/SQL procedure successfully completed.

Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.21.0.0.0

Note: I have yet to see any guidance from Oracle on the use of nconnect with Oracle 19c or 23c databases, so maybe something for just for the lab at the moment.

Summary

If you are familiar with running Oracle on NFS, and love the simplicity of working with NFS storage then FlashArray File NFSv3 could be an interesting option to consider.

In my next post I will share some of the benefits, and detail how to implement and test Oracle Direct NFS (dNFS).