Site icon Ron Ekins' – Oracle Technology, DevOps and Kubernetes Blog

How to configure Oracle 19c database and DirectNFS with NFSv4.1

I have been using NFSv3 and Oracle Direct NFS (dNFS) for many years and over that time have written quite a few Oracle dNFS blogs on its use and configuration.

However, even though NFSv3 remains popular, NFSv4.1 is now the default for many Linux distributions and should therefore be considered for support, performance and security reasons.

So I thought it was time for a NFSv4.1 blog, for this post I will use being using a Pure Storage FlashBlade in my lab.

FlashBlade Configuration

From the FlashBlade UI navigate to Storage -> File Systems and click the + the righthand panel to create a new filesystem, provide Name, Provisioned Size and select NFSv4.1.

In the above I have configured the Export Rules to lockdown access to my databases servers and configured the following options:

NFSv4 introduced a new export option fsid=0 and as per Oracle MOS NOTE: How To Configure NFSv4 (Doc ID 456246.1) this is required to indicate that this is the root filesystem and maybe required on your NFS server.

Linux Configuration

As the root user update the /etc/fstab file to include the new test file systems, for this test I have created 3 file systems, for example.

192.168.4.100:/z-rac_pstgpdb3 /u08/oradata/PSTG/datafile nfs rw,bg,hard,nointr,tcp,vers=4,minorversion=1,sec=sys,timeo=600,actimeo=0  0 0
192.168.4.100:/z-rac_pstgpdb4/oradata       /u09/oradata nfs rw,bg,hard,nointr,tcp,vers=4,minorversion=1,sec=sys,timeo=600,actimeo=0  0 0
192.168.4.100:/z-rac_pstgpdb5                       /u10 nfs rw,bg,hard,nointr,tcp,vers=4,minorversion=1,sec=sys,timeo=600,actimeo=0  0 0

Then create mount points, mount file system and change ownership.

$ mount -t nfs4
192.168.4.100:/z-rac_pstgpdb3 on /u08/oradata/PSTG/datafile type nfs4 (rw,relatime,vers=4.1,rsize=524288,wsize=524288,namlen=255,acregmin=0,acregmax=0,acdirmin=0,acdirmax=0,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.4.2,local_lock=none,addr=192.168.4.100)
192.168.4.100:/z-rac_pstgpdb4/oradata on /u09/oradata type nfs4 (rw,relatime,vers=4.1,rsize=524288,wsize=524288,namlen=255,acregmin=0,acregmax=0,acdirmin=0,acdirmax=0,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.4.2,local_lock=none,addr=192.168.4.100)
192.168.4.100:/z-rac_pstgpdb5 on /u10 type nfs4 (rw,relatime,vers=4.1,rsize=524288,wsize=524288,namlen=255,acregmin=0,acregmax=0,acdirmin=0,acdirmax=0,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.4.2,local_lock=none,addr=192.168.4.100)

The 3 test FlashBlade file systems are mounted under /u08, /u09 & /u10.

$ df -h
192.168.4.100:/z-rac_pstgpdb3                 10T   60G   10T   1% /u08/oradata/PSTG/datafile
192.168.4.100:/z-rac_pstgpdb4/oradata   10T     0   10T   0% /u09/oradata
192.168.4.100:/z-rac_pstgpdb5                  10T     0   10T   0% /u10

Oracle with NFSv4.1

To confirm that our Oracle 19c database can use NFSv4.1 without any additional configuration changes I will create an Oracle tablespace in a new NFSv4.1 file system, for example.

SQL> select * from v$dnfs_servers;

   ID    SVRNAME            DIRNAME    MNTPORT    NFSPORT    NFSVERSION     WTMAX     RTMAX    CON_ID    RDMAENABLE    RDMAPORT    SECURITY 
_____ __________ __________________ __________ __________ _____________ _________ _________ _________ _____________ ___________ ___________ 
    1 fb1v3      /z-rac_pstgpdb1          2049       2049 NFSv3.0          524288    524288         0 No                      0 sys         


SQL> create tablespace test1 datafile '/u10/oradata/PSTG/datafile/test1.dbf' size 30g;

Tablespace created. 

And from the operating system.

$ ls -lah /u10/oradata/PSTG/datafile/
total 31G
drwxr-xr-x. 2 oracle dba   0 Dec 14 10:28 .
drwxr-xr-x. 3 oracle dba   0 Dec  8 15:52 ..
-rw-r-----. 1 oracle dba 31G Dec 14 10:28 test.dbf

Oracle dNFS Configuration

However, to use Oracle dNFSv4.1 we need to provide the Oracle 19c database additional information about the NFSv4.1 exports. This is done by updating the $ORACLE_HOME/dbs/oranfstab file as the oracle user, for example.

server: fb1v41
local: 192.168.4.2 path: 192.168.4.100
local: 192.168.5.2 path: 192.168.5.100
local: 192.168.6.2 path: 192.168.6.100
local: 192.168.7.2 path: 192.168.7.100
export: /z-rac_pstgpdb3                  mount: /u08/oradata/PSTG/datafile
export: /z-rac_pstgpdb4/oradata    mount: /u09/oradata
export: /z-rac_pstgpdb5                 mount: /u10
nfs_version: nfsv4.1
security_default: sys

Note: the oranfstab file is very particular in its parsing and requires the local parameter to be defined before the path.

Bounce the database to read the updated oranfstab file, and repeat for all nodes in a RAC cluster.

OK, so let’s try and create a datafile in our new NFSv4.1 file systems.

Test 1 – using sub-directory

Let’s start by trying to repeat the NFSv4.1 test.

Disappointingly, what we see is the create tablespace statement fails instantly, with an ODM error message.

SQL> select * from v$dnfs_servers;

   ID    SVRNAME            DIRNAME    MNTPORT    NFSPORT    NFSVERSION     WTMAX     RTMAX    CON_ID    RDMAENABLE    RDMAPORT    SECURITY 
_____ __________ __________________ __________ __________ _____________ _________ _________ _________ _____________ ___________ ___________ 
    1 fb1v3      /z-rac_pstgpdb1          2049       2049 NFSv3.0          524288    524288         0 No                      0 sys         
    2 fb1v41     /z-rac_pstgpdb3             0       2049 NFSv4.1          524288    524288         0 No                      0 sys         


SQL> create tablespace test1 datafile '/u10/oradata/PSTG/datafile/test1.dbf' size 30g;
create tablespace test1 datafile '/u10/oradata/PSTG/datafile/test1.dbf' size 30g
*
ERROR at line 1:
ORA-01119: error in creating database file
'/u10/oradata/PSTG/datafile/test1.dbf'
ORA-17503: ksfdopn:11 Failed to open file /u10/oradata/PSTG/datafile/test1.dbf
ORA-17500: ODM err:Device or resource busy

Test 2 – mounting sub-directory

Ok, let’s try another create tablespace test, but this time using an NFSv4.1 sub-mounted directory.

This time the test fails with a ORA error message, however from the alert.log file we can see another ODM error code.

SQL> create tablespace test2 datafile '/u09/oradata/PSTG/datafile/test2.dbf' size 30g;
create tablespace test2 datafile '/u09/oradata/PSTG/datafile/test2.dbf' size 30g
*
ERROR at line 1:
ORA-01157: cannot identify/lock data file 36 - see DBWR trace file
ORA-01110: data file 36: '/u09/oradata/PSTG/datafile/test2.dbf'

From alert.log file
ORA-01157: cannot identify/lock data file 36 - see DBWR trace file
ORA-01110: data file 36: '/u09/oradata/PSTG/datafile/test2.dbf'
ORA-17503: ksfdopn:11 Failed to open file /u09/oradata/PSTG/datafile/test2.dbf
ORA-17500: ODM err:Interrupted system call.

Test 3 – mounting into target directory

Finally, let’s try another create tablespace test, but this time using an NFS4.1 export mounted at the target directory.

This time, the Oracle database is able to create the file and from the alert.log file we can see dNFSv4.1 has been used.

SQL> create tablespace test3 datafile '/u08/oradata/PSTG/datafile/test3.dbf' size 30g;
Tablespace created.

From alert.log file:
create tablespace test3 datafile '/u08/oradata/PSTG/datafile/test3.dbf' size 30g
...
Direct NFS: mount complete dir /z-rac_pstgpdb3 on fb1v41 nfsport 2049 using NFS version 4.1
...

And yes big files are also supported.

SQL> create bigfile tablespace test333 datafile '/u08/oradata/PSTG/datafile/test333.dbf' size 1T;

Tablespace created.

Summary

So from the above we can see Oracle 19c can use NFS4.1 without any issues, however using Oracle dNFSv4.1 can be problematic, and the oranfstab remains a bit fussy.

So I suggest before you migrate from NFSv3 and dNFSv3 to NFSv4.1 and dNFSv4.1 you spend a bit of effort validating your configuration and allow plenty of testing time to get it right.

[twitter-follow screen_name=’RonEkins’ show_count=’yes’]

Exit mobile version