I have been using NFSv3 and Oracle Direct NFS (dNFS) for many years and over that time have written quite a few Oracle dNFS blogs on its use and configuration.
However, even though NFSv3 remains popular, NFSv4.1 is now the default for many Linux distributions and should therefore be considered for support, performance and security reasons.
So I thought it was time for a NFSv4.1 blog, for this post I will use being using a Pure Storage FlashBlade in my lab.
FlashBlade Configuration
From the FlashBlade UI navigate to Storage -> File Systems and click the + the righthand panel to create a new filesystem, provide Name, Provisioned Size and select NFSv4.1.
In the above I have configured the Export Rules to lockdown access to my databases servers and configured the following options:
rwmounts of the exported file system are read writeno_root_squashallows root user on the NFS client host to access the NFS-mounted directory with the same rights and privileges that the superuser would normally have.insecureallows access using ports higher than 1024sec=sysuse local UNIX UIDs and GIDs by usingAUTH_SYSto authenticate NFS operations.
NFSv4 introduced a new export option fsid=0 and as per Oracle MOS NOTE: How To Configure NFSv4 (Doc ID 456246.1) this is required to indicate that this is the root filesystem and maybe required on your NFS server.
Linux Configuration
As the root user update the /etc/fstab file to include the new test file systems, for this test I have created 3 file systems, for example.
192.168.4.100:/z-rac_pstgpdb3 /u08/oradata/PSTG/datafile nfs rw,bg,hard,nointr,tcp,vers=4,minorversion=1,sec=sys,timeo=600,actimeo=0 0 0 192.168.4.100:/z-rac_pstgpdb4/oradata /u09/oradata nfs rw,bg,hard,nointr,tcp,vers=4,minorversion=1,sec=sys,timeo=600,actimeo=0 0 0 192.168.4.100:/z-rac_pstgpdb5 /u10 nfs rw,bg,hard,nointr,tcp,vers=4,minorversion=1,sec=sys,timeo=600,actimeo=0 0 0
Then create mount points, mount file system and change ownership.
$ mount -t nfs4 192.168.4.100:/z-rac_pstgpdb3 on /u08/oradata/PSTG/datafile type nfs4 (rw,relatime,vers=4.1,rsize=524288,wsize=524288,namlen=255,acregmin=0,acregmax=0,acdirmin=0,acdirmax=0,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.4.2,local_lock=none,addr=192.168.4.100) 192.168.4.100:/z-rac_pstgpdb4/oradata on /u09/oradata type nfs4 (rw,relatime,vers=4.1,rsize=524288,wsize=524288,namlen=255,acregmin=0,acregmax=0,acdirmin=0,acdirmax=0,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.4.2,local_lock=none,addr=192.168.4.100) 192.168.4.100:/z-rac_pstgpdb5 on /u10 type nfs4 (rw,relatime,vers=4.1,rsize=524288,wsize=524288,namlen=255,acregmin=0,acregmax=0,acdirmin=0,acdirmax=0,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.4.2,local_lock=none,addr=192.168.4.100)
The 3 test FlashBlade file systems are mounted under /u08, /u09 & /u10.
$ df -h 192.168.4.100:/z-rac_pstgpdb3 10T 60G 10T 1% /u08/oradata/PSTG/datafile 192.168.4.100:/z-rac_pstgpdb4/oradata 10T 0 10T 0% /u09/oradata 192.168.4.100:/z-rac_pstgpdb5 10T 0 10T 0% /u10
Oracle with NFSv4.1
To confirm that our Oracle 19c database can use NFSv4.1 without any additional configuration changes I will create an Oracle tablespace in a new NFSv4.1 file system, for example.
SQL> select * from v$dnfs_servers;
ID SVRNAME DIRNAME MNTPORT NFSPORT NFSVERSION WTMAX RTMAX CON_ID RDMAENABLE RDMAPORT SECURITY
_____ __________ __________________ __________ __________ _____________ _________ _________ _________ _____________ ___________ ___________
1 fb1v3 /z-rac_pstgpdb1 2049 2049 NFSv3.0 524288 524288 0 No 0 sys
SQL> create tablespace test1 datafile '/u10/oradata/PSTG/datafile/test1.dbf' size 30g;
Tablespace created.
And from the operating system.
$ ls -lah /u10/oradata/PSTG/datafile/ total 31G drwxr-xr-x. 2 oracle dba 0 Dec 14 10:28 . drwxr-xr-x. 3 oracle dba 0 Dec 8 15:52 .. -rw-r-----. 1 oracle dba 31G Dec 14 10:28 test.dbf
Oracle dNFS Configuration
However, to use Oracle dNFSv4.1 we need to provide the Oracle 19c database additional information about the NFSv4.1 exports. This is done by updating the $ORACLE_HOME/dbs/oranfstab file as the oracle user, for example.
server: fb1v41 local: 192.168.4.2 path: 192.168.4.100 local: 192.168.5.2 path: 192.168.5.100 local: 192.168.6.2 path: 192.168.6.100 local: 192.168.7.2 path: 192.168.7.100 export: /z-rac_pstgpdb3 mount: /u08/oradata/PSTG/datafile export: /z-rac_pstgpdb4/oradata mount: /u09/oradata export: /z-rac_pstgpdb5 mount: /u10 nfs_version: nfsv4.1 security_default: sys
Note: the oranfstab file is very particular in its parsing and requires the local parameter to be defined before the path.
Bounce the database to read the updated oranfstab file, and repeat for all nodes in a RAC cluster.
OK, so let’s try and create a datafile in our new NFSv4.1 file systems.
Test 1 – using sub-directory
Let’s start by trying to repeat the NFSv4.1 test.
Disappointingly, what we see is the create tablespace statement fails instantly, with an ODM error message.
SQL> select * from v$dnfs_servers;
ID SVRNAME DIRNAME MNTPORT NFSPORT NFSVERSION WTMAX RTMAX CON_ID RDMAENABLE RDMAPORT SECURITY
_____ __________ __________________ __________ __________ _____________ _________ _________ _________ _____________ ___________ ___________
1 fb1v3 /z-rac_pstgpdb1 2049 2049 NFSv3.0 524288 524288 0 No 0 sys
2 fb1v41 /z-rac_pstgpdb3 0 2049 NFSv4.1 524288 524288 0 No 0 sys
SQL> create tablespace test1 datafile '/u10/oradata/PSTG/datafile/test1.dbf' size 30g;
create tablespace test1 datafile '/u10/oradata/PSTG/datafile/test1.dbf' size 30g
*
ERROR at line 1:
ORA-01119: error in creating database file
'/u10/oradata/PSTG/datafile/test1.dbf'
ORA-17503: ksfdopn:11 Failed to open file /u10/oradata/PSTG/datafile/test1.dbf
ORA-17500: ODM err:Device or resource busy
Test 2 – mounting sub-directory
Ok, let’s try another create tablespace test, but this time using an NFSv4.1 sub-mounted directory.
This time the test fails with a ORA error message, however from the alert.log file we can see another ODM error code.
SQL> create tablespace test2 datafile '/u09/oradata/PSTG/datafile/test2.dbf' size 30g; create tablespace test2 datafile '/u09/oradata/PSTG/datafile/test2.dbf' size 30g * ERROR at line 1: ORA-01157: cannot identify/lock data file 36 - see DBWR trace file ORA-01110: data file 36: '/u09/oradata/PSTG/datafile/test2.dbf' From alert.log file ORA-01157: cannot identify/lock data file 36 - see DBWR trace file ORA-01110: data file 36: '/u09/oradata/PSTG/datafile/test2.dbf' ORA-17503: ksfdopn:11 Failed to open file /u09/oradata/PSTG/datafile/test2.dbf ORA-17500: ODM err:Interrupted system call.
Test 3 – mounting into target directory
Finally, let’s try another create tablespace test, but this time using an NFS4.1 export mounted at the target directory.
This time, the Oracle database is able to create the file and from the alert.log file we can see dNFSv4.1 has been used.
SQL> create tablespace test3 datafile '/u08/oradata/PSTG/datafile/test3.dbf' size 30g; Tablespace created. From alert.log file: create tablespace test3 datafile '/u08/oradata/PSTG/datafile/test3.dbf' size 30g ... Direct NFS: mount complete dir /z-rac_pstgpdb3 on fb1v41 nfsport 2049 using NFS version 4.1 ...
And yes big files are also supported.
SQL> create bigfile tablespace test333 datafile '/u08/oradata/PSTG/datafile/test333.dbf' size 1T; Tablespace created.
Summary
So from the above we can see Oracle 19c can use NFS4.1 without any issues, however using Oracle dNFSv4.1 can be problematic, and the oranfstab remains a bit fussy.
So I suggest before you migrate from NFSv3 and dNFSv3 to NFSv4.1 and dNFSv4.1 you spend a bit of effort validating your configuration and allow plenty of testing time to get it right.
[twitter-follow screen_name=’RonEkins’ show_count=’yes’]
Leave a Reply