Tag: dNFS

Oracle 19c Database, NFSv4.1, Oracle Direct NFS (dNFS) and the Oracle ZFS Storage Appliance simulator

Introduction

In a previous post I have shared how to install and configure the Oracle ZFS Storage Appliance simulator on VirtualBox, and also on VMware Fusion.

In this post I am going to show how we can use an Oracle ZFS Storage Appliance simulator to provide an environment to learn about NFSv4.1 and Oracle Direct NFS (dNFS).

Note: this configuration is for purely educational and training purposes and should not be considered suitable for Production use.

Oracle ZFS Storage VM

On my laptop I have started an Oracle ZFS storage appliance simulator on VirtualBox , and have created a 75G share.

/export/oradata

Linux Server

For my database server I am using a Vagrant built Oracle Linux 7.9 server, the Vagrant and Ansible playbooks are available on my GitHub repo.

[oracle@oracle19c CDB1]$ cat /etc/oracle-release 
Oracle Linux Server release 7.9

Using the Linux showmount command we can see the available exported shares.

[oracle@oracle19c ~]$ showmount -e zfsa
Export list for zfsa:
/export/oradata (everyone)

The rpcinfo command provides a convenient way of checking supported NFS versions, for example.

[root@oracle19c ~]# rpcinfo zfsa |egrep "service|nfs"
   program version netid     address                service    owner
    100003    3    udp6      ::.8.1                 nfs        superuser
    100227    3    udp6      ::.8.1                 nfs_acl    superuser
    100003    3    tcp6      ::.8.1                 nfs        superuser
    100003    4    tcp6      ::.8.1                 nfs        superuser
    100227    3    tcp6      ::.8.1                 nfs_acl    superuser
    100003    3    udp       0.0.0.0.8.1            nfs        superuser
    100227    3    udp       0.0.0.0.8.1            nfs_acl    superuser
    100003    3    tcp       0.0.0.0.8.1            nfs        superuser
    100003    4    tcp       0.0.0.0.8.1            nfs        superuser
    100227    3    tcp       0.0.0.0.8.1            nfs_acl    superuser

The above confirms the ZFSA simulator is exporting an NFSv4.1 share and it’s mountable.

Looking at the Oracle Linux database servers /var/log/messages file we can see NFSv4.1 is registered.

[root@oracle19c ~]# grep NFSv4.1 /var/log/messages
Mar 14 08:56:33 oracle19c kernel: RPC: Registered tcp NFSv4.1 backchannel transport module.

The mount command can be used to show type and version for example.

[root@oracle19c ~]# mount -t nfs4 
zfsa:/export/oradata on /u01/oradata type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.56.20,local_lock=none,addr=192.168.56.101)

Use df -t nfs4 to report file system usage, limit output with type to NFS4, for example. 

[root@oracle19c ~]# df -h -t nfs4 -T
Filesystem           Type  Size  Used Avail Use% Mounted on
zfsa:/export/oradata nfs4   74G  3.4G   71G   5% /u01/oradata

Oracle Database

The Oracle database reads the oranfstab to obtain information about the required dNFS mounts, during the database start-up, therefore any change to the oranfstab file requires a database bounce.

[oracle@oracle19c ~]$ cat $ORACLE_HOME/dbs/oranfstab 
server: zfsa
local: 192.168.56.20
path: 192.168.56.101
export: zfsa:/export/oradata  mount: /u01/oradata
nfs_version: NFS4.1
security_default: sys

Note: the oranfstab file is very particular in its parsing and requires the local parameter to be defined before the path, they can be on the same line or split over two.

Alert Log File

The Oracle alert.log file should include an entry confirming the Oracle Direct NFS ODM library is running (Oracle 19c uses ODM v6.0), and the channels used.

Confirm that path IP address, filer name and the local IP address are as expected.

...
Oracle instance running with ODM: Oracle Direct NFS ODM Library Version 6.0
...
Direct NFS: mount complete dir /export/oradata on zfsa nfsport 2049 using NFS version 4.1
...
Direct NFS: channel id [0] path [192.168.56.101] to filer [zfsa] via local [192.168.56.20] is UP
..

Note: if filer is reporting an IP address rather than a server name, there is likely an issue with the formatting for the oranfstab file.

Oracle 19c Database

From our Oracle 19c database we can also confirm Oracle dNFS is configured correcting by logging on to the database and querying the v$dnfs_servers and v$dns_channels views.

[oracle@oracle19c CDB1]$ sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Fri Mar 11 15:27:52 2022
Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.


Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0

SQL> COLUMN svrname FORMAT A10
SQL> COLUMN dirname FORMAT A20
SQL> SELECT svrname, dirname, nfsversion FROM v$dnfs_servers;

SVRNAME    DIRNAME		NFSVERSION
---------- -------------------- ----------------
zfsa	   /export/oradata	NFSv4.1

SQL> COLUMN path FORMAT A20
SQL> SELECT distinct svrname, path FROM v$dnfs_channels;

SVRNAME    PATH
---------- --------------------
zfsa	   192.168.56.101

dNFS views

Oracle provides a number of v$ views specifically for the management of Oracle dNFS, these can be used to obtain further details and metrics, for example.

  • v$dnfs_stats          – shows dNFS performance statistics
  • v$dnfs_channels  – shows network channels is use by dNFS
  • v$dnfs_servers     – shows dNFS servers
  • v$dnfs_files           – shows files open for dNFS

Oracle dNFS tests

Logon, open and change to the newly created pluggable database (pdb). 

SQL> show con_name;

CON_NAME
------------------------------
CDB$ROOT

SQL> show pdbs;

    CON_ID CON_NAME			  OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
	 2 PDB$SEED			  READ ONLY  NO
	 3 PDB1 			  MOUNTED

SQL> ALTER PLUGGABLE DATABASE pdb1 open;

Pluggable database altered.

SQL> ALTER PLUGGABLE DATABASE pdb1 SAVE STATE;

Pluggable database altered.

SQL> ALTER SESSION SET CONTAINER=pdb1;

Session altered.

Oracle dNFS Test 1

The first test we will try is creating datafile in mounted directory.

SQL> create tablespace test1 datafile '/u01/oradata/test1.dbf' size 1g; 

Tablespace created.

Oracle dNFS Test 2

The second test I will try is creating a datafile in a sub-directory.

SQL> create tablespace test2 datafile '/u01/oradata/CDB1/pdb1/test2.dbf' size 2g; 

Tablespace created.

Oracle dNFS Test 3

The third test I will try is creating bigfile tablespace.

SQL> create bigfile tablespace test3 datafile '/u01/oradata/CDB1/pdb1/test3.dbf' size 3g; 

Tablespace created.

Oracle dNFS test results

Using the Oracle v$dnfs_files view we can confirm the datafile files have been successfully created.

SQL> column filename heading 'File Name' format a50 
SQL> column filesize heading 'File Size MB' format a10
SQL> select filename, to_char(filesize/1024/1024,'9,999,999') as filesize from v$dnfs_files;

File Name					   File Size
-------------------------------------------------- ----------
...
/u01/oradata/test1.dbf			                        1,024
/u01/oradata/CDB1/pdb1/test2.dbf			2,048
/u01/oradata/CDB1/pdb1/test3.dbf			3,072

Checking the Oracle altert.log file we can confirm there were no dNFS issues reported.

...
PDB1(3):create tablespace test1 datafile '/u01/oradata/test1.dbf' size 1g
2022-03-14T11:32:00.151542+00:00
PDB1(3):Completed: create tablespace test1 datafile '/u01/oradata/test1.dbf' size 1g
2022-03-14T11:32:54.580086+00:00
...
PDB1(3):create tablespace test2 datafile '/u01/oradata/CDB1/pdb1/test2.dbf' size 2g
2022-03-14T11:33:29.464193+00:00
PDB1(3):Completed: create tablespace test2 datafile '/u01/oradata/CDB1/pdb1/test2.dbf' size 2g
2022-03-14T11:35:06.973992+00:00
...
PDB1(3):create bigfile tablespace test3 datafile '/u01/oradata/CDB1/pdb1/test3.dbf' size 3g
2022-03-14T11:36:01.222563+00:00
PDB1(3):Completed: create bigfile tablespace test3 datafile '/u01/oradata/CDB1/pdb1/test3.dbf' size 3g
...

Finally, returning to the Oracle ZFS storage VM, we can see the usage.

Share usage

Drop Tablespace

SQL> drop tablespace test1 including contents and datafiles;

Tablespace dropped.

Summary

In this post I have demonstrated how the Oracle ZFS Storage VM can be used to deliver NFS4.1 shares, and shown how an Oracle 19c database can be configured to use Oracle dNFS with NFSv4.1.

[twitter-follow screen_name=’RonEkins’ show_count=’yes’]

1

How to configure Oracle 19c database and DirectNFS with NFSv4.1

Oracle Database 19c

I have been using NFSv3 and Oracle Direct NFS (dNFS) for many years and over that time have written quite a few Oracle dNFS blogs on its use and configuration.

However, even though NFSv3 remains popular, NFSv4.1 is now the default for many Linux distributions and should therefore be considered for support, performance and security reasons.

So I thought it was time for a NFSv4.1 blog, for this post I will use being using a Pure Storage FlashBlade in my lab.

FlashBlade Configuration

From the FlashBlade UI navigate to Storage -> File Systems and click the + the righthand panel to create a new filesystem, provide Name, Provisioned Size and select NFSv4.1.

In the above I have configured the Export Rules to lockdown access to my databases servers and configured the following options:

  • rw mounts of the exported file system are read write
  • no_root_squash allows root user on the NFS client host to access the NFS-mounted directory with the same rights and privileges that the superuser would normally have.
  • insecure allows access using ports higher than 1024
  • sec=sys use local UNIX UIDs and GIDs by using AUTH_SYS to authenticate NFS operations.

NFSv4 introduced a new export option fsid=0 and as per Oracle MOS NOTE: How To Configure NFSv4 (Doc ID 456246.1) this is required to indicate that this is the root filesystem and maybe required on your NFS server.

Linux Configuration

As the root user update the /etc/fstab file to include the new test file systems, for this test I have created 3 file systems, for example.

192.168.4.100:/z-rac_pstgpdb3 /u08/oradata/PSTG/datafile nfs rw,bg,hard,nointr,tcp,vers=4,minorversion=1,sec=sys,timeo=600,actimeo=0  0 0
192.168.4.100:/z-rac_pstgpdb4/oradata       /u09/oradata nfs rw,bg,hard,nointr,tcp,vers=4,minorversion=1,sec=sys,timeo=600,actimeo=0  0 0
192.168.4.100:/z-rac_pstgpdb5                       /u10 nfs rw,bg,hard,nointr,tcp,vers=4,minorversion=1,sec=sys,timeo=600,actimeo=0  0 0

Then create mount points, mount file system and change ownership. 

$ mount -t nfs4
192.168.4.100:/z-rac_pstgpdb3 on /u08/oradata/PSTG/datafile type nfs4 (rw,relatime,vers=4.1,rsize=524288,wsize=524288,namlen=255,acregmin=0,acregmax=0,acdirmin=0,acdirmax=0,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.4.2,local_lock=none,addr=192.168.4.100)
192.168.4.100:/z-rac_pstgpdb4/oradata on /u09/oradata type nfs4 (rw,relatime,vers=4.1,rsize=524288,wsize=524288,namlen=255,acregmin=0,acregmax=0,acdirmin=0,acdirmax=0,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.4.2,local_lock=none,addr=192.168.4.100)
192.168.4.100:/z-rac_pstgpdb5 on /u10 type nfs4 (rw,relatime,vers=4.1,rsize=524288,wsize=524288,namlen=255,acregmin=0,acregmax=0,acdirmin=0,acdirmax=0,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.4.2,local_lock=none,addr=192.168.4.100)

The 3 test FlashBlade file systems are mounted under /u08, /u09 & /u10.

$ df -h
192.168.4.100:/z-rac_pstgpdb3                 10T   60G   10T   1% /u08/oradata/PSTG/datafile
192.168.4.100:/z-rac_pstgpdb4/oradata   10T     0   10T   0% /u09/oradata
192.168.4.100:/z-rac_pstgpdb5                  10T     0   10T   0% /u10

Oracle with NFSv4.1

To confirm that our Oracle 19c database can use NFSv4.1 without any additional configuration changes I will create an Oracle tablespace in a new NFSv4.1 file system, for example.

SQL> select * from v$dnfs_servers;

   ID    SVRNAME            DIRNAME    MNTPORT    NFSPORT    NFSVERSION     WTMAX     RTMAX    CON_ID    RDMAENABLE    RDMAPORT    SECURITY 
_____ __________ __________________ __________ __________ _____________ _________ _________ _________ _____________ ___________ ___________ 
    1 fb1v3      /z-rac_pstgpdb1          2049       2049 NFSv3.0          524288    524288         0 No                      0 sys         


SQL> create tablespace test1 datafile '/u10/oradata/PSTG/datafile/test1.dbf' size 30g;

Tablespace created.

And from the operating system.

$ ls -lah /u10/oradata/PSTG/datafile/
total 31G
drwxr-xr-x. 2 oracle dba   0 Dec 14 10:28 .
drwxr-xr-x. 3 oracle dba   0 Dec  8 15:52 ..
-rw-r-----. 1 oracle dba 31G Dec 14 10:28 test.dbf

Oracle dNFS Configuration

However, to use Oracle dNFSv4.1 we need to provide the Oracle 19c database additional information about the NFSv4.1 exports. This is done by updating the $ORACLE_HOME/dbs/oranfstab file as the oracle user, for example.

server: fb1v41
local: 192.168.4.2 path: 192.168.4.100
local: 192.168.5.2 path: 192.168.5.100
local: 192.168.6.2 path: 192.168.6.100
local: 192.168.7.2 path: 192.168.7.100
export: /z-rac_pstgpdb3                  mount: /u08/oradata/PSTG/datafile
export: /z-rac_pstgpdb4/oradata    mount: /u09/oradata
export: /z-rac_pstgpdb5                 mount: /u10
nfs_version: nfsv4.1
security_default: sys

Note: the oranfstab file is very particular in its parsing and requires the local parameter to be defined before the path.

Bounce the database to read the updated oranfstab file, and repeat for all nodes in a RAC cluster.

OK, so let’s try and create a datafile in our new NFSv4.1 file systems.

Test 1 – using sub-directory

Let’s start by trying to repeat the NFSv4.1 test.

Disappointingly, what we see is the create tablespace statement fails instantly, with an ODM error message.

SQL> select * from v$dnfs_servers;

   ID    SVRNAME            DIRNAME    MNTPORT    NFSPORT    NFSVERSION     WTMAX     RTMAX    CON_ID    RDMAENABLE    RDMAPORT    SECURITY 
_____ __________ __________________ __________ __________ _____________ _________ _________ _________ _____________ ___________ ___________ 
    1 fb1v3      /z-rac_pstgpdb1          2049       2049 NFSv3.0          524288    524288         0 No                      0 sys         
    2 fb1v41     /z-rac_pstgpdb3             0       2049 NFSv4.1          524288    524288         0 No                      0 sys         


SQL> create tablespace test1 datafile '/u10/oradata/PSTG/datafile/test1.dbf' size 30g;
create tablespace test1 datafile '/u10/oradata/PSTG/datafile/test1.dbf' size 30g
*
ERROR at line 1:
ORA-01119: error in creating database file
'/u10/oradata/PSTG/datafile/test1.dbf'
ORA-17503: ksfdopn:11 Failed to open file /u10/oradata/PSTG/datafile/test1.dbf
ORA-17500: ODM err:Device or resource busy

Test 2 – mounting sub-directory

Ok, let’s try another create tablespace test, but this time using an NFSv4.1 sub-mounted directory.

This time the test fails with a ORA error message, however from the alert.log file we can see another ODM error code. 

SQL> create tablespace test2 datafile '/u09/oradata/PSTG/datafile/test2.dbf' size 30g;
create tablespace test2 datafile '/u09/oradata/PSTG/datafile/test2.dbf' size 30g
*
ERROR at line 1:
ORA-01157: cannot identify/lock data file 36 - see DBWR trace file
ORA-01110: data file 36: '/u09/oradata/PSTG/datafile/test2.dbf'

From alert.log file
ORA-01157: cannot identify/lock data file 36 - see DBWR trace file
ORA-01110: data file 36: '/u09/oradata/PSTG/datafile/test2.dbf'
ORA-17503: ksfdopn:11 Failed to open file /u09/oradata/PSTG/datafile/test2.dbf
ORA-17500: ODM err:Interrupted system call.

Test 3 – mounting into target directory

Finally, let’s try another create tablespace test, but this time using an NFS4.1 export mounted at the target directory.

This time, the Oracle database is able to create the file and from the alert.log file we can see dNFSv4.1 has been used. 

SQL> create tablespace test3 datafile '/u08/oradata/PSTG/datafile/test3.dbf' size 30g;
Tablespace created.

From alert.log file:
create tablespace test3 datafile '/u08/oradata/PSTG/datafile/test3.dbf' size 30g
...
Direct NFS: mount complete dir /z-rac_pstgpdb3 on fb1v41 nfsport 2049 using NFS version 4.1
...

And yes big files are also supported.

SQL> create bigfile tablespace test333 datafile '/u08/oradata/PSTG/datafile/test333.dbf' size 1T;

Tablespace created.

Summary

So from the above we can see Oracle 19c can use NFS4.1 without any issues, however using Oracle dNFSv4.1 can be problematic, and the oranfstab remains a bit fussy.

So I suggest before you migrate from NFSv3 and dNFSv3 to NFSv4.1 and dNFSv4.1 you spend a bit of effort validating your configuration and allow plenty of testing time to get it right.

[twitter-follow screen_name=’RonEkins’ show_count=’yes’]

0

Create a website or blog at WordPress.com

Up ↑