Oracle Direct NFS
In my previous Oracle dNFS Blog post I detailed how to configure Oracle 12cR2 to use Oracle dNFS multi-path, in this Blog I will share some test results that I achieved in my Oracle 12cR2 Pure Storage FlashBlade lab.
Theoretical Limits
A Gigabit Ethernet network can transmit 1,000,000,000 bits per second, as we normally talk about X Bytes per second we need to divide this by 8 as there are 8 bits in a byte.
1GbE = (1,000,000,000 / 8) = 125,000,000 bytes/second
=125,000KB/s
=125MB/s
=0.125GB/s
Unfortunately we can not use all 125,000,000 bytes/second to send data as we have multiple layers of overhead, when using a default MTU of 1500 we can expect to experience ~6% overhead, this gives us:
1GbE= (1,000,000,000 / 8) less 6% overhead = 117,500,000 bytes/second
=117,500KB/s
=117.5MB/s
=0.118GB/s
1 x 1GbE = 0.118GB/s
1 x 10GbE = 1.175GB/s
4 x 10GbE = 4.7GB/s
8 x 10GbE = 9.4GB/s (2 Node RAC Cluster)
The dNFS Throughput Test
In the tests below I used Kevin Closson’s excellent SLOB with a custom configuration to produce Oracle FTS (Full Table Scans), many thanks to Frits Hoogland for sharing. This set-up allowed me to generate sufficient load to utilise the available bandwidth and see how close I can get to the ‘theoretical limits’.
Test 1 – Oracle 12cR2 dNFS disabled
OK, during this test we can see that Oracle EM Express is showing a single instance driving ~1.08GB/s IO Throughput
The Flashblade reporting throughput at ~1.09GB/s
Test 2 – Oracle 12cR2 dNFS enabled
For out next test, I have enabled dNFS with make -f ins_rdbms.mk dnfs_on but have not activated multipath, this was confirmed with the SQL below.
SELECT svrname, dirname, wtmax, rtmax FROM v$dnfs_servers; SVRNAME DIRNAME WTMAX RTMAX ------- ------------------ ------ ------ 192.168.4.100 /z-fbhosts_oradata 524288 524288 192.168.4.100 /z-fbhosts_orafra 524288 524288 192.168.4.100 /z-fbhosts_oraconfig 524288 524288
Oracle Express EM reports almost identical results with and without dNFS
And the Flashblade dashboard reports a similar result.
So lets enable multi-path and see the impact.
Test 3 – Oracle 12cR2 dNFS multi-path
So, lets install our oranfstab file and bounce the database and check again using the previous SQL script.
SELECT svrname, dirname, wtmax, rtmax FROM v$dnfs_servers; SVRNAME DIRNAME WTMAX RTMAX ------- ---------------- ------ ------ flashblade /z-fbhosts_oradata 524288 524288 flashblade /z-fbhosts_orafra 524288 524288 flashblade /z-fbhosts_oraconfig 524288 524288
We can see the Server Name has been picked up from the oranfstab rather than the mtab, we can also check that we are using multipath with the v$dnfs_channels view.
SELECT distinct svrname, path FROM v$dnfs_channels; SVRNAME PATH -------------------- ------------- flashblade 192.168.7.100 flashblade 192.168.4.100 flashblade 192.168.6.100 flashblade 192.168.5.100
Oracle EM Express is now showing we are using all 4 x 10GbE interfaces as expected and driving 4.03GB/s, also note the increase in Host CPU utilisation.
Ok, our final test will be to see if we get similar results with Oracle RAC
Test 4 – Oracle 12cR2 2 node RAC with dNFS multi-path
For this Oracle RAC test the slob.conf SQLNET_SERVICE_MAX parameter has been changed to 2 and to distribute the workload across the 2 nodes.
Oracle EM Express is showing that the workload has been evenly balanced across the 2 RAC nodes, achieving impressive throughput figures exceeding 8.3GB/s.
The Flashblade dashboard show that the 7 blades were able to deliver a consitent bandwidth of over 8.8GB/s for the Oracle workload utilising all 8 x 10GbE interfaces getting close to wire speed.
In my next Blog Post I plan to enable Jumbo Frames (JF) end-to-end and repeat the above tests. With JF (MTU 9000) configured, I am hoping to see a ~5% improvement in bandwidth utilisation.
[twitter-follow screen_name=’RonEkins’ show_count=’yes’]