Oracle Direct NFS (dNFS for short) is an NFS Client functionality integrated directly in the Oracle database software, optimizing the I/O (multi)path to your NFS storage without the overhead of the OS client/kernel software.
In this blog post I’ll describe network considerations, configurations and problems I have encountered during set-ups I have done.
dNFS uses two kinds of NFS mounts, the OS mount of NFS (also referred to as kernel NFS of kNFS) and, of course, Oracle’s database NFS mount, Direct NFS or dNFS.
According to [Direct NFS: FAQ (Doc ID 954425.1)] and [How to configure DNFS to use multiple IPs (Doc ID 1552831.1)], an kNFS mount is needed, although Oracle also claims it will also work on platforms that don’t natively support NFS, e.g. Windows… [Oracle Database 11g Direct NFS Client White Paper] (I don’t know how yet…).
Because dNFS implements multipath I/O internally, these is no need for bonding the interfaces to storage via active-backup or Link Aggregation. However, it’s good practice to bond the OS kNFS connection:
1 - eth0 -\ - bond0 - OS / kNFS 2 - eth1 -/ 3 - eth2 --------- - dNFS path 1 4 - eth3 --------- - dNFS path 2
Above schematic shows [How to configure DNFS to use multiple IPs (Doc ID 1552831.1)]:
“A good solution could be to use bonded NICs (…) to perform the mount and then use unbonded NICs via dNFS for the performance critical path.”
Kernel NFS mount
Mounting NFS on filesystem level for Oracle files does need special mounting options for different OS’es and Oracle files. Please check [Mount Options for Oracle files when used with NFS on NAS devices (Doc ID 359515.1)] for these different options.
Single path dNFS
The easiest setup is a single path route to the NFS filer, just set up kNFS as you normally would and enable the NFS ODM libraries (see ‘How to verify dNFS usage’). Even no ‘oranfstab’ file is necessary in this setup, as Oracle can get the settings from an existing kNFS mount from /etc/mtab.
See: ‘No oranfstab’ below.
“Direct NFS client best practices recommend always to use multipaths in separate subnets.”
For multipath dNFS Oracle advises to use different subnets for each interface, but setting up multipath dNFS using for the same subnet is possible [How to Setup Direct NFS client multipaths in same subnet (Doc ID 822481.1)]. The key to this solution is not to let the OS determine the dynamic route of your path, because it will return only the first route possible for both paths.
The document mentioned above shows how this is actually done, using ‘dontroute’ in the ‘oranfstab’ (for POSIX systems) or create static routes on Linux.
Example no-static routing
See what goes wrong here, the routing table tells both traffic-‘lanes’ can go via eth0, because the destination and mask fit for both routes:
interface client NFS server eth0 10.0.0.105 10.0.0.205 eth1 10.0.0.106 10.0.0.206 [root@NOPLDB03 network-scripts]# route Kernel IP routing table Destination Gateway Genmask Flags Iface default 192.168.2.1 0.0.0.0 UG eth2 10.0.0.0 * 255.255.255.0 U eth0 <- first route 10.0.0.0 * 255.255.255.0 U eth1 <- route ignored (...)
Traffic for .205 goes via eth0 (.105)
Traffic for .206 also goes via interface eth0 (.105), the first route!
Example with static routing
When creating extra static routes for specifically the eth0 and eth1 interface with those two end-point ip-addresses, those traffic-‘lanes’ will be chosen because of the ‘best fit’ in the destination and mask:
[root@NOPLDB03 network-scripts]# route Kernel IP routing table Destination Gateway Genmask Flags Iface default 192.168.2.1 0.0.0.0 UG eth2 10.0.0.0 * 255.255.255.0 U eth0 10.0.0.0 * 255.255.255.0 U eth1 10.0.0.105 10.0.0.205 255.255.255.255 UGH eth0 <- static route 10.0.0.106 10.0.0.206 255.255.255.255 UGH eth1 <- static route (...)
Traffic for .205 goes via eth0 (.105), static route
Traffic for .206 goes via eth1 (.106), static route
Set up set up static routing on Linux
Create a ‘route’ file for each interface and add the route information:
echo "10.0.0.205 via 10.0.0.105" > /etc/sysconfig/network-scripts/route-eth0 echo "10.0.0.206 via 10.0.0.106" > /etc/sysconfig/network-scripts/route-eth1
/etc/sysconfig/network-scripts/ifup-routes eth0 /etc/sysconfig/network-scripts/ifup-routes eth1
Only one end-point
When having only one ip-address at the destination, the above will still fail. Even there are multiple routes to the end-point with a ‘best fit’ for the destination and mask, the OS will choose the first, always.
When a interface is brought down, with ‘ifdown eth0’ for instance, the routing table is updated, but this will not be done when a nic fails or a cable is pulled.
The only option is to use bonding on those interfaces. dNFS still works and seems to use both routes, but the database will completely hang when cables are pulled because of network issues due to the routing table not being updated.
When there are different subnets, routing will be ‘automatically’ be correct and is actually the way to go:
How to configure DNFS to use multiple IPs (Doc ID 1552831.1) (on different subnets)
interface client NFS server eth0 10.0.0.105 10.0.0.205 eth1 10.0.1.106 10.0.1.206 [root@NOPLDB03 network-scripts]# route Kernel IP routing table Destination Gateway Genmask Flags Iface default 192.168.2.1 0.0.0.0 UG eth2 10.0.0.0 * 255.255.255.0 U eth0 <- dymanic route 10.0.1.0 * 255.255.255.0 U eth1 <- dymanic route (...)
Remember, the routing table is very important in the route being followed, the best matched destination/path will be used and if there are multiple ‘best matches’, the first will be used!
The oranfstab file
The dNFS configuration file ‘oranfstab’ is used by Oracle to set up the dNFS paths (channels). If this file is not present, Oracle tries to create the channels using information using the /etc/mtab file (file with currently mounted file systems). In that case, it will create two channels with the same ip path.
/etc/fstab: 10.0.0.105:/oraclenfs /oradata nfs rw,bg,hard,rsize=32768,wsize=32768,vers=3,nointr,timeo=600,tcp 0 0 /etc/mtab: 10.0.0.105:/oraclenfs /oradata nfs rw,bg,hard,rsize=32768,wsize=32768,nfsvers=3,nointr,timeo=600,tcp,addr=10.0.0.105 0 0 Alert log: Oracle instance running with ODM: Oracle Direct NFS ODM Library Version 3.0 Direct NFS: channel id  path [10.0.0.105] to filer [10.0.0.105] via local  is UP Direct NFS: channel id  path [10.0.0.105] to filer [10.0.0.105] via local  is UP
954425.1 “If a volume is not listed in oranfstab, it will also look through the OS mount tab to find a match. Only if that fails, control is handed back to the database and file access is made through kernel NFS.”
When using a oranfstab file (in /etc or $ORACLE_HOME/dbs) (multi)pathing can be established and load-balancing / HA will be used according the configuration in the file:
server: zfs_filer local: 10.0.0.205 path: 10.0.0.105 local: 10.0.1.206 path: 10.0.1.106 export: /oraclenfs mount: /oradata
- server: can by any name. Just an identifier in views and logging.
- local: the ip of the interface locally (optional)
- path: the ip of the interface of the NFS filer (up to four)
– local: is optional, but when local: is used all ‘local:-path:’ pairs must exist, otherwise Oracle mixes up creating channels (first local: will the paired with first path: no matter the position in the file or on the same line).
- export: the NFS export – file system mount pair, same pair as in the /etc/fstab
- dontroute (no colon) packages will be sent to the network interfaces specified by ocal: parameter directly. (only on POSIX, use static routing on Linux).
- mnt_timeout: Specifies (in seconds) the time for which Direct NFS Client should wait for a successful mount before timing out. Default is 600 (ten minutes!).
Multiple exports and paths per export can be set up, for instance:
## - zfs_filer - oradata and orafra server: zfs_filer local: 10.0.0.205 path: 10.0.0.105 local: 10.0.1.206 path: 10.0.1.106 mnt_timeout: 10 export: /oraclenfs mount: /oradata export: /oraclefra mount: /orafra ## - zfs_filer - orabackup local: 192.168.0.207 path: 192.168.0.107 mnt_timeout: 30 export: /oraclebck mount: /orabackup ## - nfs_nas - oraexpdp server: nfs_nas local: 10.0.2.205 path: 10.0.2.105 export: /exportnfs mount: /oraexpdp etc...
How to enable and verify dNFS usage
When mounting the database data files, Oracle will first load dNFS functionality (if the Direct NFS Client ODM Library is enabled).
On 11.2 you can enable it with the following make command:
cd $ORACLE_HOME/rdbms/lib make -f ins_rdbms.mk dnfs_on
cd $ORACLE_HOME/lib mv libodm11.so libodm11.so_bak ln -s libnfsodm11.so libodm11.so
If the library is successfully loaded, next line will be logged in the alert.log when the instance is started:
Oracle instance running with ODM: Oracle Direct NFS ODM Library Version 3.0 (or 2.0)
…and ‘select * from v$dnfs_servers’ should return rows.
However, this view entry is not enough! It will only show dNFS libraries are loaded and an ‘oranfstab’ file is found. I had a scenario in which the actual dNFS mount could not be made, and a only a trace file with a certain error was logged. See [Trace Files From dNFS Database: kgnfscrechan … Failed to get root fsinfo … on filer … error 1 (Doc ID 1507212.1)]
The alert.log also must show the following:
Direct NFS: channel id  path [10.0.0.105] to filer [ovmm_filer] via local [10.0.0.205] is UP Direct NFS: channel id  path [10.0.1.106] to filer [ovmm_filer] via local [10.0.1.206] is UP
The database will compare the data file names with the NFS mount if dNFS can be used for the files. Check ‘select * from v$dnfs_files’ for all the files subjected to dNFS and ‘select * from v$dnfs_channels’ if the expected channels/paths are set up!
Other issues in practice
Oracle Virtual Box
I have been testing several set-ups with Oracle Virtual Box, but one ‘bug’ I hit in Virtual Box was the bonding of network interfaces. By default the ‘fail_over_mac=0’ bonding option is used, but somehow the internal network (Host-only interface) of Virtual Box does not like the bonding and eth’s to all get the same MAC-address when using mode=1 (active-backup). When using the bonding option ‘fail_over_mac=1’, each interface will keep it’s own MAC address and the bonding will get the MAC-address of the active interface:
BONDING_OPTS="mode=1 miimon=100 fail_over_mac=..."
bond0 Link encap:Ethernet HWaddr 08:00:27:DB:DF:92 eth0 Link encap:Ethernet HWaddr 08:00:27:DB:DF:92
fail_over_mac=0 : eth1 Link encap:Ethernet HWaddr 08:00:27:DB:DF:92
fail_over_mac=1 : eth1 Link encap:Ethernet HWaddr 08:00:27:9C:A0:10
Oracle Linux 5.x
I have been testing with Oracle Linux 5.10 (particular case), maybe this is not an issue with 6.x, but the bonding options for the bonding mode only was accepted when BONDING_OPTIONS was present in the ifcfg-bond(n) file, not in the /etc/modprobe.conf file.
/etc/sysconfig/network-scripts/ifcfg-bond0 BONDING_OPTS="mode=1 miimon=100 primary=eth0 fail_over_mac=1"
/etc/modprobe.conf alias bond0 bonding options bond0 mode=1 miimon=100
Creating NFS server/filer export
I had set up my NFS server export on a Linux server I also use for Oracle VM Server repositories on NFS. I copied one of my existing export entries, of course this did not work. I forgot to include the ‘insecure’ export option which took me a long time to figure out. A database trace file was created with ‘Failed to get root fsinfo’, which pointed me in the direction of this note: [Trace Files From dNFS Database: kgnfscrechan … Failed to get root fsinfo … on filer … error 1 (Doc ID 1507212.1)]. (see ‘How to verify dNFS usage’).
Used documents and references
Oracle Database 11g Direct NFS Client White Paper:
Direct NFS on Oracle FAQ’s
Yury Velikanov: Sharing experience implementing Direct NFS
Pythian – Yury Velikanov: Oracle Direct NFS Configuration File Explained
Oracle – Enabling Direct NFS Client Oracle Disk Manager Control of NFS
Metalink (ok, ok, Oracle Support)
Kernel / OS NFS for Oracle
Mount Options for Oracle files when used with NFS on NAS devices (Doc ID 359515.1)
ZFS Storage Appliance: Oracle Database 11g R2 NFS Mount Point Recommendations (Doc ID 1567137.1)
Direct NFS: FAQ (Doc ID 954425.1)
Step by Step – Configure Direct NFS Client (DNFS) on Linux (11g) (Doc ID 762374.1)
This note covers some Frequently Asked Questions related to Direct NFS (Doc ID 1496040.1)
How to configure DNFS to use multiple IPs (Doc ID 1552831.1) (on different subnets)
How to Setup Direct NFS client multipaths in same subnet (Doc ID 822481.1)
Trace Files From dNFS Database: kgnfscrechan … Failed to get root fsinfo … on filer … error 1 (Doc ID 1507212.1)
dNFS Hangs for Few Minutes During NIC Failover (Doc ID 1350245.1)
Direct NFS not load balancing the client-side NICs (Doc ID 746656.1)
Very nice article.
I haven’t seen a more complete summary anywhere.
Thank you Maris,
When I’m deep into a subject like this, it’s good to document (and share) knowledge…
It could become very handy when I forget details ;)
I have 2 IP’s configured on eth0 (10.174.19.120 ) and eth1 (10.174.19.122 ) and destination side ( NFS ) we have only one IP configured ( 10.174.19.97 ). What settings I need to make in Linux to use multipath ?
Unfortunatly, you can’t. You also need to have two ip’s on the NFS side.
If the two paths are in the same subnet (which is your case), you need to set-up static routing described in ‘Same subnets’.
If you have ip’s in different subnets, the routing is done ‘automatically’ correct by Linux.
I have only 1x IP at the redhat server and 4x IP at the NAS side all belong to the same IP subnet (192.168.44.0/24). Can I just list out the “path:” and skip the “local:” specification and DNFS multipathing still works?
And for the oranfstab, can I use * to specify the mount path on NFS side? e.g. nas1:/export/share/* /mnt/share*
Yes, you can use ‘path:’ without ‘local:’, that last one is optional.
Remember that paths are created with the interface information of the question ‘how do I get there?’. Now this is important when you have multiple IPs at the server side (not your case) and can be a problem when all are in the same subnet.
I’m not sure what the outcome in the alert.log will be. I hope it’s four ‘Direct NFS: channel id’s, so will have four channels, but you only have 1 interface…
Please let me know, I have never actually tested this.
What is your network design because of your choice of 1 server interface? Would that be a 10Gb interface and storage 4x 1Gb?