Oracle Direct NFS (dNFS for short) is an NFS Client functionality integrated directly in the Oracle database software, optimizing the I/O (multi)path to your NFS storage without the overhead of the OS client/kernel software.

In this blog post I’ll describe network considerations, configurations and problems I have encountered during set-ups I have done.

dNFS uses two kinds of NFS mounts, the OS mount of NFS (also referred to as kernel NFS of kNFS) and, of course, Oracle’s database NFS mount, Direct NFS or dNFS.

According to [Direct NFS: FAQ (Doc ID 954425.1)] and [How to configure DNFS to use multiple IPs (Doc ID 1552831.1)], an kNFS mount is needed, although Oracle also claims it will also work on platforms that don’t natively support NFS, e.g. Windows… [Oracle Database 11g Direct NFS Client White Paper] (I don’t know how yet…).

Because dNFS implements multipath I/O internally, these is no need for bonding the interfaces to storage via active-backup or Link Aggregation. However, it’s good practice to bond the OS kNFS connection:

1 - eth0 -\
           - bond0 - OS / kNFS
2 - eth1 -/
3 - eth2 --------- - dNFS path 1
4 - eth3 --------- - dNFS path 2

Above schematic shows [How to configure DNFS to use multiple IPs (Doc ID 1552831.1)]:
“A good solution could be to use bonded NICs (…) to perform the mount and then use unbonded NICs via dNFS for the performance critical path.”

Kernel NFS mount

Mounting NFS on filesystem level for Oracle files does need special mounting options for different OS’es and Oracle files. Please check [Mount Options for Oracle files when used with NFS on NAS devices (Doc ID 359515.1)] for these different options.

Single path dNFS

The easiest setup is a single path route to the NFS filer, just set up kNFS as you normally would and enable the NFS ODM libraries (see ‘How to verify dNFS usage’). Even no ‘oranfstab’ file is necessary in this setup, as Oracle can get the settings from an existing kNFS mount from /etc/mtab.

See: ‘No oranfstab’ below.

Multipath dNFS

Same subnets

“Direct NFS client best practices recommend always to use multipaths in separate subnets.”

For multipath dNFS Oracle advises to use different subnets for each interface, but setting up multipath dNFS using for the same subnet is possible [How to Setup Direct NFS client multipaths in same subnet (Doc ID 822481.1)]. The key to this solution is not to let the OS determine the dynamic route of your path, because it will return only the first route possible for both paths.

The document mentioned above shows how this is actually done, using ‘dontroute’ in the ‘oranfstab’ (for POSIX systems) or create static routes on Linux.

Example no-static routing

See what goes wrong here, the routing table tells both traffic-‘lanes’ can go via eth0, because the destination and mask fit for both routes:

interface client        NFS server
eth0      10.0.0.105    10.0.0.205
eth1      10.0.0.106    10.0.0.206

[root@NOPLDB03 network-scripts]# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Iface
default         192.168.2.1     0.0.0.0         UG    eth2
10.0.0.0        *               255.255.255.0   U     eth0  <- first route
10.0.0.0        *               255.255.255.0   U     eth1  <- route ignored
(...)

Traffic for .205 goes via eth0 (.105)
Traffic for .206 also goes via interface eth0 (.105), the first route!

Example with static routing

When creating extra static routes for specifically the eth0 and eth1 interface with those two end-point ip-addresses, those traffic-‘lanes’ will be chosen because of the ‘best fit’ in the destination and mask:

[root@NOPLDB03 network-scripts]# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Iface
default         192.168.2.1     0.0.0.0         UG    eth2
10.0.0.0        *               255.255.255.0   U     eth0
10.0.0.0        *               255.255.255.0   U     eth1
10.0.0.105      10.0.0.205      255.255.255.255 UGH   eth0  <- static route
10.0.0.106      10.0.0.206      255.255.255.255 UGH   eth1  <- static route
(...)

Traffic for .205 goes via eth0 (.105), static route
Traffic for .206 goes via eth1 (.106), static route

Set up set up static routing on Linux

Create a ‘route’ file for each interface and add the route information:

echo "10.0.0.205 via 10.0.0.105" > /etc/sysconfig/network-scripts/route-eth0
echo "10.0.0.206 via 10.0.0.106" > /etc/sysconfig/network-scripts/route-eth1

and run:

/etc/sysconfig/network-scripts/ifup-routes eth0
/etc/sysconfig/network-scripts/ifup-routes eth1

Only one end-point

When having only one ip-address at the destination, the above will still fail. Even there are multiple routes to the end-point with a ‘best fit’ for the destination and mask, the OS will choose the first, always.

When a interface is brought down, with ‘ifdown eth0’ for instance, the routing table is updated, but this will not be done when a nic fails or a cable is pulled.

The only option is to use bonding on those interfaces. dNFS still works and seems to use both routes, but the database will completely hang when cables are pulled because of network issues due to the routing table not being updated.

Different subnets

When there are different subnets, routing will be ‘automatically’ be correct and is actually the way to go:

How to configure DNFS to use multiple IPs (Doc ID 1552831.1)     (on different subnets)

interface client        NFS server
eth0    10.0.0.105      10.0.0.205
eth1    10.0.1.106      10.0.1.206

[root@NOPLDB03 network-scripts]# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Iface
default         192.168.2.1     0.0.0.0         UG    eth2
10.0.0.0        *               255.255.255.0   U     eth0  <- dymanic route
10.0.1.0        *               255.255.255.0   U     eth1  <- dymanic route
(...)

Routing table

Remember, the routing table is very important in the route being followed, the best matched destination/path will be used and if there are multiple ‘best matches’, the first will be used!

The oranfstab file

The dNFS configuration file ‘oranfstab’ is used by Oracle to set up the dNFS paths (channels). If this file is not present, Oracle tries to create the channels using information using the /etc/mtab file (file with currently mounted file systems). In that case, it will create two channels with the same ip path.

No oranfstab

/etc/fstab:
10.0.0.105:/oraclenfs /oradata nfs rw,bg,hard,rsize=32768,wsize=32768,vers=3,nointr,timeo=600,tcp  0 0

/etc/mtab:
10.0.0.105:/oraclenfs /oradata nfs rw,bg,hard,rsize=32768,wsize=32768,nfsvers=3,nointr,timeo=600,tcp,addr=10.0.0.105 0 0

Alert log:
Oracle instance running with ODM: Oracle Direct NFS ODM Library Version 3.0
Direct NFS: channel id [0] path [10.0.0.105] to filer [10.0.0.105] via local [] is UP
Direct NFS: channel id [1] path [10.0.0.105] to filer [10.0.0.105] via local [] is UP

954425.1 “If a volume is not listed in oranfstab, it will also look through the OS mount tab to find a match. Only if that fails, control is handed back to the database and file access is made through kernel NFS.”

/etc/oranfstab

When using a oranfstab file (in /etc or $ORACLE_HOME/dbs) (multi)pathing can be established and load-balancing / HA will be used according the configuration in the file:

server: zfs_filer
local: 10.0.0.205 path: 10.0.0.105
local: 10.0.1.206 path: 10.0.1.106
export: /oraclenfs mount: /oradata

• server: can by any name. Just an identifier in views and logging.
• local: the ip of the interface locally (optional)
• path: the ip of the interface of the NFS filer (up to four)
  – local: is optional, but when local: is used all ‘local:-path:’ pairs must exist, otherwise Oracle mixes up creating channels (first local: will the paired with first path: no matter the position in the file or on the same line).
• export: the NFS export – file system mount pair, same pair as in the /etc/fstab
• dontroute (no colon) packages will be sent to the network interfaces specified by ocal: parameter directly. (only on POSIX, use static routing on Linux).
• mnt_timeout: Specifies (in seconds) the time for which Direct NFS Client should wait for a successful mount before timing out. Default is 600 (ten minutes!).

Multiple exports and paths per export can be set up, for instance:

## - zfs_filer - oradata and orafra
server: zfs_filer
local: 10.0.0.205 path: 10.0.0.105
local: 10.0.1.206 path: 10.0.1.106
mnt_timeout: 10
export: /oraclenfs mount: /oradata
export: /oraclefra mount: /orafra
## - zfs_filer - orabackup
local: 192.168.0.207 path: 192.168.0.107
mnt_timeout: 30
export: /oraclebck mount: /orabackup
## - nfs_nas - oraexpdp
server: nfs_nas
local: 10.0.2.205 path: 10.0.2.105
export: /exportnfs mount: /oraexpdp
etc...

How to enable and verify dNFS usage

When mounting the database data files, Oracle will first load dNFS functionality (if the Direct NFS Client ODM Library is enabled).

On 11.2 you can enable it with the following make command:

cd $ORACLE_HOME/rdbms/lib
make -f ins_rdbms.mk dnfs_on

Otherwise:

cd $ORACLE_HOME/lib
mv libodm11.so libodm11.so_bak
ln -s libnfsodm11.so libodm11.so

If the library is successfully loaded, next line will be logged in the alert.log when the instance is started:

Oracle instance running with ODM: Oracle Direct NFS ODM Library Version 3.0 (or 2.0)

…and ‘select * from v$dnfs_servers’ should return rows.

However, this view entry is not enough! It will only show dNFS libraries are loaded and an ‘oranfstab’ file is found. I had a scenario in which the actual dNFS mount could not be made, and a only a trace file with a certain error was logged. See [Trace Files From dNFS Database: kgnfscrechan … Failed to get root fsinfo … on filer … error 1 (Doc ID 1507212.1)]

The alert.log also must show the following:

Direct NFS: channel id [0] path [10.0.0.105] to filer [ovmm_filer] via local [10.0.0.205] is UP
Direct NFS: channel id [1] path [10.0.1.106] to filer [ovmm_filer] via local [10.0.1.206] is UP

The database will compare the data file names with the NFS mount if dNFS can be used for the files. Check ‘select * from v$dnfs_files’ for all the files subjected to dNFS and ‘select * from v$dnfs_channels’ if the expected channels/paths are set up!

Other issues in practice

Oracle Virtual Box

I have been testing several set-ups with Oracle Virtual Box, but one ‘bug’ I hit in Virtual Box was the bonding of network interfaces. By default the ‘fail_over_mac=0’ bonding option is used, but somehow the internal network (Host-only interface) of Virtual Box does not like the bonding and eth’s to all get the same MAC-address when using mode=1 (active-backup). When using the bonding option ‘fail_over_mac=1’, each interface will keep it’s own MAC address and the bonding will get the MAC-address of the active interface:

BONDING_OPTS="mode=1 miimon=100 fail_over_mac=..."

bond0     Link encap:Ethernet  HWaddr 08:00:27:DB:DF:92
eth0      Link encap:Ethernet  HWaddr 08:00:27:DB:DF:92

fail_over_mac=0 :
eth1      Link encap:Ethernet  HWaddr 08:00:27:DB:DF:92

fail_over_mac=1 :
eth1      Link encap:Ethernet  HWaddr 08:00:27:9C:A0:10

Oracle Linux 5.x

I have been testing with Oracle Linux 5.10 (particular case), maybe this is not an issue with 6.x, but the bonding options for the bonding mode only was accepted when BONDING_OPTIONS was present in the ifcfg-bond(n) file, not in the /etc/modprobe.conf file.

/etc/sysconfig/network-scripts/ifcfg-bond0
BONDING_OPTS="mode=1 miimon=100 primary=eth0 fail_over_mac=1"

/etc/modprobe.conf
alias bond0 bonding
options bond0 mode=1 miimon=100

Creating NFS server/filer export

I had set up my NFS server export on a Linux server I also use for Oracle VM Server repositories on NFS. I copied one of my existing export entries, of course this did not work. I forgot to include the ‘insecure’ export option which took me a long time to figure out. A database trace file was created with ‘Failed to get root fsinfo’, which pointed me in the direction of this note: [Trace Files From dNFS Database: kgnfscrechan … Failed to get root fsinfo … on filer … error 1 (Doc ID 1507212.1)]. (see ‘How to verify dNFS usage’).

/oraclenfs 10.0.0.0/16(rw,sync,all_squash,insecure)

Used documents and references

URL’s

Oracle Database 11g Direct NFS Client White Paper:
http://www.oracle.com/technetwork/articles/directnfsclient-11gr1-twp-129785.pdf

Direct NFS on Oracle FAQ’s
http://www.orafaq.com/wiki/Direct_NFS

Yury Velikanov: Sharing experience implementing Direct NFS
http://www.slideshare.net/yvelikanov/sharing-experience-implementing-direct-nfs#

Pythian – Yury Velikanov: Oracle Direct NFS Configuration File Explained
http://www.pythian.com/blog/oracle-direct-nfs-configuration-file-explained/

Oracle – Enabling Direct NFS Client Oracle Disk Manager Control of NFS
http://docs.oracle.com/cd/E11882_01/install.112/e22489/storage.htm#CWLIN279

Metalink (ok, ok, Oracle Support)

Kernel / OS NFS for Oracle

Mount Options for Oracle files when used with NFS on NAS devices (Doc ID 359515.1)
ZFS Storage Appliance: Oracle Database 11g R2 NFS Mount Point Recommendations (Doc ID 1567137.1)

Direct NFS

Direct NFS: FAQ (Doc ID 954425.1)
Step by Step – Configure Direct NFS Client (DNFS) on Linux (11g) (Doc ID 762374.1)
This note covers some Frequently Asked Questions related to Direct NFS (Doc ID 1496040.1)

Multpaths

How to configure DNFS to use multiple IPs (Doc ID 1552831.1)     (on different subnets)
How to Setup Direct NFS client multipaths in same subnet (Doc ID 822481.1)

Troubleshooting

Trace Files From dNFS Database: kgnfscrechan … Failed to get root fsinfo … on filer … error 1 (Doc ID 1507212.1)
dNFS Hangs for Few Minutes During NIC Failover (Doc ID 1350245.1)
Direct NFS not load balancing the client-side NICs (Doc ID 746656.1)

Happy directing!