Tips for Working with SMB Direct for Windows Server

Starting with the 2012 edition, Microsoft began including a feature called SMB Direct in Windows Server. SMB Direct has the potential to improve the efficiency with which Windows is able to access SMB storage, usually resulting in better overall performance.

SMB Direct is based around the use of RDMA (Remote Direct Memory Access). RDMA allows network data to be transferred directly to application memory, or directly from application memory to the network, without having to involve the CPU. To use RDMA (or SMB Direct, for that matter), your Windows Servers must be equipped with RDMA-compliant network adapters.

The best thing about SMB Direct may be that you don’t have to do anything special to use it. In fact, SMB Direct is enabled by default in Windows Server 2012 and higher operating systems. The server will automatically use SMB Direct if it determines that the required hardware is available.

In most cases, it's best to leave SMB Direct enabled. However, if you discover that SMB Direct is causing problems, you can disable it. You can use PowerShell to determine which of your network adapters support RDMA by using this cmdlet:

Get-NetAdapterRDMA

When the cmdlet returns the list of RDMA-enabled network adapters, make note of the name of the adapter for which you want to disable RDMA. You can then disable the adapter’s RDMA support by using this command:

Disable-NetAdapterRDMA 

You also can disable RDMA support for all of your network adapters by using this command:

Set-NetOffloadGlobalSetting -NetworkDirect Disabled

Although using SMB Direct might best be described as effortless, there are a few important things that you need to know about it. For starters, RDMA is not the only technology supported by Windows Server for the purpose of improving network performance. Another such technology is NIC Teaming.

NIC Teaming works by binding multiple physical network adapters together into a team. The idea behind this is that Windows can automatically distribute network traffic streams across the adapters within the team, potentially allowing much higher throughput than a single network adapter would be able to achieve by itself.

Those who are configuring Windows Server networking should consider RDMA (and SMB Direct) and NIC teaming to be either/or options. While you can create a team of RDMA-compliant network adapters, doing so causes those adapters to be treated as regular network adapters, not RDMA-compliant adapters.

Given a choice between using NIC teaming and SMB Direct, it’s important to remember that SMB Direct is specifically designed for use with SMB storage, whereas NIC teaming is not.

This brings up another important point. One of the most popular use cases for SMB Direct is to allow a Hyper-V server to access remote SMB storage. Oftentimes, especially in smaller organizations, virtual machine components such as checkpoints and virtual hard disk files will be stored on remote SMB storage, rather than residing within direct-attached storage on the Hyper-V server. SMB Direct can help Hyper-V access this storage more efficiently.

Even though Hyper-V does support SMB Direct for remote storage access, SMB Direct usually isn’t supported by the Hyper-V virtual machines.

Virtual machines that need access to the physical network connect to a virtual switch that is bound to a physical network adapter. Hyper-V will allow you to link a virtual switch to an RDMA-compliant physical network adapter. However, the virtual machines connecting to that virtual switch will not recognize that the underlying hardware is RDMA-compliant. Instead, the virtual machines’ virtual network adapter will be treated as a non-RDMA compliant adapter. This means that SMB Direct is supported by the host operating system, but not by the virtual machines.

Finally, it is worth noting that SMB Direct is dependent upon another Windows Server feature called SMB Multichannel. SMB Multichannel is the component that allows Windows Server to determine whether a network adapter is RDMA-compliant. SMB Multichannel can also provide fault tolerance if a server is equipped with multiple RDMA compliant adapters. In any case, SMB Direct has a dependency upon SMB Multichannel.

In most cases, you don’t have to do anything special to use SMB Direct. It just works by default. The important thing, though, is to avoid accidentally configuring Windows Server in a way that causes RDMA support to be ignored.