SAN Storage Guide
A central server rack is cabled to two storage arrays on a dark navy background, one topped with block cubes and the other with file folders.
storage-networking

iSCSI vs NFS for VMware Datastores: Which to Pick

iSCSI vs NFS for VMware datastores: NFS is simpler to run on a NAS, iSCSI gives multipathing and VMFS features. Pick by array and sync-write needs.

By SAN Storage Guide Editorial · · 8 min read

For most homelabs and small clusters, NFS v3 is the better pick in the iSCSI vs NFS for VMware datastores decision: it is simpler to set up, thin by default, and easier to resize and back up on a NAS. Choose iSCSI when you need Round Robin multipathing, VMFS6 automatic UNMAP, or your array is block-first.

Neither protocol is faster in the abstract. The disks, the network and whether writes are actually made safe decide the latency your VMs see, and those vary far more between a four-bay NAS and a dual-controller array than NFS and iSCSI do. If block storage is new to you, what a SAN storage area network is covers the LUN side before you spend a weekend on it.

How the two were compared

This comparison is built from documentation, not a lab run. The capability and feature-support data comes from Broadcom’s vSphere 8 NFS datastore concepts page, which tabulates NFS 3 against NFS 4.1. Protocol guidance comes from VMware’s iSCSI and NFS best-practice papers and the TechDocs page on software iSCSI port binding. NAS-side behaviour comes from TrueNAS documentation and its community forum.

No throughput or latency figures appear below, deliberately. A published NFS-versus-iSCSI benchmark measures one array, one network and one set of sync settings; change any of them and the ranking can flip, as the ZFS section shows.

iSCSI vs NFS at a glance

NFS 3NFS 4.1iSCSI with VMFS6
What ESXi mountsA file share; the NAS owns the filesystemA file share; the NAS owns the filesystemA block LUN that ESXi formats as VMFS
MultipathingNot supportedSession trunking, if the NAS supports itPort binding plus a path policy such as Round Robin
LockingLock files created on the NFS serverServer-side share reservationsVMFS locking on the LUN
Thick virtual disksNeed a vendor VAAI-NAS plug-in with Reserve SpaceSame as NFS 3Native
Returning deleted space to the arrayDepends on the NAS vendorDepends on the NAS vendorVMFS6 sends UNMAP automatically
Security mechanismsAUTH_SYS onlyAUTH_SYS or Kerberos (krb5, krb5i)CHAP, one-way or mutual
Storage I/O ControlSupportedNot supportedSupported
Growing the datastoreRaise the share quotaRaise the share quotaGrow the LUN, then expand VMFS
VM files visible on the NASYesYesNo, the LUN is opaque to the NAS

What changes when ESXi sees a LUN instead of a share?

With iSCSI, ESXi gets a raw block device and puts its own clustered filesystem, VMFS, on it; the NAS only sees an opaque volume. With NFS, the NAS owns the filesystem and ESXi reads and writes VM files over the network. Almost every practical difference between the two follows from who owns the filesystem.

On NFS, your NAS snapshots and replication see individual VM folders, so restoring one VMDK from a share snapshot is a file copy. The cost is that anything clever needs the NAS vendor’s VAAI-NAS plug-in installed on every host. Broadcom’s page on hardware acceleration for NAS devices lists Full File Clone, Fast File Clone (array-native snapshots and linked clones), Reserve Space and Extended Statistics as plug-in operations. Without Reserve Space, every disk you create is thin.

On iSCSI, a NAS snapshot captures the whole LUN. Restoring one VM means presenting that snapshot to a host and copying the VM out, which is a slower and more nervous restore drill. Run it once before you need it. In return, VMFS6 does its own housekeeping: per Broadcom’s VMFS space reclamation docs, it reclaims free space on thin-provisioned arrays automatically and asynchronously, at 25 to 50 MB per second at the default Low priority. Access control moves to the LUN too; the SAN zoning and LUN masking guide explains why the initiator mapping is the thing to get right once several hosts share a LUN.

Is iSCSI faster than NFS for VMware?

Not inherently. On the same hardware, the gap people report usually comes from write-safety settings, NIC count or NAS software rather than the protocol. ZFS is the clearest example: ESXi’s NFS client requests synchronous writes and ZFS honours them, while a zvol behind iSCSI treats those writes as asynchronous by default, so iSCSI looks faster by skipping a safety step.

The TrueNAS ZFS Primer calls a fast, power-protected SLOG device “a necessity for ESXi over NFS” and says synchronous writes are relatively rare with iSCSI. A long-running TrueNAS community resource on ESXi sync writes spells out the catch: with iSCSI’s default, VM data is written async, “which is hazardous to your VM’s.” The honest fix is the same for both protocols. Put a power-loss-protected SSD in as SLOG and run zfs set sync=always on the VM zvol, or accept the risk for VMs you can rebuild. Setting sync=disabled on an NFS dataset to “fix” its speed is the same bet with a worse failure mode, because it also stops sync writes for ZFS metadata.

The second factor is connection count. NFS 3 cannot multipath, but vSphere 8.0 Update 1 added nConnect, and the NFS networking page allows 4 connections per NFS datastore by default, raisable to 8, with no more than 256 across all mounted NFS datastores. William Lam’s write-up notes it was initially set through ESXCLI or the vSphere API, not the vCenter UI:

# New NFS 3 mount with 4 TCP connections
esxcli storage nfs add -H 192.168.30.6 -s /mnt/nfs -v NFS -c 4

# Change an existing mount
esxcli storage nfs param set -v NFS -c 4

That spreads I/O across more connections. It does not add path failover. iSCSI with port binding, by contrast, creates sessions from every bound VMkernel port to every configured target portal, and the path policy balances I/O and fails over across them.

On a single-NIC ESXi host, none of this matters: you have one path whatever you choose, and the NAS disks bottleneck long before the protocol does. If latency is already hurting, SAN high latency: where to look first separates host queues, network and array before you blame the protocol.

Where NFS wins

  • Setup time. One export, one mount, done. No initiator names, no LUN mapping, no VMFS.
  • Restores. VM folders are plain files on the NAS, so share-level snapshots and replication can restore a single VM with a copy.
  • Small, shared pools. The NAS allocates space as data is written, and the datastore shares one filesystem with your SMB shares instead of a fixed-size LUN. Growing it is a quota change.

Where iSCSI wins

  • Real multipathing. VMware’s iSCSI paper recommends port binding over NIC teaming so the VMkernel multipathing stack handles failover and Round Robin. Port binding requires initiator and target ports on the same subnet unless each VMkernel port has its own gateway.
  • VMFS features without a vendor plug-in. Thick disks are native, and automatic UNMAP works on any array that supports it.
  • Block-first arrays. If the box is a SAN with no file personality, or iSCSI is what the vendor certifies for vSphere, the decision is made for you.
  • A shaky plug-in story. William Lam got Synology’s NFS VAAI plug-in running on vSphere 8.0 Update 1 but noted Synology does not officially support vSphere 8.0 with it. iSCSI sidesteps that dependency entirely.

If you are also weighing iSCSI against NVMe/TCP, NVMe-oF vs iSCSI vs Fibre Channel covers the transport side of that decision.

Three mistakes that cost a weekend

  1. Mixing NFS versions on one share. Broadcom’s NFS file locking page says the two versions’ locking mechanisms are not compatible, and accessing the same virtual disks from both “might result in incorrect behavior and cause data corruption.” ESXi does not stop you, so pin the version on the NAS export.
  2. Treating NFS 4.1 as the automatic upgrade. It adds Kerberos and session trunking, but the vSphere 8 support table shows it losing Storage I/O Control (which works on iSCSI and NFS 3), limiting Site Recovery Manager to vSphere Replication, and restricting Storage DRS to space-based placement and balancing. Session trunking also needs NAS-side support. NFS 3 stays the sensible default unless you need Kerberos.
  3. Letting storage traffic wander. Give storage its own VMkernel adapter and VLAN. Broadcom’s NFS networking page notes that without VMkernel binding, NFS traffic “might unintentionally flow through a random VMkernel adapter” when the preferred one fails. Neither AUTH_SYS on NFS 3 nor CHAP on iSCSI protects data in flight; Broadcom’s CHAP configuration page opens by noting the IP networks iSCSI uses “do not protect the data they transport.”

Which one to pick

One or two ESXi hosts, a Synology, QNAP or TrueNAS box, one storage NIC per host: NFS 3. Pin the version on the export, keep it on a storage VLAN, and if the NAS runs ZFS, budget for a proper SLOG device or accept the sync trade knowingly.

Two or more storage NICs per host, a need for path failover, or a block-only array: iSCSI with port binding and Round Robin, plus sync=always on any ZFS zvol behind it.

A compliance reason to authenticate the datastore mount: NFS 4.1 with Kerberos, accepting the Storage I/O Control and Site Recovery Manager limits.

Already running either one and it works: stay put. Rebuilding a working NFS setup as iSCSI to chase a benchmark screenshot is more work than it’s worth, unless you wanted an excuse to learn multipathing.

FAQ

does nfs 3 support multipathing in esxi

No. Broadcom’s vSphere 8 support table lists multipathing as not supported for NFS 3 and supported for NFS 4.1 through session trunking. Since vSphere 8.0 Update 1, nConnect lets an NFS 3 datastore use up to 4 TCP connections by default, 8 maximum, which adds throughput headroom but not path failover.

can i convert an nfs datastore to iscsi in vmware

Not in place. NFS and VMFS are different datastore types, so you create a new iSCSI datastore and move VMs onto it. With vCenter, Storage vMotion does this while VMs keep running; on a standalone ESXi host, plan on powering VMs off and copying or re-registering them. Keep the old share until every moved VM boots cleanly.

do i need a slog for vmware nfs on truenas

For acceptable write performance, yes. The TrueNAS ZFS Primer calls a fast, power-protected SLOG device a necessity for ESXi over NFS, because ESXi requests synchronous writes and ZFS honours them. Without one, NFS feels slow. Disabling sync to compensate buys speed with the integrity of your VMs and pool if the NAS loses power.

does vmware support thick provisioning on nfs datastores

Yes, but only with the storage vendor’s VAAI-NAS plug-in installed on each host and an array that supports the Reserve Space operation. Without both, the NAS allocates space as data is written and ESXi creates thin disks. Check that your vendor’s plug-in is supported on your exact ESXi build before relying on it.

Sources

  1. NFS Datastore Concepts and Operations in vSphere Environment (vSphere 8.0) — Broadcom TechDocs
  2. NFS File Locking (vSphere 8.0) — Broadcom TechDocs
  3. NFS Networking (vSphere 8.0) — Broadcom TechDocs
  4. vSphere Hardware Acceleration on NAS Devices (vSphere 8.0) — Broadcom TechDocs
  5. Space Reclamation on vSphere VMFS Datastores (vSphere 8.0) — Broadcom TechDocs
  6. Best Practices for Configuring Networking with Software iSCSI (vSphere 7.0) — Broadcom TechDocs
  7. Configuring CHAP Parameters for iSCSI or iSER Storage Adapters (vSphere 8.0) — Broadcom TechDocs
  8. Storage I/O Control Requirements (vSphere 8.0) — Broadcom TechDocs
  9. Best Practices For Running VMware vSphere On iSCSI — VMware
  10. Best Practices For Running NFS with VMware vSphere — VMware
  11. ZFS Primer — TrueNAS Documentation Hub
  12. Sync writes, or: Why is my ESXi NFS so slow, and why is iSCSI faster? — TrueNAS Community
  13. NFS Multi-Connections in vSphere 8.0 Update 1 — William Lam
  14. Synology NFS VAAI Plug-in support for vSphere 8.0 — William Lam
#vmware#vsphere #iscsi #nfs#datastores

Related