NVMe SAN Guide: NVMe-oF vs iSCSI vs Fibre Channel

NVMe over Fibre Channel (FC-NVMe, also written NVMe/FC) sends NVMe commands across a Fibre Channel SAN instead of SCSI, and supported HBAs run it alongside classic SCSI FCP on the same adapter and fabric. That is also the core of NVMe-oF vs iSCSI: NVMe/TCP and FC-NVMe carry NVMe commands, while iSCSI and FCP carry SCSI.

An NVMe SAN is shared block storage that hosts reach with NVMe commands over a fabric rather than a local PCIe slot: NVMe/TCP, NVMe over RDMA or FC-NVMe. Planning one means choosing that fabric, then confirming host, adapter and array support for it. Because Fibre Channel can carry SCSI through FCP or NVMe through FC-NVMe, the three labels in this comparison do not describe mutually exclusive networks.

The short verdict: on a supported FC SAN, evaluate FC-NVMe first, because it reuses the fabric you already run. On Ethernet, compare NVMe/TCP with iSCSI. iSCSI stays the safe default wherever the host, hypervisor or array lacks a supported NVMe-oF path; released Windows Server 2025, for one, has no in-box NVMe-oF initiator.

For a homelab or small-business SAN, start by checking what the host operating system and storage array support together. Then choose the fabric, path layout and recovery behaviour. A protocol name alone cannot establish application latency or prove that an existing NIC, HBA or array supports the proposed configuration. If the block-storage model itself is new, start with what is a SAN storage area network. For the access controls behind these choices, read SAN zoning and LUN masking explained.

NVMe-oF vs iSCSI: what changes

NVMe-oF extends the NVMe command and queue model beyond a local PCIe connection. NVM Express documents the architecture in its specifications, with separate transport specifications for TCP and RDMA. The older standalone NVMe-oF specification is now a historical reference; its architecture was incorporated into the NVMe Base specification.

iSCSI uses the SCSI command model defined for remote block access in RFC 7143. A host addresses logical units through an iSCSI target. NVMe uses controllers and namespaces, with NVMe Qualified Names identifying hosts and subsystems. These differences affect discovery, access configuration and host software; changing the transport is more than changing an IP address.

Multipathing changes with it. RHEL 9 enables native NVMe multipathing by default, which reads ANA (Asymmetric Namespace Access) path states and offers the numa and round-robin I/O policies, where iSCSI LUNs sit under DM Multipath. On ESXi, NVMe-oF devices go to the High-Performance Plug-in because NMP cannot claim them, and vSphere 8.0 still lists RDMs, SCSI-2 reservations and VAAI hardware acceleration as unsupported on NVMe-oF storage. Check those against any clustering or backup tooling that depends on them.

Scale limits differ too. VMware’s vSphere 7.0 documentation allows 32 namespaces and 128 paths per host for NVMe storage, against 1,024 LUNs and 4,096 paths for SCSI storage. The vSphere 8.0 release notes raise the NVMe-oF figures to 256 namespaces and 4,000 paths.

The practical question is whether a supported NVMe path helps the workload that the existing storage stack must serve. Keep the comparison tied to its read/write mix, I/O size, concurrency and latency objective. There is no standards-defined percentage by which every NVMe-oF deployment beats every iSCSI deployment.

SAN transports at a glance

Compare FCP against FC-NVMe when evaluating command protocols on an existing FC network; compare NVMe/TCP against iSCSI when evaluating TCP-based block storage. The cross pairings, NVMe/TCP vs Fibre Channel and Fibre Channel vs iSCSI, are fabric decisions and get their own sections below.

What is NVMe over Fibre Channel (FC-NVMe)?

NVMe over Fibre Channel is the T11 transport binding that carries NVMe-oF over Fibre Channel; FC-NVMe, NVMe/FC and FC NVMe name the same thing. The T11 committee finalised the first FC-NVMe standard in August 2017, and NVMe and SCSI traffic can run simultaneously on one FC fabric, so it is an upgrade path rather than a new SAN.

Each layer still has to support it:

• Switches. Cisco says NVMe on its MDS and Nexus switches is enabled by a nondisruptive NX-OS upgrade with no hardware changes. Check your switch vendor’s equivalent statement and firmware level.
• HBAs. Driver and firmware must enable NVMe. Broadcom’s Emulex Gen 7 adapters, including the dual-port LPe36002, support NVMe/FC and SCSI FCP concurrently and auto-negotiate 64, 32 or 16Gb FC links from a PCIe 4.0 x8 slot.
• Array. Target ports must present NVMe subsystems, and the storage vendor has to qualify its product for FC-NVMe.
• Hosts. RHEL 9 fully supports NVMe/FC in host mode with certain Broadcom Emulex (lpfc) and Marvell QLogic (qla2xxx) adapters. On ESXi, an NVMe-capable FC HBA and its controller need no configuration; the adapter connects to the targets and controllers it can reach.

Zoning, WWPNs and dual-fabric design carry over unchanged. Above the fabric, the host is identified by the NVMe host NQN that nvme-cli generates, so the array’s host definitions need that value.

Bringing up NVMe/FC on a RHEL 9 host

Red Hat’s NVMe/FC procedure reduces to six steps:

1. Confirm the HBA model, firmware and RHEL release in the HBA and array vendors’ NVMe/FC support lists.

2. Zone the host port to the array’s NVMe-capable target port, exactly as for FCP.

3. Run dnf install nvme-cli. It creates /etc/nvme/hostnqn, the identity the array must allow.

4. Read the port names: cat /sys/class/scsi_host/host*/nvme_info on Emulex, or reload qla2xxx and run dmesg | grep traddr on QLogic.

5. Discover and connect, giving both ends as nn-0x<WWNN>:pn-0x<WWPN>:

   nvme discover --transport fc --traddr nn-0x<target-wwnn>:pn-0x<target-wwpn> --host-traddr nn-0x<host-wwnn>:pn-0x<host-wwpn>
   nvme connect --transport fc --traddr nn-0x<target-wwnn>:pn-0x<target-wwpn> --host-traddr nn-0x<host-wwnn>:pn-0x<host-wwpn> -n <subnqn> -k 5
   nvme list

6. Confirm multipathing: cat /sys/module/nvme_core/parameters/multipath should print Y, and nvme list-subsys should list every expected path. The -k 5 option sets a five-second keep-alive; Red Hat advises raising it if the log reports the keep-alive timer expired.

Choose the NVMe-oF fabric before sizing ports

NVMe/TCP maps NVMe communication onto TCP/IP. The NVM Express TCP transport specification defines that mapping. It does not make an iSCSI target into an NVMe subsystem, and an Ethernet connection alone does not establish end-to-end support. Check the host and array support matrices before treating existing network equipment as a complete solution.

FC-NVMe carries NVMe over Fibre Channel. It can preserve the zoning and fabric-management concepts of an FC deployment, but hardware, firmware and host support still need verification. An existing FC installation is a reason to evaluate FC-NVMe, not proof that every component can carry it.

NVMe over RDMA uses a remote-memory transport. RDMA implementations include RoCE, iWARP and InfiniBand; they do not all have identical network requirements. Confirm the selected transport’s adapter, switch and congestion-control requirements against the supported deployment guide. Avoid applying a RoCE configuration recipe to every RDMA network.

For an Ethernet-based first evaluation, compare the supported NVMe/TCP configuration with the supported iSCSI configuration. Keep RDMA or FC-NVMe in consideration when the available equipment and operational requirements justify them.

NVMe over TCP vs iSCSI

NVMe/TCP and iSCSI both run over TCP on ordinary Ethernet NICs, but NVMe/TCP carries NVMe commands to namespaces while iSCSI carries SCSI commands to LUNs, so each needs its own host and target stack. Reusing an IP network does not reuse an iSCSI session, its authentication settings or its LUN mapping.

Where they differ in practice:

• Ports. iSCSI defaults to TCP 3260 under RFC 7143. NVMe/TCP uses 8009 for discovery and 4420 for storage access, both assigned by IANA to NVM Express, so iSCSI firewall rules do not carry over.
• Host software. Linux has shipped NVMe/TCP host and target drivers since kernel 5.0, released 3 March 2019, and NVM Express says NVMe/TCP software needs no special hardware or firmware. ESXi needs its software NVMe over TCP adapter enabled and controllers added by hand. Windows Server 2025 as released has no in-box NVMe-oF initiator; one for NVMe/TCP and NVMe/RDMA is in Insider preview builds 29558 and later.
• Authentication and multipathing. iSCSI uses CHAP, and RFC 7143 requires implementations to support IPsec. On RHEL 9, NVMe/TCP supports DH-HMAC-CHAP authentication, TLS with a pre-shared key is still a Technology Preview, the Linux nvmet target is unsupported, and DM Multipath is not supported with NVMe/TCP, so paths run under native NVMe multipathing.

Published speed comparisons come from vendors, so treat them as hypotheses to test. Blockbridge, a storage vendor, ran both protocols against its own storage from one Proxmox VE 7.2 host over 100GbE. At queue depth 4, 4K random I/O over NVMe/TCP delivered over 50% more IOPS with 33% less latency; at queue depth 1 the average IOPS gain was 18.2%; at large block sizes iSCSI was marginally ahead. NVM Express also notes that software NVMe/TCP uses more CPU than NVMe/RDMA.

Keep this initial comparison to three checks: whether both ends support the protocol, how storage access is restricted, and how the host recovers after a path disappears. Record the exact host version, array firmware and supported multipath configuration for each candidate. Those details provide a useful basis for a later deployment decision without assuming a speed gain. If iSCSI wins, how to implement SAN storage covers the build order. If the real choice is block versus file for a VMware datastore, see iSCSI vs NFS for VMware datastores.

For target, initiator, CHAP and LUN configuration, see What is an iSCSI LUN? Targets, initiators and MPIO on iSCSI Hub. That guide owns the iSCSI setup steps; the fabric choices remain the focus here.

NVMe over TCP vs Fibre Channel

NVMe/TCP runs on Ethernet NICs and switches you may already own. FC-NVMe needs FC HBAs and FC switches, but inherits a lossless, zoned fabric and shares it with existing FCP LUNs. Both present the same NVMe namespaces and NQNs to the host, so the choice is about the network, not the storage model.

• Delivery. Fibre Channel delivers frames losslessly through a credit mechanism. NVMe/TCP relies on TCP to recover lost segments, so congestion on a shared Ethernet design shows up as retransmits and latency.
• Discovery. On ESXi an FC-NVMe adapter connects to every controller it can reach. The NVMe/TCP adapter needs controllers added, or a compliant NVMe discovery service, which vSphere 8.0 can discover dynamically.
• Mixing. VMware’s rule is “Do not mix transport types to access the same namespace.” Pick one transport per namespace and build redundancy within it.

If FC HBAs, switches and FC skills are already paid for, FC-NVMe is usually the shorter step, because new namespaces sit beside existing FCP LUNs. If the SAN is Ethernet-only, NVMe/TCP avoids buying a second network.

Fibre Channel vs iSCSI

Fibre Channel and iSCSI both carry SCSI commands to LUNs; the difference is the network underneath. Fibre Channel (often searched as fiber channel) uses dedicated HBAs and switches with lossless, credit-based delivery. iSCSI runs over TCP/IP on standard Ethernet, so it is cheaper to start and needs no separate fabric skills, but it shares that network’s congestion behaviour.

Link speeds are the nominal figures in the Fibre Channel speed table. ESXi treats both as SCSI storage with the same per-host limits, so cost, skills and isolation decide this pairing rather than LUN counts.

Fibre Channel: zoning and congestion still matter

Fibre Channel provides a storage fabric with its own addressing and credit-based flow control. Whether it carries FCP or FC-NVMe, the design still needs an explicit map of host ports, target ports and permitted communication.

Plan two independent routes when continued access after a single path failure is a requirement. Document shared switches, inter-switch links and controllers rather than counting cables alone. The zoning and masking guide explains why fabric access and volume access are separate controls.

Credit-based flow control does not remove congestion. When investigating delayed I/O, use the SAN high latency triage guide to separate host queues, fabric symptoms and array activity before changing the transport.

Compare latency using the same workload

Write down the acceptance criteria before evaluating alternatives. Include application latency percentiles, required throughput, concurrent operations and what should happen during a supported failover exercise. Use the same workload and comparable storage layout for each candidate.

Keep normal operation and degraded operation as separate results. A configuration that meets the throughput requirement with every path available may need more capacity when one path is unavailable. Conversely, adding ports does not establish that the application can use them concurrently.

The SAN LUN Capacity Calculator supplies a first estimate of RAID-adjusted space and aggregate FC bandwidth. Its fixed capacity ratios and bandwidth assumptions are planning aids. Use the array’s supported capacity calculation and workload evidence before committing a design.

A decision order for a small SAN

1. List supported combinations. Record the host, adapter, switch, array and firmware requirements for each transport under consideration.
2. Draw storage ownership. Identify which host or coordinated cluster may write to each LUN or namespace.
3. Draw independent paths. Identify the components shared by all routes, including power and switching.
4. Set a workload objective. Define the latency, throughput and recovery requirements that would justify changing the current design.
5. Estimate capacity and bandwidth. Allow for array reserves and the path capacity available during maintenance.
6. Plan the transition. Follow the array and operating-system migration procedures rather than assuming the same data can be exposed safely through two protocols at once.

An existing supported iSCSI deployment can remain a sensible choice when it meets those requirements. A supported NVMe-oF deployment becomes a candidate when its command model, fabric options and host support fit the intended workload. Neither decision removes the need for access controls, backups or documented recovery.

Related on this site

• iSCSI vs NFS for VMware Datastores: Which to Pick
• How to Implement SAN Storage: Design, Paths and Tests

FAQ

is nvme over tcp faster than iscsi

Often, for small random I/O, but not by a fixed margin. Blockbridge’s vendor test on Proxmox VE 7.2 over 100GbE found over 50% more 4K IOPS and 33% less latency at queue depth 4, while iSCSI was marginally ahead at large block sizes. Your NIC, array, host CPU and workload decide the real result.

can fc-nvme and fcp run on the same fibre channel fabric

Yes. NVMe and SCSI traffic can share one Fibre Channel fabric at the same time, and Emulex Gen 7 HBAs run NVMe/FC and SCSI FCP concurrently on the same adapter. Existing FCP LUNs keep working while new NVMe namespaces are added. VMware does warn against reaching the same namespace through more than one transport type.

what port does nvme over tcp use

NVMe/TCP discovery controllers listen on TCP 8009, and I/O controllers normally use TCP 4420; IANA assigns both ports to NVM Express. A host runs nvme discover against 8009, then connects to the subsystem it returns. iSCSI, by comparison, defaults to TCP 3260, so firewall rules from an iSCSI SAN do not carry over.

is fibre channel better than iscsi

Not universally. Fibre Channel gives a dedicated, lossless fabric, 64GFC links and a direct path to FC-NVMe, but it needs HBAs, FC switches and zoning skills. iSCSI runs on Ethernet you already own. For a new small SAN, iSCSI usually wins on cost; a working FC SAN is rarely worth ripping out.