Fibre Channel vs iSCSI vs NVMe-oF: SAN Transports

Almost every comparison of SAN transports is written as a bandwidth table, which is the least useful way to make the decision. All three of these move blocks fast enough for most workloads. What separates them is what happens when something breaks, who has to be on call for it, and how much of the network you are willing to own.

This is a comparison of Fibre Channel, iSCSI and NVMe over Fabrics as operational commitments. If the underlying vocabulary is new, start with SAN fundamentals: fabrics, zoning, LUN masking and multipathing and come back.

The three transports at a glance

	Fibre Channel	iSCSI	NVMe over Fabrics
Command set	SCSI (FCP)	SCSI over TCP	NVMe
Network	Dedicated FC fabric	Existing or dedicated Ethernet	FC, RDMA Ethernet, or TCP
Host adapter	HBA (required)	Software initiator or NIC	HBA, RNIC, or software
Typical port speeds	16, 32, 64 Gb	10, 25, 100 GbE	Follows the chosen fabric
Access control	Fabric zoning + LUN masking	IP/VLAN + CHAP + masking	Fabric-dependent + NQN masking
Congestion behaviour	Credit-based, lossless	TCP retransmit	Depends on transport
Who runs it	Storage team	Network team, in practice	Both, and that is the problem
Fails toward	Predictability	Familiarity	Throughput

The last three rows decide more deployments than the speed row does.

Fibre Channel: you are buying isolation

Fibre Channel is a purpose-built storage fabric with its own switches, its own host bus adapters, its own addressing and its own operational culture. Nothing else runs on it. That single property is most of the value.

The Fibre Channel Industry Association’s published speed roadmap moved the mainstream from 16GFC to 32GFC to 64GFC, with each generation roughly doubling per-port throughput; 32GFC delivers on the order of 3,200 MB/s per direction. Those numbers matter less than the flow control model behind them. Fibre Channel uses buffer-to-buffer credits, so a sender only transmits when the receiver has advertised room. Frames are not dropped under load in the way Ethernet drops packets, which is why latency on a healthy FC fabric stays inside a narrow band instead of degrading gradually.

The cost is real and it is not only the switch ports. It is a second skill set, a second set of firmware upgrade windows, a second monitoring surface, and HBAs in every host. Organisations keep paying it because the fabric behaves the same on a busy Tuesday as it does at 3 a.m.

The failure mode people underestimate is credit starvation. One misbehaving or slow-draining device holds credits and the congestion spreads back through the fabric to hosts that have nothing to do with it. Fibre Channel notices this and can signal it, but the diagnosis is a fabric-level exercise, not a host-level one. That pattern is covered in SAN latency troubleshooting.

iSCSI: you are buying familiarity

iSCSI carries the same SCSI command set over ordinary TCP/IP, standardised in RFC 7143, which consolidated the original iSCSI RFCs into a single document. It listens on TCP port 3260. Initiators and targets are named with IQNs rather than world-wide names, and authentication is typically CHAP.

The appeal is that everything is already in the building: switches, NICs, cabling, VLANs, and people who understand them. A software initiator ships with every mainstream operating system, so a host needs no storage-specific card at all. For a first SAN, the cost difference against Fibre Channel is not marginal, it is categorical.

What you take on is that the storage network is now a general network with all of a general network’s behaviours. TCP recovers from loss by retransmitting, and retransmission at the wrong moment shows up in an application as an unexplained latency spike rather than as a network alarm. Congestion on an unrelated VLAN can become a storage incident if the switching is shared. This is why iSCSI is deployed on isolated VLANs or physically separate switches by anyone who has run it at scale, and why that isolation is the first thing to check when it misbehaves.

Security also becomes explicit rather than structural. A Fibre Channel initiator cannot see the fabric without being zoned into it. An iSCSI initiator can reach any target it has IP connectivity to, so CHAP, target-side initiator lists and network segmentation are doing work that zoning does for free elsewhere. RFC 3723 covers the IPsec side for deployments that need traffic protection rather than just access control. The practical build is in iSCSI SAN setup: targets, initiators, multipath.

NVMe over Fabrics: you are buying the queue model

NVMe over Fabrics is not a faster cable. It is a different command set carried over a fabric of your choosing.

SCSI was designed around a device that could serve one request at a time, and its queueing reflects that. The NVM Express specification defines a model with up to 65,535 I/O queues, each holding up to 65,535 outstanding commands, mapped to host CPU cores. On a flash array with tens of thousands of concurrent operations, the difference is not the transfer rate, it is the absence of a single deep queue that everything has to funnel through. Latency under concurrency is where NVMe-oF separates from SCSI-based transports, and workloads with shallow queue depths may see very little change.

There are three transport bindings in common use, and choosing among them is a second decision as consequential as the first:

• FC-NVMe runs NVMe over an existing Fibre Channel fabric. It reuses the switches, the zoning and the operational model, and it can run alongside FCP traffic on the same fabric. For a site that already has Fibre Channel, this is by far the lowest-risk route.
• NVMe over RDMA (RoCEv2 or iWARP) delivers the lowest latency on Ethernet but requires a lossless or near-lossless network. That means priority flow control, explicit congestion notification and a switching configuration that has to be right on every hop. Getting this wrong produces a fabric that works in a lab and collapses under production burst.
• NVMe/TCP is the transport binding that runs on unmodified Ethernet, with no special adapters and no lossless requirement. It gives up some latency relative to RDMA and gives back the entire configuration burden. For most teams evaluating NVMe-oF for the first time, this is the honest starting point.

Namespaces replace LUNs and NQNs replace IQNs and WWNs, but the access-control thinking is unchanged: something at the fabric layer decides who can reach the subsystem, and something at the array decides which namespaces they get.

Cost is mostly staffing

Port and adapter prices are easy to compare and are usually not the deciding number. The recurring costs are firmware currency across two fabrics, change windows, monitoring coverage, and the depth of the on-call bench at 3 a.m.

Fibre Channel concentrates that cost in a specialist team and keeps it predictable. iSCSI spreads it across the existing network team, who now own storage-affecting incidents whether or not that is written down anywhere. NVMe over RDMA is the expensive case, because it needs both the Ethernet expertise and a lossless configuration discipline that most Ethernet teams have never had to maintain.

Sizing the hardware side of that decision is what the SAN LUN and NVMe-oF sizer is for: it converts raw capacity, RAID level, port speed and initiator count into usable capacity and aggregate fabric throughput, which is enough to sanity-check a proposed port count before anyone quotes it.

Choosing

• Existing Fibre Channel fabric, flash array, latency-sensitive workload. FC-NVMe. Everything operational stays the same and the queue model changes underneath.
• No SAN today, general virtualisation or file-serving workload, small team. iSCSI on dedicated switching. Revisit in two years.
• No SAN today, latency-sensitive database, and no appetite for Ethernet tuning. Fibre Channel. Pay the specialist cost deliberately rather than discovering it during an outage.
• Ethernet-only mandate and a flash array whose value depends on concurrency. NVMe/TCP first. Move to RDMA only if measurement shows the transport is the limit and the network team can commit to lossless configuration as a standing obligation.
• Mixed estate mid-migration. Run both. Arrays commonly present the same volumes over more than one transport, and a phased cutover per host group is safer than a fabric-wide switch.

What does not change

Whichever transport wins, the controls that prevent the expensive failures are the same ones. Zoning or its equivalent still decides who can see the target. Masking still decides which volumes they get. Multipathing is still mandatory, still needs genuinely independent paths, and still needs to be tested before an outage tests it. Thin provisioning can still oversubscribe an array into a hard stop no matter how fast the wire is.

Transport choice changes the shape of the latency curve and the identity of the team that owns it. It does not change the discipline.