NVMe SAN storage is shared block storage that hosts reach with NVMe commands over a network, through NVMe/TCP, NVMe over RDMA or NVMe over Fibre Channel, rather than SCSI over iSCSI or FC. NVMe SSDs inside the array do not make it one. For a homelab on 2.5GbE, the practical pick is still iSCSI to an NVMe-backed pool; NVMe/TCP is worth a weekend if your target runs TrueNAS 25.10 or later and you want to learn it.
The reason is arithmetic, not snobbery. A 2.5GbE link moves about 312 MB/s before protocol overhead, while SNIA’s reference figure for one PCIe Gen3 x4 NVMe drive is 3+ GB/s of sustained bandwidth. The wire loses by roughly ten to one, so swapping the command protocol on top of it moves latency at the margins, not throughput. If block storage itself is new, start with what a SAN storage area network is; plenty of homelabs don’t need one at all.
What makes storage an NVMe SAN?
Storage is an NVMe SAN when the host-facing protocol is NVMe, whatever the drives are. Hosts connect to NVMe subsystems and see namespaces, identified by NVMe Qualified Names (NQNs), instead of iSCSI targets and LUNs. An array full of NVMe SSDs that presents iSCSI LUNs is an NVMe-backed SCSI SAN, which is a perfectly good thing to own.
SNIA’s Q&A put the distinction plainly: “NVMe over Fabrics is not a SCSI model, it’s extending the NVMe model onto other fabric types.” The same answer recommends that the front-end fabric and back-end NVMe drives be “bandwidth balanced,” which is the polite version of the arithmetic above.
It matters when shopping. Synology’s SAN Manager specifications cover iSCSI and Fibre Channel, with no NVMe-oF target, so NVMe SSDs in a Synology still sit behind iSCSI LUNs. Fast iSCSI, not an NVMe SAN.
| iSCSI SAN | NVMe-oF SAN | |
|---|---|---|
| Command set | SCSI | NVMe |
| What the host mounts | LUN | Namespace |
| Container on the target | Target | Subsystem |
| Host identity | IQN | Host NQN |
| Access control | LUN masking by initiator | Allowed host NQNs, optional DH-HMAC-CHAP |
| Linux multipath | dm-multipath | Native NVMe multipath |
Which NVMe-oF transport fits a small lab?
NVMe/TCP, almost every time. It runs over ordinary TCP/IP on any Ethernet NIC, and Linux gained NVMe/TCP host and target drivers in kernel 5.0, released 3 March 2019; Kernel Newbies summed it up as “Now NVMe over Fabrics can run on every Ethernet port in the world.” The NVMe/TCP transport specification reached revision 1.3 on 31 July 2026, with TLS and data digests as optional features.
The other two transports cost hardware. NVMe over RDMA needs RDMA-capable NICs, and TrueNAS 25.10 only offers RDMA ports on Enterprise-licensed systems with RDMA hardware. FC-NVMe needs Fibre Channel HBAs and switches. It makes sense if you already own an FC fabric, which almost nobody’s spare room does.
On speed, treat the numbers as vendor numbers. Blockbridge, a storage vendor, ran both protocols against its own storage on Proxmox 7.2 over 100GbE: at queue depth 4, NVMe/TCP delivered over 50% more 4K IOPS with 33% less latency, while iSCSI edged ahead by about 0.1% at large block sizes once bandwidth peaked. That was a 32-core EPYC host, so treat the small-block gain as a hypothesis for an N100 on 2.5GbE.
Who should build one, and who should skip it?
Build an NVMe SAN if you run Proxmox VE against TrueNAS 25.10 or later and want NVMe-oF experience for work, or if small random I/O on shared storage is your real complaint and you already have 10GbE or faster. Skip it on Synology, on a single host where local NVMe wins outright, or anywhere iSCSI already works without a latency problem.
ESXi labs with a TrueNAS target should skip it too. TrueNAS documents that its NVMe/TCP target is incompatible with VMware ESXi, because the Linux kernel target driver lacks the fused commands ESXi needs to bring up paths.
Time cost: an evening to connect one namespace to one host, then a weekend for autoconnect, timeouts, multipath and backups. The first connect feels like magic. The first NAS reboot that leaves LVM missing a physical volume feels like a hobby.
What hardware works?
An N100 or N305 mini-PC makes a fine initiator, because NVMe/TCP asks nothing special of the NIC. ServeTheHome’s review of the N305-based Beelink EQ12 Pro shows the typical shape and its quirks: two 2.5GbE ports on Intel i225-V B3, and the internal NVMe SSD on a PCIe Gen3 x1 link. One port goes to the LAN and one to storage, which means a single storage path unless you add a USB NIC.
The target is the harder half:
- TrueNAS 25.10 “Goldeye” or later. Version 25.10.0 shipped on 28 October 2025 with NVMe over TCP in Community Edition and RDMA in Enterprise. Each subsystem holds one namespace, either a zvol or a file, and ANA path states need an Enterprise license.
- Plain Linux with the kernel nvmet target. Teaches you the most, but Red Hat, for one, does not support nvmet, so you are your own vendor.
Spend on the network before the protocol. Even 10GbE, about 1,250 MB/s raw, sits below a single Gen3 x4 drive.
The stack: TrueNAS target, Proxmox initiator
On TrueNAS, following the 25.10 NVMe-oF documentation:
- Create a zvol for VM storage.
- Enable the NVMe-oF service under System > Services.
- Add a subsystem with the zvol as its namespace and a TCP port on 4420, the default.
- Clear “Allow any host to connect” and add each Proxmox node’s host NQN. Enable DH-CHAP if you want hosts to authenticate.
Step 4 is LUN masking under a new name, and leaving it open is how a namespace gets formatted by the wrong machine. The zoning and masking guide explains why that layer is not optional.
Proxmox has no NVMe-oF type in its storage list, so each node connects with nvme-cli and Proxmox gets shared LVM on top. The commands follow Red Hat’s NVMe/TCP procedure, adapted for Debian; the address and NQN are placeholders:
# On every Proxmox node
apt install nvme-cli
[ -s /etc/nvme/hostnqn ] || nvme gen-hostnqn > /etc/nvme/hostnqn
cat /etc/nvme/hostnqn # add this to the subsystem's allowed hosts
nvme discover --transport=tcp --traddr=10.10.10.1 --trsvcid=4420
nvme connect --transport=tcp --traddr=10.10.10.1 --trsvcid=4420 --nqn=<subsystem-nqn> --ctrl-loss-tmo=1800
nvme list-subsys
cat /sys/module/nvme_core/parameters/multipath # expect Y
# Reconnect at boot
echo "--transport=tcp --traddr=10.10.10.1 --trsvcid=4420 --ctrl-loss-tmo=1800" >> /etc/nvme/discovery.conf
systemctl enable nvmf-autoconnect.service
# On ONE node only, then add vg_nvme as shared LVM in the GUI
nvme list # pick the fabric namespace, not the boot SSD
vgcreate vg_nvme /dev/nvmeXn1Skip multipath-tools here. Red Hat makes native NVMe multipathing the default and states that DM Multipath is not supported with NVMe/TCP. With two storage ports, connect through each and nvme list-subsys should list both paths. Two paths into a single-controller TrueNAS box survive a cable or switch failure, not a NAS reboot. The shared LVM setup and the pull-a-cable drill match the iSCSI build in how to implement SAN storage.
TrueNAS also maintains a Proxmox VE storage plugin with an NVMe/TCP mode that needs Proxmox VE 9.x and TrueNAS 25.10 or later, and uses ZFS for snapshots. Its install command pulls from the alpha branch, which tells you where it sits on the maturity curve.
Storage and backups: what bites at month 9?
The reconnect timeout and the belief that a zvol snapshot is a backup. nvme-cli defines --ctrl-loss-tmo as the “maximum time in seconds to retry reconnect attempts after controller loss.” If a TrueNAS update and reboot outlast it, the host stops retrying and the namespace disappears from under LVM and running VMs.
Set it longer than your slowest NAS reboot; the 1800 seconds above is a choice, not a standard. Shut the VMs down before a NAS update anyway.
A zvol snapshot is a crash-consistent copy on the same pool. It saves you from a bad guest upgrade, not a dead pool. Replicate snapshots to a second ZFS box, or back up at the VM level to Proxmox Backup Server on separate hardware.
The restore drill, once a quarter: restore one VM to a new ID, boot it with networking disconnected, check the application data, delete it and note how long it took. Run the first drill before anything you care about lives on the namespace.
Networking: should NVMe/TCP leave the storage segment?
No. Keep NVMe/TCP on a dedicated VLAN or cable with no gateway. The specification makes TLS optional, and Red Hat still labels NVMe/TCP with TLS a Technology Preview, so assume your blocks cross the wire in the clear. DH-HMAC-CHAP proves which host is connecting; it does not hide the data.
Don’t port-forward 4420, and don’t stretch block storage across a Tailnet or a Cloudflare Tunnel. Reach the TrueNAS UI and Proxmox console over Tailscale and leave the data path local.
Operations: updates, logs and alerts
Update this layer by hand: no Watchtower, no Renovate. TrueNAS releases do touch the NVMe target; 25.10.6 shipped a kernel update fixing out-of-bounds reads in the NVMe target discovery log page and authentication reply handling. Shut down the VMs on the namespace, update TrueNAS, confirm with nvme list-subsys that every path is back, then update Proxmox nodes one at a time.
nvme list-subsys shows path state, and journalctl -k shows why a controller dropped. Alert on fewer paths than expected, pool capacity and a backup job that didn’t run. If it is connected but slow, work through where SAN latency hides before blaming the protocol.
FAQ
does nvme over tcp need special network cards
No. NVMe/TCP runs over standard TCP/IP on any Ethernet NIC, and Linux has shipped NVMe/TCP host and target drivers since kernel 5.0 in March 2019. RDMA-capable NICs matter only for NVMe over RDMA, and Fibre Channel HBAs only for FC-NVMe. A mini-PC’s 2.5GbE port works; it just caps throughput well below one NVMe drive.
can a synology nas do nvme over tcp
No. Synology’s SAN Manager specifications cover iSCSI and Fibre Channel, with no NVMe-oF target listed. NVMe SSDs in a Synology still help, but they sit behind iSCSI LUNs, which makes the box an NVMe-backed iSCSI SAN rather than an NVMe SAN. For an NVMe/TCP target on NAS hardware, look at TrueNAS 25.10 or later.
does truenas support nvme-of
Yes, since TrueNAS 25.10.0, released on 28 October 2025. Community Edition gets NVMe over TCP, while RDMA and ANA path states need an Enterprise license. Each subsystem exports one zvol or file namespace, on port 4420 by default. TrueNAS also warns that its NVMe/TCP target is incompatible with VMware ESXi hosts.
Related across the network
- What Is an iSCSI LUN? Targets, Initiators and MPIO — iscsihub.com
- Unraid iSCSI Target Setup: Plugins, LUNs and CHAP — iscsihub.com
