SAN Latency Troubleshooting: Find the Bottleneck

“The SAN is slow” is a conclusion, not a symptom, and it is wrong often enough that starting there wastes the first hour. A block I/O crosses an application, a filesystem, a host queue, an adapter, a fabric, an array port, an array cache and finally some media. Latency is measured at the top and blamed at the bottom, and the layer that actually queued the request is usually somewhere in between.

This is a triage order that works from the symptom outward, with the checks that distinguish each layer from the one below it. It assumes the concepts in SAN fundamentals: fabrics, zoning, LUN masking and multipathing.

Step 0: turn the complaint into a measurement

Before touching a switch, pin down four things: which host, which volume, what time window, and what number moved. “Slow” that means average latency rose from 0.4 ms to 1.1 ms is a different investigation from “slow” that means a p99 spike to 400 ms twice an hour.

Then check whether the symptom is latency, throughput, or a queue, because they are not interchangeable. The relationship is arithmetic: outstanding I/O equals throughput multiplied by latency. If a host is issuing more concurrent requests than before, latency rises even though nothing broke. A workload that doubled its queue depth and doubled its latency is behaving exactly as the maths says it should, and the array is not at fault.

The single most useful discipline is to measure at both ends over the same interval. If the host reports 8 ms service time and the array reports 0.6 ms for the same volume in the same minute, the missing 7.4 ms is above the array, and the array is where most teams spend the outage.

Step 1: the host queue

Start where the request starts. On Linux, iostat -x gives per-device average request time and average queue size; compare the multipath device against the individual paths beneath it. Windows exposes the equivalent through logical disk latency counters.

Two host-side conditions produce array-shaped symptoms:

Queue depth exhaustion. Every layer has a queue limit: a per-LUN limit, a per-adapter limit, and a limit on the array’s target port. When the array’s port queue fills, it returns a TASK SET FULL or QUEUE FULL status, and the host responds by throttling. The visible result is latency, and the invisible cause is that too many initiators are pointed at one target port. This is the classic consolidation failure: nothing changed on the array, four more hosts were zoned to the same port, and everyone got slower at once.

A saturated single path. If multipath is configured but all I/O is riding one path, the host sees adapter-level queueing while the fabric looks idle. Check the per-path counters, not just the aggregate.

Step 2: multipath, which is wrong more often than it looks

multipath -ll reporting all paths active is not the same as all paths working. Three specific misconfigurations produce steady, unexplained latency:

Path grouping inverted on an ALUA array. Arrays that present asymmetric access mark some paths active/optimised and others active/non-optimised. If the path groups are set so that non-optimised paths carry production I/O, every request takes an internal detour across the array’s controller interconnect. Nothing errors. Everything is slower, permanently. This is the highest-value single check in the entire list.

Path selector unsuited to the topology. Round-robin distributes evenly, which is only correct when the paths are actually equivalent. Where they are not — different port speeds, different congestion levels, one path riding a busy inter-switch link — a load-aware selector such as service-time weights by observed throughput instead. The kernel’s device-mapper documentation describes the available selectors and how each one chooses.

Paths that are not independent. Two paths that traverse the same switch, the same inter-switch link or the same array controller are two names for one path. Draw the topology and confirm that no single device appears on every route. This matters for availability and it also matters here, because a shared bottleneck makes both paths slow simultaneously and hides itself by looking symmetric.

Step 3: the fabric

On Fibre Channel, go to the port error counters first, and note that they are cumulative: clear the baselines, wait a defined interval, and read them again. Rising CRC errors, encoding errors, or loss-of-sync events on a specific port point at the physical layer — an optic, a patch lead, a dirty connector — and no amount of array tuning will help. A single degrading optic that corrupts a small fraction of frames generates retries and produces exactly the intermittent-latency profile that gets blamed on the array.

The subtler fabric problem is congestion rather than corruption. Fibre Channel uses buffer-to-buffer credits, so a device that is slow to return credits does not simply slow itself down: it holds credits, back-pressure propagates through the fabric, and unrelated hosts sharing the path degrade. This is the slow-drain pattern, and its signature is that the affected hosts have nothing in common except a switch. Modern fabrics can emit performance-impact notifications identifying the offending device, defined in the T11 Fibre Channel link services work, which turns a multi-hour hunt into a lookup. Check whether the switches support them and whether anyone is collecting them.

On iSCSI or NVMe/TCP, the equivalents are interface discards, pause frames and TCP retransmissions. Retransmits are the number to watch, because TCP hides loss from the application by recovering from it, converting a network fault into a latency symptom with no network alarm attached. Also verify MTU consistency end to end: a jumbo-frame mismatch passes ping, passes login, and degrades on large transfers only. The build-order checks for this are in iSCSI SAN setup: targets, initiators, multipath.

Oversubscribed uplinks belong in this step too. Storage traffic crossing a shared inter-switch link competes with everything else on it, and the storage team usually has no visibility into what else is there.

Step 4: the array

Only now is the array a reasonable suspect. Four conditions account for most of it:

Cache saturation. Write cache absorbs bursts and then has to destage to media. When the destage rate cannot keep up, latency rises sharply rather than gradually, because the array switches from cached to write-through behaviour. The signature is a cliff, not a slope.

A background job. RAID rebuilds, capacity rebalancing, snapshot consolidation and replication catch-up all consume backend bandwidth. Correlate the latency window against the array’s job history before anything else, because it is a two-minute check that closes a meaningful share of cases.

Thin pool pressure. A nearly full thin pool degrades before it fails, and allocation-based dashboards do not show it. Monitor actual pool consumption. Watching allocated capacity while the pool fills is how thin provisioning turns into an outage.

Noisy neighbours. One volume’s workload is served from the same pool and the same controllers as everyone else’s. Per-volume latency across the array, sorted, usually identifies the culprit in one look.

Step 5: when it is not the SAN

If host-side and array-side latency agree, the fabric is clean, and the array’s own metrics are unremarkable, the storage stack is doing what it was asked to do and the request rate is the change. That is a capacity conversation, not a fault. Sizing the port count and aggregate throughput a workload actually needs is what the SAN LUN and NVMe-oF sizer is for, and choosing a transport whose latency curve suits the workload is covered in Fibre Channel vs iSCSI vs NVMe-oF.

Triage order, condensed

1. Define the symptom: host, volume, window, metric.
2. Compare host-reported latency against array-reported latency for the same volume and interval.
3. Check host queue depth and whether QUEUE FULL is being returned.
4. Verify multipath path grouping, selector and genuine path independence.
5. Clear fabric error counters, wait, re-read; look for CRC and sync errors on specific ports.
6. Look for congestion and slow-drain signatures, or TCP retransmits and MTU mismatch on Ethernet.
7. Correlate against array background jobs and cache destage behaviour.
8. Check thin pool consumption as a percentage of real capacity.
9. If everything agrees, size the fabric for the workload it now has.

Working the list in order is slower for the first ten minutes and faster for everything after that. Starting at step 7 because the array is the thing with a support contract is how outages become long ones.