IB protocol & transport: layers, QPs, RDMA, subnet manager

InfiniBand was designed from the start as a fabric for computers talking to each other's memory, not for packets between strangers. Its layers, addressing and management look unfamiliar to Ethernet engineers but are simple once the vocabulary is in place — and they explain why the physical layer is held to such tight standards. This page is the protocol view; the optics are in InfiniBand optics.

Layers

Layer	Contents	Notes
Physical	lanes (1x, 4x, 8x, 12x), per-lane signalling (NRZ to HDR's PAM4… see speeds), encoding (8B/10B to QDR, 64B/66B from FDR), FEC (from FDR optional, RS-FEC at HDR/NDR)	the transceiver's world
Link	local routing header (LRH) with LIDs, virtual lanes (VL0–VL14 data, VL15 management), credit-based flow control per VL, link-layer CRC (ICRC/VCRC)	lossless by design
Network	global routing header (GRH) with 128-bit GIDs for multi-subnet routing	rarely used inside one cluster
Transport	queue pairs (QPs) with send/receive queues, service types RC/UC/UD/XRC, segmentation to MTU, ACK/NAK for reliable service	RDMA lives here
Upper	verbs API, MPI, NCCL (via UCX/SHARP), storage (SRP, iSER, NVMe/IB), IPoIB	what applications use

Addressing and identity

Identifier	Size	Assigned by	Meaning
GUID	64-bit	manufacturer	permanent identity of a port, node or system (like a MAC)
LID (local identifier)	16-bit	subnet manager	address within a subnet; forwarding tables use LIDs
GID	128-bit	SM (prefix) + GUID	subnet prefix + GUID; used across subnets and by RoCE
PKey (partition key)	16-bit	SM	membership in a partition (like a VLAN); full/limited members
QPN	24-bit	HCA	queue pair number — the endpoint within a port

Queue pairs and transport services

A queue pair is one send queue and one receive queue in the HCA, owned by an application. Work requests posted to it are executed by hardware; completions land in a completion queue. Service types:

Type	Reliable	Connected	Use
RC — reliable connected	yes (ACK/NAK, retransmit)	yes, one QP per peer	most RDMA traffic; MPI, NCCL, storage
UC — unreliable connected	no	yes	rare
UD — unreliable datagram	no	no, one QP for all peers	management, IPoIB, multicast
XRC — extended reliable connected	yes	shared receive queues across processes	large MPI jobs to cut QP count
RD — reliable datagram	yes	no	essentially unused

RDMA operations (READ, WRITE, atomic) move data directly between application buffers on two hosts without either CPU copying it; SEND/RECEIVE carries messages. Memory must be registered (pinned and given a key) before the HCA may touch it. GPUDirect RDMA extends this to GPU memory (GPU fabrics).

Flow control and MTU

Every link runs credit-based flow control per virtual lane: the receiver advertises buffer space, the sender never transmits more than it can absorb. No frame is dropped for congestion; instead, back-pressure propagates — the same trade-off as Fibre Channel's credits and the reason a single slow or errored link degrades an entire job. Link MTU is 256–4096 bytes (4096 typical); path MTU is negotiated end to end. Service levels (SL) map to VLs to separate traffic classes, and adaptive routing can steer flows around congested links.

The subnet manager

Exactly one subnet manager (SM) is active per subnet (others stand by). It:

Discovers the topology by walking the fabric with directed-route management packets (SMPs on VL15/QP0) — every switch and HCA port.
Assigns LIDs and configures ports (speed, width, MTU, VL arbitration, PKeys).
Computes routes — min-hop, up-down, fat-tree, dragonfly+, adaptive routing — and programs each switch's linear forwarding table.
Sweeps periodically and on traps (port up/down), re-routing around failures.
Runs the subnet administrator (SA) that answers path queries from hosts.

OpenSM (open source) and vendor fabric managers (UFM) implement it; the fabric manager is also where port errors, link speed/width negotiation results and optics DDM are collected fleet-wide (Link reliability & monitoring).

Link negotiation

On link-up both ends negotiate width (1x/2x/4x) and speed (per-lane rate) to the highest common value. A link that comes up at 2x or one generation slower than expected is the InfiniBand equivalent of Ethernet's "links at 10G, not 25G": one lane's fibre or laser, a marginal cable, or a mismatched transceiver generation (Speed & rate). The fabric manager reports the negotiated state; ibstat / ibportstate show it on a host.

Management and diagnostics tools

Tool	Purpose
`ibstat`, `ibstatus`	local port state, rate, width, LID
`ibnetdiscover`	topology dump
`ibdiagnet`	fabric-wide check: errors, speed/width mismatches, duplicate GUIDs, cable info
`perfquery`, `ibqueryerrors`	per-port counters: symbol errors, link downed, link recovery, port rcv errors, excessive buffer overrun, VL15 dropped, FEC counters
`mlxlink`, `mlxcables`	per-port PHY details and cable/transceiver EEPROM, DDM, FEC histogram
`ibportstate`	force speed/width, reset a port

Symbol errors and link recoveries are the physical-layer counters — the equivalents of Ethernet CRC/FEC, tied to Rx power and cleanliness (Fabric).

InfiniBand vs RoCE in one paragraph

RoCE v2 carries the same verbs/RDMA transport over UDP/IP/Ethernet; it replaces the subnet manager with IP routing and PKeys with VLANs, and replaces credit flow control with PFC/ECN (DCQCN) — which must be configured correctly to be lossless (Lossless Ethernet, Interconnects). The transceivers are physically the same; only the identity coding and the host policies differ.

In CodingBox

InfiniBand modules are read like any QSFP/OSFP: identity (including the InfiniBand rate bits in SFF-8636 byte 164 or CMIS applications), per-lane DDM and checksums on Check transceiver and DDM. A port negotiating below its rated speed is worth a bench read of both ends.