AWS Interconnect Technologies for Distributed Training — Why AWS Uses EFA

GPU-to-GPU communication can consume 22%–47% of training time, reaching about 43% on H100 systems.

InfiniBand provides lossless RDMA via dedicated hardware and credit-based flow control for low latency and high bandwidth.

RoCE implements RDMA over Ethernet with RoCEv2 adding IP routing but requires PFC and ECN to avoid packet loss and PFC Storms.

AWS developed EFA with the SRD protocol to enable OS-bypass RDMA in cloud environments and recommends enabling EFA for GPU instance communication.