Direct Memory Access (DMA)

Why DMA matters

• Line-rate networking. A 100 Gbps NIC moves 12.5 GB/s — at PIO that's 12.5 billion four-byte CPU instructions per second, which no CPU can sustain. DMA lets the NIC's silicon be the data mover; the CPU spends a few microseconds setting up rings and processing completions.
• NVMe latency. A modern PCIe Gen4 NVMe SSD does a 4 KB random read in ~10 microseconds end-to-end. Half of that budget is in the SSD; the other half is descriptor setup and DMA on the bus. Without DMA, just copying the 4 KB by CPU would dwarf the device latency.
• GPU compute. CUDA's cudaMemcpy uses DMA between system RAM and GPU HBM. Pinned (page-locked) memory plus DMA hits PCIe Gen5 line rate (~63 GB/s); pageable memory falls back to bounce buffers and runs at half that.
• Kernel bypass. DPDK and SPDK pin huge pages and let userspace post DMA descriptors directly to the device. Eliminates kernel involvement on every packet/IO; DPDK can route 14.88 Mpps per core.
• Audio/video. Sound cards DMA-stream audio buffers to the codec; the CPU only fills new buffer halves. The same model drives every video display via the GPU's display DMA engine ("scanout").
• Disk encryption / checksumming. Modern controllers do AES, CRC, and parity in DMA path. The CPU sees only the plaintext data after decryption, with no extra copy.

The DMA lifecycle

A single I/O operation through DMA looks like:

1. Allocate. Kernel obtains pages and pins them so the MMU cannot reclaim them mid-DMA. dma_alloc_coherent() or get_user_pages() for user buffers.
2. Map. Kernel calls dma_map_single() / dma_map_sg() to obtain a bus address (an IOVA if IOMMU is on, else a physical address). The IOMMU page table is updated.
3. Build descriptor. Kernel writes a (bus address, length, flags, completion-queue index) entry into the device's command ring.
4. Doorbell. Kernel writes to a memory-mapped register to tell the device "new commands available." This is one MMIO write — the only synchronous CPU cost in the data path.
5. Device DMAs. The device's bus master reads or writes RAM autonomously, multiple bus transactions, possibly out of order with respect to other commands.
6. Completion. Device writes a completion entry to the host's completion ring (also via DMA), then raises an MSI-X interrupt or marks the entry for polling.
7. Unmap. Kernel calls dma_unmap_*() to flush caches if needed, tear down the IOMMU mapping, and release the pin.

Cache coherency and DMA

On x86 and modern ARMv8, DMA is cache-coherent: the device's reads see CPU writes that are still in cache, and the device's writes invalidate stale CPU cache lines. The cost is paid by the cache-coherency interconnect (Intel UPI, AMD Infinity Fabric, ARM CCN). On older or embedded systems without coherency, the kernel must explicitly flush or invalidate caches around each DMA — that's what dma_sync_single_for_device() and dma_sync_single_for_cpu() are for. Forgetting these calls produces silent corruption that only shows up at high stress.

Example throughput numbers

• NVMe Gen4 (line rate). Single drive: 7 GB/s sequential, 1.5 M IOPS random 4K. Eight drives in RAID0 saturate PCIe Gen4 x16 (~31 GB/s).
• 100 Gbps NIC. 12.5 GB/s sustained, ~150 M packets/sec at 64-byte packets. Without DMA, impossible on any CPU shipping in 2026.
• GPU PCIe transfer. Pinned memory + DMA: 24-25 GB/s on PCIe Gen4 x16; 50+ GB/s on Gen5. Pageable memory drops to ~10 GB/s.
• RDMA verbs. Userspace posts a Work Request describing remote and local virtual addresses; the NIC does both ends of the transfer. Latency ~1 microsecond, throughput line-rate.

Common misconceptions

• "DMA is slow because of setup." Setup is microseconds; the data transfer dominates. For large transfers (>= 1 KB) DMA is unambiguously faster than PIO. Only for sub-cache-line operations does PIO break even.
• "IOMMU adds significant overhead." Modern Intel VT-d and AMD-Vi handle TLB lookups in hardware, often free or low single-digit nanoseconds. The expensive operation is invalidation, which the kernel batches.
• "DMA buffers must be physically contiguous." Not since scatter-gather. SG descriptors handle dozens of physically-discontiguous pages in one transfer. dma_alloc_coherent() is for small structures only.
• "DMA is automatic." Every byte path through DMA was designed by a driver author — including page pinning, DMA mask validation (32-bit vs 64-bit addressing), and unmapping order. Bugs here cause kernel panics or silent corruption.
• "DMA is safe by default." Without an IOMMU, any DMA-capable device can access all RAM. This is why "Kernel DMA Protection" and IOMMU strict mode exist.
• "DMA = kernel bypass." They are different. DMA is the hardware mechanism; kernel bypass is the software architecture (DPDK, SPDK, RDMA verbs) that lets userspace drive DMA without a syscall per operation.

Inside an Ethernet RX path

• NIC receives a frame, runs Receive Side Scaling to pick a queue, reads the next descriptor from the host's RX ring (DMA read).
• NIC writes the frame into the buffer pointed to by the descriptor (DMA write), updates the descriptor's status, optionally writes hash/timestamp.
• NIC raises MSI-X to the CPU that owns this queue; or, if NAPI polling is active, just sets the status bit and the polling thread picks it up next round.
• Kernel reads the descriptor, hands the buffer (now containing the frame) up the network stack, allocates a fresh buffer, writes a new descriptor — ready for the next frame.
• At 64-byte packets and 10 Gbps line rate that's 14.88 million round-trips per second through this loop. Modern NICs amortize via interrupt coalescing and batched descriptor writes.