I/O Multiplexing (select, poll, epoll, kqueue)

Why multiplexing exists

Naive thread-per-connection costs ~8 KB of kernel stack plus a default 8 MB of userspace stack per thread; at 10,000 connections that's 80 GB of address space and the scheduler is in tears (the C10K problem Dan Kegel posed in 1999). Multiplexing flips the model: one thread (or a small pool) registers many descriptors and asks the kernel "tell me when any are ready." Idle connections cost effectively nothing.

The five APIs

	select	poll	epoll	kqueue	io_uring
Platform	POSIX	POSIX	Linux 2.6+	BSD, macOS	Linux 5.1+
Model	Readiness	Readiness	Readiness	Readiness	Completion
Per-call cost	O(N)	O(N)	O(active)	O(active)	O(0) when busy
FD limit	FD_SETSIZE (~1024)	RLIMIT_NOFILE	None	None	None
Registration	Re-passed each call	Re-passed each call	Persistent	Persistent	Persistent ring
Edge / level	Level	Level	Both (EPOLLET)	Both (EV_CLEAR)	N/A (completion)
Beyond sockets	fds only	fds only	+ signalfd/timerfd/eventfd	Sockets, files, signals, timers, processes	Sockets, files, splice, NVMe
Notable extras	—	—	EPOLLEXCLUSIVE, EPOLLONESHOT	EVFILT_PROC, EVFILT_TIMER	SQPOLL, fixed buffers, linked SQEs

Readiness vs completion

The most common source of confusion. These APIs answer different questions:

• Readiness (select, poll, epoll, kqueue): the kernel says "fd 7 has data buffered — your next non-blocking read will not EAGAIN." You then issue the read. Two syscalls per operation.
• Completion (io_uring, Windows IOCP, POSIX AIO): you submit the read in advance; the kernel posts a completion with the bytes copied when done. No "is it ready?" round trip.

Completion scales better because you remove the syscall-per-op tax. It's harder to reason about because the buffer must remain valid until the kernel reports completion — possibly milliseconds later.

An edge-triggered epoll server

Edge-triggered is the high-performance default. The contract: every time epoll wakes you, you must drain every buffered byte for that fd, or you'll never hear about it again until more data arrives.

// epoll_server.c — minimal ET echo server (no error handling)
#include <sys/epoll.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <unistd.h>
#include <errno.h>

int main(void) {
  int lfd = socket(AF_INET, SOCK_STREAM | SOCK_NONBLOCK, 0);
  int yes = 1; setsockopt(lfd, SOL_SOCKET, SO_REUSEADDR, &yes, sizeof yes);
  struct sockaddr_in a = { .sin_family = AF_INET, .sin_port = htons(8080) };
  bind(lfd, (struct sockaddr*)&a, sizeof a);
  listen(lfd, 4096);

  int ep = epoll_create1(EPOLL_CLOEXEC);
  struct epoll_event ev = { .events = EPOLLIN | EPOLLET, .data.fd = lfd };
  epoll_ctl(ep, EPOLL_CTL_ADD, lfd, &ev);

  struct epoll_event events[256];
  for (;;) {
    int n = epoll_wait(ep, events, 256, -1);
    for (int i = 0; i < n; i++) {
      int fd = events[i].data.fd;
      if (fd == lfd) {
        // Drain ALL pending accepts — we're edge-triggered
        for (;;) {
          int c = accept4(lfd, NULL, NULL, SOCK_NONBLOCK);
          if (c == -1) { if (errno == EAGAIN) break; continue; }
          struct epoll_event e = { .events = EPOLLIN | EPOLLET | EPOLLRDHUP, .data.fd = c };
          epoll_ctl(ep, EPOLL_CTL_ADD, c, &e);
        }
      } else {
        char buf[4096];
        for (;;) {                                      // drain ALL bytes
          ssize_t r = read(fd, buf, sizeof buf);
          if (r > 0) write(fd, buf, r);
          else if (r == 0 || (r == -1 && errno != EAGAIN)) { close(fd); break; }
          else break;                                   // EAGAIN — done
        }
      }
    }
  }
}

Forget either drain loop and the server drops connections under load — the most common epoll bug.

Python's selectors.DefaultSelector() picks the best API (epoll, kqueue, falling back to poll then select):

import selectors, socket
sel = selectors.DefaultSelector()
ls = socket.socket(); ls.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
ls.bind(('', 8080)); ls.listen(4096); ls.setblocking(False)
sel.register(ls, selectors.EVENT_READ, data='listen')
while True:
    for key, _ in sel.select(timeout=None):
        if key.data == 'listen':
            c, _ = key.fileobj.accept(); c.setblocking(False)
            sel.register(c, selectors.EVENT_READ, data='conn')
        else:
            try:
                d = key.fileobj.recv(4096)
                if not d: raise ConnectionResetError
                key.fileobj.sendall(d)
            except (BlockingIOError, ConnectionResetError):
                sel.unregister(key.fileobj); key.fileobj.close()

Node has no select binding because libuv hides it: net.createServer already runs an event loop on top of epoll/kqueue/IOCP. The pattern looks blocking but is fully multiplexed:

const net = require('node:net');
net.createServer(c => c.on('data', d => c.write(d))).listen(8080);
// libuv pulls events from epoll; one thread services thousands of sockets.

Costs that decide which wins

• select, 10,000 idle fds: ~3.7 KB bitmap copied each call, kernel walks all 10,000 — several hundred µs per call.
• epoll_wait, same N idle fds: cost proportional only to ready fds. A few active among 100,000 idle costs the same as a few among 100.
• Edge-triggered drain miss: forgetting to loop on EAGAIN leaves the connection stuck — a latency tail with no CPU symptom.
• io_uring: 4 KiB NVMe reads with SQPOLL sustain 7–10 M IOPS/core vs ~2 M for epoll + read.

Triggers, herds, and the fine print

Level-triggered vs edge-triggered

• Level-triggered (epoll default, kqueue default): readiness is reported as long as the condition holds. Easy to use; you can stop reading mid-way and pick up later.
• Edge-triggered (EPOLLET, EV_CLEAR): readiness is reported only on the transition. Faster — fewer wakes — but you must drain in a non-blocking loop. Pair with EPOLLONESHOT if you want to hand the fd to a worker thread without races.

EPOLLEXCLUSIVE

The thundering-herd cure. Multiple workers register the same listener with EPOLLEXCLUSIVE; the kernel wakes one waiter per accept event instead of all. Combine with SO_REUSEPORT for kernel-side load balancing across separate epoll fds.

kqueue's filters

Where epoll watches fds, kqueue watches events. EVFILT_READ/WRITE mirror epoll; EVFILT_TIMER, EVFILT_PROC (child exit), EVFILT_SIGNAL, EVFILT_VNODE (file change) make it a Swiss-army event facility. macOS Grand Central Dispatch sits on top of it.

io_uring tricks

• SQPOLL: kernel thread polls the submission ring, eliminating syscalls on busy paths.
• Linked SQEs: chain accept → recv → send for one-submission requests.
• Registered buffers / fixed files: pre-pin pages and fds, skipping per-op lookups.

Common pitfalls

• Edge-triggered without draining. Read once and you wait forever for the next byte. Always loop until EAGAIN.
• Lost-wakeup race. Userspace observes "no data," then the peer sends, then userspace registers — but the readiness edge already happened. Register before the first read.
• Stale fd in the epoll set. Closing a duplicated fd doesn't remove it from epoll if other references survive. epoll_ctl(EPOLL_CTL_DEL) before close.
• FD_SETSIZE truncation. FD_SET with an fd ≥ 1024 silently writes past the bitmap. Switch to poll or epoll above a few hundred fds.
• Mixing blocking and non-blocking. A blocking write on a full socket buffer parks the whole event loop. Mark every multiplexed fd O_NONBLOCK.
• Treating EPOLLRDHUP as error. It's informational — peer closed write. You may still have buffered bytes worth reading. Drain first, close after.
• io_uring buffer lifetime. Submitting a read into a stack buffer popped before completion is UB. Use long-lived or registered buffers.

Choosing

For portable code with a few hundred connections, use poll or your language's selectors abstraction. On Linux servers handling thousands+, use epoll edge-triggered with EPOLLEXCLUSIVE for accept fan-out. On BSD or macOS, use kqueue. For high-throughput storage stacks where syscalls dominate, learn io_uring — and accept that you'll re-read the docs at least three times.