GELU ·
x · Φ(x)
un
seel
.com · The Gaussian-gated nonlinearity of BERT & GPT
Input x
—
Gate Φ(x)
—
GELU
—
State
—
GELU curve
ReLU (hard gate)
Gaussian gate Φ
Min ≈ −0.17
▶ Play
←
→
🔇 Unmute
Reset
Un
seel
.com · GELU Activation
Un
seel
.com · GELU Activation: The Gaussian-Gated Nonlinearity Behind Modern Transformers