Rotary Position Embedding ·
RoPE
un
seel
.com · Rotate Q & K · attention sees distance
Query pos
m
Key pos
n
State
—
Query vector
Key vector
Relative angle m−n
Frequency dials
▶ Play
←
→
🔇 Unmute
Reset
Un
seel
.com · Rotary Position Embedding