positional encoding

_{scroll ↓ to Resources}

Note

• Adds information about the position of each token in the sequence to help the transformer understand word order, because attention is position-invariant
• see decoder architecture
• in simplest form, position encoding vectors are added to input sequences

Rotary Position Embeddings (RoPE)

• RoFormer: Enhanced Transformer with Rotary Position Embedding Explained - YouTube
• RoPE (Rotary positional embeddings) explained: The positional workhorse of modern LLMs - YouTube
• Original Paper : https://arxiv.org/pdf/2104.09864.pdf
• https://blog.eleuther.ai/rotary-embeddings/
• https://nn.labml.ai/transformers/rope/index.htmly.
• https://serp.ai/rotary-position-embedding/
• https://medium.com/@andrew_johnson_4/understanding-rotary-position-embedding
• https://github.com/lucidrains/rotary-embedding-torch
• http://krasserm.github.io/2022/12/13/rotary-position-embedding/

Resources

---

Links to this File

table file.inlinks, filter(file.outlinks, (x) => !contains(string(x), ".jpg") AND !contains(string(x), ".pdf") AND !contains(string(x), ".png")) as "Outlinks" from [[]] and !outgoing([[]])  AND -"Changelog"