Positional Encoding in Transformers

Positional encoding is a technique that adds information about the position of each token in a sequence to its input embedding. This helps transformers understand the order of tokens and capture the structure of a sequence. Without positional information, the self-attention mechanism alone does not inherently know the order of the tokens.

Unlike traditional sequential models, transformers process tokens in parallel. Positional encoding provides information about token order that is needed to model sequential relationships.

Working

The most common method for calculating positional encodings is based on sinusoidal functions. The intuition behind using sine and cosine functions is that they provide a smooth, periodic encoding of positions that allows for easy interpolation and generalization across sequences of varying lengths.

Sinusoidal Formula

For each position (pos) in the sequence and each dimension i in the positional encoding vector, the following formula is used:

1. Even-indexed dimensions: PE_{(pos, 2i)} = \sin\left(\frac{pos}{10000^{\frac{2i}{d_{\text{model}}}}}\right)
2. Odd-indexed dimensions: PE_{(pos, 2i+1)} = \cos\left(\frac{pos}{10000^{\frac{2i}{d_{\text{model}}}}}\right)

Where:

• PE(pos, 2i): The positional encoding at position pos for the 2i-th dimension.
• pos: The position of the token in the sequence, usually starting from 0.
• i: The index used to determine the pair of dimensions 2i and 2i+1.
• d_model: The dimensionality of the model, such as 512 or 1024.
• 10000^(2i/d_model): Controls how the frequency of the sine and cosine functions changes across dimensions.

These formulas use sine and cosine functions to create wave-like patterns that changes across the sequence positions. Using sine for even dimensions and cosine for odd dimensions provides different positional patterns across the embedding dimensions.

Calculating Positional Encoding

We will calculate the positional encodings for positions 0 to 5 in a sequence. For simplicity, assume that we are working with a 4-dimensional embedding.

1. Positional Encoding for Token at Position 0: For pos = 0 (the first token), the positional encoding values will be:

\text{PE}(0) = \left[\sin\left(\frac{0}{10000^{0/4}}\right), \cos\left(\frac{0}{10000^{0/4}}\right), \sin\left(\frac{0}{10000^{2/4}}\right), \cos\left(\frac{0}{10000^{1/4}}\right)\right]

Here, the first pair of values is generated for dimensions 0 and 1, while the second pair is generated for dimensions 2 and 3.

2. Positional Encoding for Token at Position 1: Similarly, for pos = 1 (the second token), we calculate the positional encoding values:

\text{PE}(1) = \left[\sin\left(\frac{1}{10000^{0/4}}\right), \cos\left(\frac{1}{10000^{0/4}}\right), \sin\left(\frac{1}{10000^{2/4}}\right), \cos\left(\frac{1}{10000^{2/4}}\right)\right]

These values provide positional information for the second token in the sequence. The same process is followed for the remaining positions.

3. Positional Encoding for Token at Position 5: For pos = 5 (the sixth token), the positional encoding is calculated in the same way:

\text{PE}(5) = \left[\sin\left(\frac{5}{10000^{0/4}}\right), \cos\left(\frac{5}{10000^{0/4}}\right), \sin\left(\frac{5}{10000^{2/4}}\right), \cos\left(\frac{5}{10000^{2/4}}\right)\right]

Once these positional encodings are calculated for each token, they are added element-wise to the corresponding token embeddings.This gives the model both semantic information from the token embeddings and positional information from the positional encodings.

Example

Suppose we have a Transformer model which translates English sentences into French.

"The cat sat on the mat."

Before the sentence is fed into the Transformer model it gets tokenized where each word is converted into a token. Let's assume the tokens for this sentence are:

["The", "cat" , "sat", "on", "the" ,"mat"]

After that each token is mapped to a high-dimensional vector representation through an embedding layer. These embeddings encode semantic information about the words in the sentence. However they lack information about the order of the words.

Embeddings = { E₁, E₂, E₃, E₄, E₅, E₆ }

Where each Ei​ is a 4-dimensional vector. This is where positional encoding plays an important role. The positional encodings are added to the corresponding token embeddings so that the resulting representations contain information about both the token and its position in the sequence.

Implementation in Transformers

Here we will be using Numpy and Tensorflow for the implementations.

• angle_rads: Calculates the angles for each position and model dimension.
• position = 50, d_model = 512: Sets the sequence length to 50 positions and the model dimensionality to 512.

import numpy as np
import tensorflow as tf

def positional_encoding(position, d_model):

    angle_rads = np.arange(position)[:, np.newaxis] / np.power(
        10000,
        (2 * (np.arange(d_model)[np.newaxis, :] // 2)) / np.float32(d_model)
    )

    angle_rads[:, 0::2] = np.sin(angle_rads[:, 0::2])
    angle_rads[:, 1::2] = np.cos(angle_rads[:, 1::2])

    pos_encoding = angle_rads[np.newaxis, ...]

    return tf.cast(pos_encoding, dtype=tf.float32)

position = 50
d_model = 512

pos_encoding = positional_encoding(position, d_model)

print("Positional Encodings Shape:", pos_encoding.shape)
print("Positional Encodings Example:\n", pos_encoding)

Output:

The generated positional encoding has a shape of (1, 50, 512), where 50 represents the number of positions and 512 represents the model dimensionality. Each position has a 512-dimensional positional encoding.

Applications

1. Machine Translation: Understanding the order of words is important when translating one language to another. For example, the sentence “She loves reading books” needs to be translated while preserving the appropriate word order.
2. Text Generation: In language models, positional information helps the model maintain the order of tokens when generating text.
3. Time Series Forecasting: For sequential data such as stock prices or weather observations, positional information helps transformers model patterns over time.
4. Speech Recognition: Positional information helps models capture the order of speech units such as phonemes or tokens.
5. Computer Vision (Vision Transformers): Positional information can be used to represent the spatial locations of image patches.

Limitations

1. Fixed Encoding Pattern: Sinusoidal positional encoding uses a predefined mathematical pattern rather than learning positional representations from the training data. Other approaches, such as learned or relative positional encodings, use different ways to represent position.
2. Limited Flexibility for Some Tasks: Absolute positional information may not always represent relative relationships between tokens as effectively as approaches specifically designed to capture relative positions.