4.1 BERT & Variants

4.1 BERT & Variants

4.1.1 BERT

1. Position Embedding: Encodes the position of each token in the sequence. This is crucial because BERT does not have inherent order information. It learns position embeddings directly during training, assigning each position a randomly initialized embedding vector.

2. Token Embedding: Represents the meaning of each token. For example, the word "playing" might be split into "play" and "##ing", and each sub-token gets its own embedding.

3. Segment Embedding: Distinguish two different sentences (e.g., sentence A and sentence B in a pair for tasks like question answering or dialogue). Tokens from the first sentence are assigned a segment ID of 0, and tokens from the second sentence are assigned a segment ID of 1.

• 80% of these tokens are replaced with the [MASK] token: This is the core of the Masked Language Model (MLM). It allows the model to learn bidirectional contextual representations without labels leaking.

• 10% of these tokens are replaced with a random token: This helps the model learn to handle noise. Since it doesn't know whether the token at a given position is correct, it must rely more heavily on the surrounding context (which also enhances the model's ability to disambiguate word meanings—a key feature for handling polysemy).

• 10% of these tokens are left unchanged but still predicted: This introduces a form of bias or “surprise reward,” pushing the model to learn true contextual representations even when the token appears to be correct.

1. During training, 50% of the sentence pairs are actual consecutive sentences from the original text, while the other 50% are randomly paired from the corpus and do not logically follow each other.

2. The first sentence is prepended with the [CLS] token as an indicator that the feature is used for classification tasks. For non-classification tasks, this token can be removed. A [SEP] token is inserted at the end of each sentence, acting as a separator to distinguish the two input sentences.

1. Large model size and high resource demand: BERT is very large with a huge number of parameters, requiring extensive data and computational power. It is more suitable for tasks involving language understanding and embedding, but not ideal for generative tasks.

2. Mismatch between pretraining and fine-tuning: During fine-tuning, the [MASK] token does not appear (it only exists during pretraining). This causes a gap between pretraining and downstream tasks. When fine-tuning doesn’t include [MASK], the model lacks a clear signal or focus, making it harder to adapt. That’s why we replace only 80% of selected tokens with [MASK] during pretraining, but this is still just a workaround, not a real solution.

3. Low training efficiency: Compared to traditional language models, only 15% of the tokens are predicted in each training batch, which leads to slower model convergence and less efficient use of training data.

4.1.2 BART

• Like GPT, it replaces the ReLU activation function with GeLU and initializes parameters with a normal distribution N(0, 0.02)

• The BART base model has 6 layers each for the encoder and decoder, while the large model increases this to 12 layers

• Each layer of BART's decoder receives the encoder's final hidden state as an additional input for cross-attention

• BERT uses an extra Feed Forward Layer before word prediction, but BART does not

4.1.3 T5

4.1.4 RoBERTa

• Larger model parameters: The model was trained using 1024 V100 GPUs for one full day.

• Larger batch size: RoBERTa was trained with significantly larger batch sizes, ranging from 256 to 8000 during experimentation.

• More training data: It was trained on 160GB of raw text, including sources like CC-NEWS. In contrast, the original BERT was trained on just 16GB of BookCorpus and English Wikipedia.

• Removed Next Sentence Prediction (NSP): RoBERTa eliminates the NSP task used in the original BERT.

• Dynamic Masking: In BERT, tokens are masked randomly once during data preprocessing, resulting in a static masking pattern. RoBERTa, on the other hand, uses dynamic masking — a new masking pattern is generated every time a sequence is fed into the model. This allows the model to gradually adapt to different masking strategies and learn a broader range of linguistic patterns.

• Text Encoding: RoBERTa uses Byte-level Byte-Pair Encoding (BBPE). The original BERT uses WordPiece for tokenization, resulting in a vocabulary size of 30K. BART and RoBERTa, however, opt for byte-level BPE and train directly on raw input without additional preprocessing or tokenization, leading to a vocabulary size of 50K.

4.1.5 DeBERTa

• Disentangled Attention:
• In BERT, each token in the input layer is represented by a single vector — the sum of its token embedding and position embedding (also segment embedding as needed).
• In DeBERTa, each token is represented using two separate vectors that encode its content and relative position, respectively. The attention weights among tokens are computed using disentangled matrices based on their contents and relative positions.
• Attention weights between tokens depend not only on their content but also on their relative positions. For example, the dependency between “deep” and “learning” is much stronger when they appear next to each other than when they appear in different sentences.

• Enhanced Mask Decoder:
• Like BERT, DeBERTa is pre-trained using Masked Language Modeling (MLM). In this setup, the model is trained to predict the masked word based on the words surrounding the mask token. DeBERTa uses both content and position information from the surrounding context for MLM. While its disentangled attention mechanism already considers the content and relative positions of context words, it does not account for their absolute positions, which can be critical for accurate predictions in many scenarios.
• For example, consider the sentence: “a new store opened beside the new mall”. If the words “store” and “mall” are masked for prediction, relying only on the local context (i.e. relative positions and nearby words) is not sufficient for the model to distinguish between them — both share similar surrounding words and relative positions. To resolve this, the model needs to consider the absolute positions of the words as complementary information. In this sentence, for instance, the subject is “store”, not “mall”, and such syntactic differences often depend heavily on a word's absolute position in the sentence.
• There are two ways to incorporate absolute position information. (1) We know that BERT adds absolute position embeddings at the input layer. However, (2) DeBERTa incorporates absolute position embeddings right after the last transformer layer but before the softmax layer for masked token prediction. This allows DeBERTa to use the absolute positions as additional information alongside content and relative position information during the decoding process.

• Virtual Adversarial Training:
• Virtual adversarial training is a regularization method that enhances model generalization. It improves robustness by introducing small perturbations to the input, creating adversarial examples. The goal is to ensure the model produces similar output distributions for both the original and adversarial versions of the same input.
• In previous NLP tasks, adversarial perturbations are typically applied to word embeddings, not raw word sequences. However, embedding vectors can vary significantly in scale across different words and models. This variance grows with larger models—those with billions of parameters—leading to instability during adversarial training.
• The Scale-invariant-Fine-Tuning (SiFT) algorithm improves training stability by applying perturbations to normalized word embeddings. When fine-tuning DeBERTa on downstream NLP tasks, SiFT first normalizes each word embedding into a unit-length vector, then applies adversarial perturbations.