4.2 PaLM Series

4.2 PaLM Series

4.2.1 PaLM1

• Model Architecture: PaLM (Pathways Language Model) adopts a standard Transformer architecture, using a Decoder-only setup, with the following modifications:
• SwiGLU Activation: The Feed-Forward Network (FFN) part is modified to use the SwiGLU activation function.
• Parallel Layers: Instead of the standard sequential formulation, PaLM uses a parallelized structure. A Standard layer:
PaLM's parallelized structure:
• MQA (Multi-Query Attention), RoPE (Rotary Position Embedding), Shared Input-Output Embeddings, no biases in any of the dense layers or layer norm (to improve training stability)
• Vocabulary: Uses vocabulary trained with SentencePiece, involving a total of 256k tokens. Numbers are always split into individual digit tokens (e.g., "123.5" → "1 2 3 . 5").

• Training Dataset

• Training Settings
• Weight Initialization: Trainable weights (except for embeddings and layer norm scales) adopt fan-in variance scaling initialization, i.e., . In neural network training, fan-in variance scaling refers to initializing the weights of each layer based on the number of input connections (neurons) to that layer, known as the fan-in. Therefore, the is the input dimension of the kernel. Input embeddings are initialized as , because layer normalization is not applied to the embedding layer. Since the input and output embeddings are shared, the logits before softmax are scaled by , where is the embedding size.
• Optimizer: The model was trained with the Adafactor optimizer. This is similar to Adam with “parameter scaling,” which scales the learning rate by the root-mean-square of the parameter matrix. Because the weight initialization is proportional to , the effect of this is similar to scaling down the learning rate for Adam. However, parameter scaling has the benefit that parameter matrices at different scales have their learning rate scaled down at different rates.
• Hyperparameters: It uses the learning rate of for the first 10,000 steps, which is then decayed at a rate of , where is the step number. The momentum , second-order moment . This is found to be more stable than the standard when training large language models. It uses global gradient norm clipping with a value of 1.0. It uses a dynamic weight decay of during training, where is the current learning rate.
• Loss function: The model is trained with the standard language modeling loss function, which is the average log probability of all tokens without label smoothing. It additionally use an auxiliary loss of to encourage the softmax normalizer to be close to 0, which is found to increase the stability of training.
• Sequence length: A sequence length of 2048 was used for all models. Input examples are concatenated together and then split into sequences of exactly 2048 tokens, so that there are no padding tokens, but examples may be split in the middle. Input examples are differentiated from one another with a special [eod] token.
• Batch size: For all models, the batch size is increased during training. For the largest model, it uses batch size 512 (1M tokens) until step 50k, then double it to 1024 (2M tokens) until step 115k, and finally double it again to 2048 (4M tokens) until training is complete at step 255k. The smaller models followed similar schedules. The reason for using such a batch size schedule is two-fold: (1) smaller batch sizes are more sample efficient (i.e., better loss as a function of tokens is seen) in earlier training, while larger batch sizes are beneficial later in training due to better gradient estimates, and (2) larger batch sizes result in larger matrix multiplication dimensions, which increases TPU efficiency.
• Dropout: The model was trained without dropout, although a dropout of 0.1 is used for finetuning in most cases.

4.2.2 PaLM2

• Model Architecture: Not described in the report.

• Pretraining Data: PaLM2 adopts the principles of UL2 (UL2 is an LLM pre-training paradigm developed by Google that differs from GPT-3 and PaLM). For details about UL2, please check UL2: Unifying Language Learning Paradigms. UL2 trained on a mixture of different pretraining objectives to help the model learn various aspects of language understanding. PaLM2 benefits from this approach, showing strong multilingual capabilities.

• Scaling Law