5.1 Prompt Engineering

5.1 Prompt Engineering

5.1.1 In-Context Learning

ICL Concept Overview

1. Core Concepts
• Context Dependency: The essence of In-Context Learning (ICL) lies in leveraging the model’s ability to understand and reason within a given context to complete tasks. The model uses contextual information (such as examples and task descriptions) provided in the input to perform reasoning, without relying on explicit training using those examples.
• No Parameter Updates: ICL does not involve updating the model's actual parameters. Instead, it adjusts output behavior based on the provided contextual information.
• Dynamic Adaptation: During inference, the model dynamically adapts to the given context. It analyzes examples or prompts embedded in the input and generates task-appropriate responses. This adaptive behavior stems from the general knowledge learned during pretraining.

2. Operation Mechanism
• Prompts and Examples: ICL often uses prompts to guide the model's output generation. These prompts usually include task descriptions or questions that instruct the model on what to do. In few-shot learning, prompts may contain example input-output pairs, helping the model understand how to handle similar tasks.
• Contextual Task Descriptions: In ICL, task descriptions are used to specify the goal. For example: “Translate the following sentence into French.” You can provide a few input-output pairs such as: “Hello” → “Bonjour”; “Thank you” → “Merci”. These examples help the model understand the expected translation behavior for new inputs.
• Reasoning and Output Generation: Based on the contextual input, the model performs reasoning and generates outputs that align with the context. In ICL, these outputs are shaped by the model’s understanding of the examples, task descriptions, and its pretraining knowledge.

3. Formal Definition of ICL
Where:
• : Instruction or task description
• : Input-output example pairs
• : New input to be predicted
• : The model’s predicted answer

4. Example

ICL Demonstration Design

• Demonstration Selection
1. Heuristic Methods
• Based on Semantic Similarity: Use models like BERT as encoders. Select the CLS embeddings (the first token in a sequence) of candidate examples and the test input, then calculate similarity (e.g., KNN or Euclidean distance). Choose the top 10 most similar examples as demonstrations.

• Based on Diversity: To avoid overly similar demonstrations and test data, select demonstrations with diverse distributions. Use diversity-aware selection methods (e.g., KNN + clustering) to ensure broader coverage. When two demonstrations are highly similar, one can be removed.
Reference: Diverse Demonstrations Improve In-context Compositional Generalization



2. LLM-based Methods:
• Based on Prompt Retriever: This method helps the model find the most helpful examples to include in a prompt. How it works:



1. Start with a test question (e.g., “What is the length of the longest river in the USA?”).
2. Search the training set to find similar questions and answers. At this stage, we use an unsupervised retriever (like BM25 or SBERT) that looks for examples that are semantically close. These are called candidate prompts.
3. Score each candidate: We give each candidate to a language model to see how well it helps answer the test question. The model gives each one a score from 0 to 1.
4. Keep the best and worst ones: Keep the top-k highest-scoring examples as positive demonstrations (they’re helpful). Keep the bottom-k as negative examples (they’re not helpful).
5. Use these examples to train a better retriever: Now we train a new model that learns to match future questions with the most helpful demonstrations. This model uses:
• One encoder to understand the user input question (called Utterance Encoder),
• Another encoder to understand example prompts (called Prompt Encoder).
6. In the future, when you give it a new question, it can quickly find the best examples to use in the prompt.

• Based on Active Learning / Reinforcement Learning: This method treats the process of picking demonstrations like a learning problem:

“Which examples from the dataset should we use to help the model do better on a test question?”

The goal is to select the most helpful examples to build a prompt so that the model makes more accurate predictions. But here's the challenge: There are way too many possible combinations of examples to try them all. So instead of brute force, we use a smarter strategy—Reinforcement Learning (RL). How it works:
1. Think of demo selection as a Markov Decision Process (MDP). We choose one example at a time, building a demonstration list step by step.
2. At each step:
• The current state = the demonstrations we’ve picked so far
• The action = picking a new example from the dataset (or choosing to stop)
• The reward = how accurate the model is on a validation set when using current state + action
3. We can choose an RL algorithm to learn which sequences of examples give the best reward (i.e., help the model answer correctly).
4. In the end, the trained RL agent learns to build strong prompts by choosing examples that actually help the model perform better.
• Based on Self-Consistency: This approach uses the LLM’s own abilities to generate helpful demonstrations, instead of relying on an external dataset. The goal is to reduce dependence on hand-collected or labeled data. The process has two main steps:
1. Step 1: Design a good prompt and ask the LLM to generate k examples that fit the task.
2. Step 2: Take the original input and add the examples generated in Step 1. Then, let the LLM use that full prompt to make the final prediction.

• Demonstration Format
After selecting demonstrations, the next step is to assemble them into natural language prompts.
Reference: Cross-task generalization via natural language crowdsourcing instructions
This paper introduced a cross-task instruction dataset where prompts follow a standardized format. Each prompt includes:
• title: A high-level task description and related skills (e.g., question generation, answer generation).
• prompt: A short textual instruction, usually followed by input examples.
• definition: Supplemental definitions explaining the prompt in more detail.
• things to avoid: Content or rules the model should avoid.
• emphasis and caution: Warnings content.
• positive examples: Examples that illustrate desired input-output behavior.
• negative examples: Examples that show undesired input-output behavior.
• reason: Why an example is considered positive or negative.
• suggestion: Guidance on how to improve or rewrite examples.

After fine-tuning on this instruction dataset, models can generalize better to unseen samples. During test-time inference, simply include the selected demonstrations as part of the task instance.

• Demonstration Ordering
When integrating demonstrations into a prompt, the order in which they appear matters. Ordering strategies can be:
• Random
• Distance-based (e.g., order examples by similarity to the test input)
• Custom-defined (based on specific metrics like entropy or task logic)

Challenges of In-Context Learning (ICL)

• Long Context Limitation: Large language models have limits on how much context they can handle. Very long inputs may exceed the model’s capacity and affect performance.

• Choosing the Right Context: Deciding what information to include in the context and how to organize it is a key challenge. Not all information is equally helpful.

• Output Consistency: The model might produce different results when the context changes slightly. Ensuring that the model’s outputs remain consistent and accurate is important.

5.1.2 Chain-of-Thought (CoT)

Concept Definition

• Arithmetic reasoning

• Commonsense reasoning

• Symbolic reasoning

• Instruction: Describes the task and defines the expected output format.

• Rationale (the reasoning steps): Includes the model’s intermediate thinking process, step-by-step solution, and possibly some related background knowledge.

• Exemplars: Shows the model how inputs and outputs should look — each example includes a question, reasoning steps, and the answer.

• Zero-Shot-CoT. No examples are provided. You just add a line like "Let's think step by step" to the instruction, which encourages the model to reason through the problem on its own. This method acts like a small pipeline: The prompt encourages the model to generate thinking process first (called the rationale), then give the final answer. Technically, Zero-Shot-CoT:
1. Generate a reasoning chain based on a basic prompt like "Let's think step by step"
2. Combine the reasoning chain with the original question to form a more complete prompt.
3. Finally, concatenate the combined prompt with a simple prompt “The answer is” to produce the final answer.
Example of Zero-Shot-Cot:



Example of Zero-Shot-Cot with Reasoning Extraction and Answer Extraction:



“Let’s think step by step” has been tested and shown to be effective. While other prompts don’t work as well as this one.

• Few-Shot-CoT. This version includes example questions with detailed reasoning steps, so the model can mimic that reasoning process.

Chain-of-Thought (CoT) Variants & Improvements

• CoT-SC: CoT Self-Consistency
Paper: Self-Consistency Improves Chain of Thought Reasoning in Language Models
This method improves performance by generating multiple reasoning paths, then using majority voting to decide the final answer. Instead of just one reasoning chain, the model creates several and picks the most common result as the final answer. This has been shown to improve accuracy significantly.
For example: On the left side of the figure, a prompt with few-shot examples is used to independently generate multiple chains of thought. Each chain gives an answer, then we marginalize over the reasoning chains to get the final answer, which is essentially majority voting on all answers.

• ToT: Tree-of-Thoughts
Paper: Tree of Thoughts: Deliberate Problem Solving with Large Language Models
Tree-of-Thoughts models the reasoning process like a decision tree instead of a single linear path. This allows the model to explore multiple reasoning paths and backtrack if it encounters a bad one. However, the strict tree structure can limit the model’s flexibility and require more complex prompt designs. Think of it like letting the model “branch out” to test multiple ideas — but with rules that might slow it down.



📌
ToT-BFS (Breadth-First Search)
Perform breadth-first search (BFS) on a tree of thoughts. At each step, choose the top-valued thought paths for further expansion.
Input:
• x: Initial problem
• p₀: Language model (LM)
• G(): Thought generator, produces the next thought (substate) from a given state
• k: Number of thoughts to generate per state
• V(): Evaluation function to assess the quality of a thought/state (inputs are usually the LM itself)
• T: Number of search steps
• b: Number of best states to retain at each step (breadth limit)
Procedure:
1. Initialize the state set S₀ to include the input x.
2. For each round t = 1, ..., T:
• For each state s in the previous round:
• Use G(p₀, s, k) to generate k new thoughts z, and construct new states s′ = [s, z].
• Use V(p₀, s′) to evaluate each new state.
• From all new states, select the top b states based on scores to form the new state set Sₜ.
3. Return the highest-scoring state from the final state set as the final answer.

• GoT: Graph-of-Thoughts
Paper: Graph of Thoughts: Solving Elaborate Problems with Large Language Models
Graph-of-Thoughts treats the reasoning process as a graph rather than a tree or chain. Each node represents a thought step, and edges define dependencies between them.
• Using GoT, multiple input branches can be built and connected flexibly.
• This enables much more complex and collaborative reasoning, combining ideas from CoT and ToT.
• Compared to ToT, GoT adds aggregation operation and enables divide-and-conquer.




GoT uses directed acyclic graphs (DAGs). The overall structure is shown below.



Inside GoT: Modular Components. GoT is built from interconnected modules (shown in blue in the diagram). These include:
• Prompter: Prepares input messages for the LLM.
• Parser: Extracts key information from the model’s response.
• Scoring & Validation: Evaluates the quality of the LLM’s outputs.
• Controller: Coordinates and manages the reasoning process.
The controller also includes two key tools:
• GoO (Graph of Operations): A static structure that defines how the model should move through reasoning steps — like a blueprint of the task.
• GRS (Graph Reasoning State): A dynamic structure that keeps track of the model’s current state — where it is in the reasoning process and how it got there (state history).