提高生成数据集的多样性

在上一章中，我们讨论了用大语言模型生成合成数据，进一步微调本地检索模型的方法。该方法依赖大量未标注文档：为每篇文档生成一个或多个查询，再组成查询—文档对。

如果任务并不是信息检索呢？例如，你需要进行法律文档分类，但不允许把任何数据发送到外部 API，因此必须训练本地模型。这时，数据收集可能成为主要障碍，拖慢产品开发。

为方便理解，先假设目标是生成儿童故事。Eldan 等人（2023）的研究便从这个任务出发。每个故事有 2 至 3 段，情节和主题简单，而整个数据集要覆盖儿童的词汇和事实知识。

语言不只是规则与符号系统，还传达并解释含义。使用大语言模型生成训练数据时，主要难点是保证多样性。即便把生成温度设得很高，模型仍可能反复生成相似内容，连儿童语言需要的多样性都达不到。连贯性和相关性也是自然语言生成面临的问题。

为解决这一问题，研究者准备了约 1,500 个基础单词，模拟儿童的常用词汇，并分为名词、动词和形容词。每次生成时随机各选一个，再要求模型将它们融入故事。

这一方法显著增加了数据多样性，使故事覆盖全部词汇以及不同概念组合。研究者还加入对话、情节转折、悲剧结尾和道德寓意等故事特征，每次随机选取一部分，要求模型在故事中体现。

提示词模板：

Write a short story (3-5 paragraphs) which only uses very simple words that a 3 year old child would likely understand. The story should use the verb ”{random.choice(verbs_list)}”, the noun ”{random.choice(nouns_list)}” and the adjective ”{random.choice(adjectives_list)}”. The story should have the following features: {random.choice(features_list)}, {random.choice(features_list)}. Remember to only use simple words!

假设按下面的方式填入内容：

提示词：

Write a short story (3-5 paragraphs) which only uses very simple words that a 3 year old child would likely understand. The story should use the verb ”decorate”, the noun ”thunder” and the adjective ”ancient”. The story should have the following features: the story should contain at least one dialogue, the story has a bad ending. Remember to only use simple words!

输出：

Once upon a time, in an ancient house, there lived a girl named Lily. She loved to decorate her room with pretty things. One day, she found a big box in the attic. She opened it and saw many shiny decorations. Lily was very happy and decided to use them in her room. As Lily was decorating her room, the sky outside became dark. There was a loud thunder sound, and Lily got scared. She ran to her mom and said, ”Mommy, the thunder is so loud!” Her mom hugged her and said, ”Don’t worry, it will pass soon.” But the thunder did not stop. It got louder and louder, and the ancient house started to shake. Suddenly, the roof fell down on the pretty decorations. Lily was sad because her room was not pretty anymore. The end.

这里利用 GPT-3.5 和 GPT-4 等能够遵循指令、生成大量内容的模型。每次随机组合提示词，而模型按照要求生成，所以故事会有很大变化。即便有一两个词相同，叙事也可能完全不同。本质上，这是把随机性注入提示词，以生成多样化的数据集。

基本步骤如下：

1. 找出合成数据中不同样本之间可能变化的参数或实体。
2. 自动生成或手工整理这些实体的候选集合。
3. 随机选取实体填入提示词，生成数据。生成温度最好高于默认值，但低于最大值。
4. 使用 ChatGPT 或 GPT-4 的生成结果训练本地模型。

用于控制生成的实体也可以是类别标签。例如，在情感分类任务中，可以直接在提示词里指定“正面”或“负面”，生成相应情感的文本，再用这些数据训练本地分类器。

迭代式、分层式合成数据生成

还可以使用更多实体，让生成过程更复杂，其中一部分实体由模型提前生成。例如，先让模型写故事摘要，并给出一句必须出现在故事中的话；随后把这些中间结果加入最终请求。

提示词模板：

Summary: {a short summary generated by LLM, using the approach above}
Features: {copy the features from the initial prompt}
Sentence: {a sentence generated by LLM, which should be present in the story}
Words: {copy the words from the initial prompt}
Story:

填好内容后的示例：

提示词：

Summary: Lily and Timmy build a sandcastle together and learn to compromise, but it gets knocked over by a gust of wind. They find beauty in the broken sandcastle and play happily with a butterfly.
Features: Dialogue, Foreshadowing, Twist
Sentence: One day, she went to the park and saw a beautiful butterfly.
Words: disagree, network, beautiful
Story:

输出：

Once upon a time, there was a little girl named Lily. She loved to play with her toys and her friends. One day, she went to the park and saw a beautiful butterfly. She tried to catch it, but it flew away. Lily’s friend, Timmy, came to play with her. They decided to build a sandcastle, but they disagreed on how to build it. Lily wanted to make it tall, while Timmy wanted to make it wide. They argued for a while, but finally agreed to make it both tall and wide. As they were building the sandcastle, they noticed a group of ants working together. They realized that the ants were building a network of tunnels. Suddenly, a gust of wind came and knocked over their sandcastle. They were sad, but then they saw the butterfly again. It landed on the remains of their sandcastle and they realized that it was still beautiful, even in its broken state. They smiled and played together happily.

这样就能生成数十万个多样化示例来训练模型。假设要训练一个分类器，判断文本是否包含对话或情节转折，由于最初的提示词已经指定相关标签，因此每个生成样本的目标值也是已知的。

《教科书就是你所需要的》

一个重要问题是：合成数据是否真的有助于训练用于现实应用的网络？研究者对此开展了实验，验证使用先进大语言模型生成的合成数据训练较小语言模型的有效性。

Gunasekar 等人（2023）强调高质量训练数据的重要性。他们认为，如果训练材料具备优秀教科书的特点——清晰、全面、信息丰富且尽量无偏——语言模型会更有效。

这些原则被用于构建半合成数据集，训练 Phi-1。主要评估任务是根据文字描述或文档字符串生成 Python 函数，使用 HumanEval 基准（Chen 等人，2021）评估质量。

研究者强调多样性，原因包括：让模型接触不同的代码表达和解题方法；降低过拟合或依赖特定模式的风险；提高处理陌生任务或新型任务的能力。

为解决代码生成问题，他们创建了类似教科书的文档，重点涵盖推理与基础算法能力，并通过限制主题和目标读者来增加多样性。

遗憾的是，论文没有公开生成这些数据的具体提示词模板，但展示了输出结果。研究使用 ChatGPT（GPT-3.5），而非 GPT-4，即便如此，合成数据微调仍取得了很好的效果。

图片来源：Gunasekar 等人（2023）。

“目标读者”这一限制尤其值得注意。指定读者群体是增加多样性的好方法。高中生、计算机本科生和博士生的解释方式及信息处理方式明显不同，会让生成的数据更加多样。下面尝试重建这种提示词。

提示词：

Write an extract from a Computer Science textbook for a 1st-year bachelor. The coding language is Python 3.6.

This is an extract from the middle of the following topic: Singular matrices.

The extract starts with a high-level overview of the topic. Then, it presents an example and describes the solution in natural language. After that, it provides 1-2 code snippets, following the example. Each snippet has no more than 10 rows. There should be no text after code snippets.

Keep in mind that the extract should be concise and explain only one concept at a time.  The code should represent a Python function & its call. All the common ML/DS libraries are available.

Don't be too verbose. The expected length of the extract is 2 paragraphs at most.

输出：

**Singular Matrices**

In linear algebra, a matrix is termed as "singular" if it doesn't have an inverse. Another equivalent definition is when the determinant of the matrix is zero. Singular matrices are crucial because they represent systems of linear equations that do not have a unique solution.

Consider a 2x2 matrix `[[a, b], [c, d]]`. If the product of `ad` and `bc` is the same (i.e., `ad = bc`), then this matrix is singular. Let's explore this with an example: given the matrix `[[2, 4], [1, 2]]`, you'll find that the product of the diagonals yields the same result (2 * 2 = 4 * 1), thus it's a singular matrix.

import numpy as np

def is_singular(matrix):
    determinant = np.linalg.det(matrix)
    return abs(determinant) < 1e-9

matrix_example = np.array([[2, 4], [1, 2]])
print(is_singular(matrix_example))  # returns True

结果已经相当接近！

研究者共生成了 10 亿词元扩充训练集，使一个约 15 亿参数的小模型能够与规模大十倍的模型竞争。具体结果见论文。

图片来源：Gunasekar 等人（2023）。

你的任务可能不需要这么多数据，因为论文研究的是资源需求较高的预训练。按原文当时每千词元 0.002 美元的 ChatGPT 单价估算，生成这些词元约需 2,000 美元，输入提示词大约还需同等费用。这里只保留历史估算，不代表当前价格。

领域越专业，合成数据微调通常越有价值，使用非英语语言时尤其如此。该方法还能结合思维链（CoT），改善本地模型的推理能力，也可与其他提示方法结合。

Alpaca（Taori 等人，2023）和 Vicuna（Zheng 等人，2023）等开源模型，也展示了利用合成数据微调的效果。