Skip to main content
Synthetic Data Kit is a tool from Meta LLAMA that helps you generate high-quality synthetic datasets for fine-tuning large language models (LLMs). It simplifies the process of preparing data for fine-tuning by providing a command-line interface (CLI) with a modular four-command flow. One of the key features of the synthetic-data-kit is its use of the Lance format for storing and ingesting datasets. This allows for efficient storage and retrieval of data, which is crucial when working with large datasets.

​
Key Features:

  • Data Ingestion: The toolkit can ingest various file formats, including PDF, HTML, YouTube transcripts, DOCX, PPT, and TXT.
  • Fine-tuning Format Creation: It can create different fine-tuning formats, such as question-answer (QA) pairs, QA pairs with Chain-of-Thought (CoT), and summarization formats.
  • Data Curation: The tool uses Llama as a judge to curate high-quality examples, ensuring the quality of the generated dataset.
  • Flexible Saving Options: You can save the generated datasets in various formats compatible with your fine-tuning workflow, including HuggingFace, JSONL, and JSON.

​
How it Works:

The synthetic-data-kit follows a simple four-step process:
  1. Ingest: Import your input files into the toolkit. The data is stored in the Lance format for efficient processing.
  2. Create: Generate diverse fine-tuning datasets, such as reasoning, summarization, and QA pairs, from the ingested documents.
  3. Curate: Use Llama to filter and select high-quality examples from the generated dataset.
  4. Save-as: Export the curated dataset in your preferred format.

​
Usage

The synthetic-data-kit uses Lance format to store and manage the data that you ingest. The workflow is a series of commands that build on each other, starting with the ingest command. Here is an example of the end-to-end workflow:
  1. Ingest Data into a LanceDB dataset This command reads a source document and creates a Lance dataset from it.
  2. Create fine-tuning data This command uses the Lance dataset created in the previous step to generate multimodal question-answer pairs.
  3. Curate the data This step uses a language model to curate the generated data and ensure its quality.
  4. Save the final dataset Finally, save the curated data in the format you need: jsonl, alpaca, ft or chatml.
This workflow allows you to go from a collection of documents to a high-quality, fine-tuning dataset with just a few commands. The use of LanceDB in the background makes the process efficient and scalable.

​
Getting Started:

To get started with the synthetic-data-kit, you can clone the GitHub Repository and install the necessary dependencies.
Note: You will also need access to a Llama model, either running locally or via a hosted API.