34 lines
2.4 KiB
Markdown
34 lines
2.4 KiB
Markdown
# lm-dataset-preparer
|
|
|
|
This program is a comprehensive system for generating and processing text datasets for training language models.
|
|
|
|
**Purpose:**
|
|
|
|
The program aims to create diverse and high-quality text datasets by leveraging various text processing techniques and language models. It enables users to:
|
|
|
|
* **Prepare and Transform Data:** The program can load, clean, translate, rewrite, and convert text data into different formats (JSON, JSONL).
|
|
* **Generate Contextual Prompts:** It can generate question-answer pairs and prompts based on input text, encouraging models to summarize, continue, or analyze the given content.
|
|
* **Create Synthetic Stories:** A dedicated tool uses agents to analyze input text, extract plot points, generate new storylines, and write complete stories.
|
|
|
|
**Workflow:**
|
|
|
|
The program utilizes a modular design with different classes responsible for specific tasks:
|
|
|
|
1. **Data Loading and Execution:** The `Execution` class manages file processing, threading, and task execution, allowing for concurrent processing of multiple tools.
|
|
2. **Text Processing:** Classes like `TextProcessor` handle text normalization, splitting, and preamble removal.
|
|
3. **Translation and Rewriting:** Tools translate text to English and rewrite it using language models for improvement or enhancement.
|
|
4. **Prompt Generation:** Classes generate contextual prompts and question-answer pairs based on input text, leveraging character mapping and predefined templates.
|
|
5. **Story Generation:** The `Create` class uses agents to analyze input text, extract plot points, generate storylines, and write complete stories.
|
|
6. **Data Merging and Output:** Tools merge multiple JSONL files and output processed data in various formats.
|
|
|
|
**Key Features:**
|
|
|
|
* **Modularity:** The program is built with reusable components, allowing for easy customization and extension.
|
|
* **Concurrency:** It utilizes multi-threading to process data efficiently.
|
|
* **Language Model Integration:** It leverages language models for translation, rewriting, and prompt generation.
|
|
* **Customizability:** Users can configure various parameters, such as the number of stories to generate, prompt templates, and data formats.
|
|
|
|
Overall, this program provides a powerful and flexible toolkit for researchers and developers working with text data and language models.
|
|
|
|
*gemma-2-27b-it-Q4_K_M.gguf, requests=13, requestDuration=PT2M5.366501869S, inTokens=16996, outTokens=1229, tps=9.832*
|