lm-dataset-preparer/README.md

lm-dataset-preparer

This program is a comprehensive system for generating and processing text datasets for training language models.

Purpose:

The program aims to create diverse and high-quality text datasets by leveraging various text processing techniques and language models. It enables users to:

• Prepare and Transform Data: The program can load, clean, translate, rewrite, and convert text data into different formats (JSON, JSONL).
• Generate Contextual Prompts: It can generate question-answer pairs and prompts based on input text, encouraging models to summarize, continue, or analyze the given content.
• Create Synthetic Stories: A dedicated tool uses agents to analyze input text, extract plot points, generate new storylines, and write complete stories.

Workflow:

The program utilizes a modular design with different classes responsible for specific tasks:

1. Data Loading and Execution: The Execution class manages file processing, threading, and task execution, allowing for concurrent processing of multiple tools.
2. Text Processing: Classes like TextProcessor handle text normalization, splitting, and preamble removal.
3. Translation and Rewriting: Tools translate text to English and rewrite it using language models for improvement or enhancement.
4. Prompt Generation: Classes generate contextual prompts and question-answer pairs based on input text, leveraging character mapping and predefined templates.
5. Story Generation: The Create class uses agents to analyze input text, extract plot points, generate storylines, and write complete stories.
6. Data Merging and Output: Tools merge multiple JSONL files and output processed data in various formats.

Key Features:

• Modularity: The program is built with reusable components, allowing for easy customization and extension.
• Concurrency: It utilizes multi-threading to process data efficiently.
• Language Model Integration: It leverages language models for translation, rewriting, and prompt generation.
• Customizability: Users can configure various parameters, such as the number of stories to generate, prompt templates, and data formats.

Overall, this program provides a powerful and flexible toolkit for researchers and developers working with text data and language models.

gemma-2-27b-it-Q4_K_M.gguf, requests=13, requestDuration=PT2M5.366501869S, inTokens=16996, outTokens=1229, tps=9.832