exo/lm-dataset-preparer
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Readme

Browse Source
This commit is contained in:
extio-de
2025-01-11 23:20:53 +01:00
parent 2628cce48e
commit dad9915870
1 changed files with 15 additions and 19 deletions
Download Patch File Download Diff File Expand all files Collapse all files

34
README.md
View File

@@ -1,33 +1,29 @@
# lm-dataset-preparer
This program is a comprehensive system for generating and processing text datasets for training language models.
This program is a comprehensive toolkit for generating and manipulating text data, designed for use with large language models.
**Purpose:**
The program aims to create diverse and high-quality text datasets by leveraging various text processing techniques and language models. It enables users to:
The program aims to facilitate the creation of diverse text-based datasets for training and evaluating language models. It achieves this through a modular system of tools that perform various tasks:
* **Prepare and Transform Data:** The program can load, clean, translate, rewrite, and convert text data into different formats (JSON, JSONL).
* **Generate Contextual Prompts:** It can generate question-answer pairs and prompts based on input text, encouraging models to summarize, continue, or analyze the given content.
* **Create Synthetic Stories:** A dedicated tool uses agents to analyze input text, extract plot points, generate new storylines, and write complete stories.
* **Text Processing:** The program offers tools for translating, rewriting, splitting, and converting text into different formats (JSON, JSONL).
* **Dataset Creation:** It enables the generation of various types of datasets, including:
* Story datasets: Using a chain of agents, the program can analyze input text, extract plot points, generate new storylines, and write complete stories.
* Instruction completion datasets: It can create question-answer pairs, instruction-text pairs, and conversation data based on input text.
* Contextual prompt datasets: It generates prompts based on text, encouraging users to write stories or complete instructions.
* **Dataset Merging:** The program can merge multiple JSONL files into a single output file, allowing for the aggregation of datasets from different sources.
**Workflow:**
The program utilizes a modular design with different classes responsible for specific tasks:
1. **Data Loading and Execution:** The `Execution` class manages file processing, threading, and task execution, allowing for concurrent processing of multiple tools.
2. **Text Processing:** Classes like `TextProcessor` handle text normalization, splitting, and preamble removal.
3. **Translation and Rewriting:** Tools translate text to English and rewrite it using language models for improvement or enhancement.
4. **Prompt Generation:** Classes generate contextual prompts and question-answer pairs based on input text, leveraging character mapping and predefined templates.
5. **Story Generation:** The `Create` class uses agents to analyze input text, extract plot points, generate storylines, and write complete stories.
6. **Data Merging and Output:** Tools merge multiple JSONL files and output processed data in various formats.
The program utilizes a Spring Boot application as its entry point. This application manages the execution of various DatasetTool instances, each responsible for a specific task. These tools are configurable through properties and can interact with language models for tasks like translation and text generation.
**Key Features:**
* **Modularity:** The program is built with reusable components, allowing for easy customization and extension.
* **Concurrency:** It utilizes multi-threading to process data efficiently.
* **Language Model Integration:** It leverages language models for translation, rewriting, and prompt generation.
* **Customizability:** Users can configure various parameters, such as the number of stories to generate, prompt templates, and data formats.
* **Modularity:** The program is built with a modular design, allowing for easy extension and customization.
* **Concurrency:** It leverages multi-threading to process tasks concurrently, improving efficiency.
* **Configuration:** The program is highly configurable, allowing users to tailor its behavior to their specific needs.
* **Language Model Integration:** It seamlessly integrates with language models, enabling advanced text processing capabilities.
Overall, this program provides a powerful and flexible toolkit for researchers and developers working with text data and language models.
Overall, this program provides a powerful and versatile toolkit for researchers and developers working with text data and language models.
*gemma-2-27b-it-Q4_K_M.gguf, requests=13, requestDuration=PT2M5.366501869S, inTokens=16996, outTokens=1229, tps=9.832*
*gemma-2-27b-it-Q4_K_M.gguf, requests=15, requestDuration=PT2M12.702353502S, inTokens=19033, outTokens=1280, tps=9.7*