From dad991587009ee75a2634958326a118188e66d5b Mon Sep 17 00:00:00 2001 From: extio-de Date: Sat, 11 Jan 2025 23:20:53 +0100 Subject: [PATCH] Readme --- README.md | 34 +++++++++++++++------------------- 1 file changed, 15 insertions(+), 19 deletions(-) diff --git a/README.md b/README.md index 174da4a..b3a2236 100644 --- a/README.md +++ b/README.md @@ -1,33 +1,29 @@ # lm-dataset-preparer -This program is a comprehensive system for generating and processing text datasets for training language models. +This program is a comprehensive toolkit for generating and manipulating text data, designed for use with large language models. **Purpose:** -The program aims to create diverse and high-quality text datasets by leveraging various text processing techniques and language models. It enables users to: +The program aims to facilitate the creation of diverse text-based datasets for training and evaluating language models. It achieves this through a modular system of tools that perform various tasks: -* **Prepare and Transform Data:** The program can load, clean, translate, rewrite, and convert text data into different formats (JSON, JSONL). -* **Generate Contextual Prompts:** It can generate question-answer pairs and prompts based on input text, encouraging models to summarize, continue, or analyze the given content. -* **Create Synthetic Stories:** A dedicated tool uses agents to analyze input text, extract plot points, generate new storylines, and write complete stories. +* **Text Processing:** The program offers tools for translating, rewriting, splitting, and converting text into different formats (JSON, JSONL). +* **Dataset Creation:** It enables the generation of various types of datasets, including: + * Story datasets: Using a chain of agents, the program can analyze input text, extract plot points, generate new storylines, and write complete stories. + * Instruction completion datasets: It can create question-answer pairs, instruction-text pairs, and conversation data based on input text. + * Contextual prompt datasets: It generates prompts based on text, encouraging users to write stories or complete instructions. +* **Dataset Merging:** The program can merge multiple JSONL files into a single output file, allowing for the aggregation of datasets from different sources. **Workflow:** -The program utilizes a modular design with different classes responsible for specific tasks: - -1. **Data Loading and Execution:** The `Execution` class manages file processing, threading, and task execution, allowing for concurrent processing of multiple tools. -2. **Text Processing:** Classes like `TextProcessor` handle text normalization, splitting, and preamble removal. -3. **Translation and Rewriting:** Tools translate text to English and rewrite it using language models for improvement or enhancement. -4. **Prompt Generation:** Classes generate contextual prompts and question-answer pairs based on input text, leveraging character mapping and predefined templates. -5. **Story Generation:** The `Create` class uses agents to analyze input text, extract plot points, generate storylines, and write complete stories. -6. **Data Merging and Output:** Tools merge multiple JSONL files and output processed data in various formats. +The program utilizes a Spring Boot application as its entry point. This application manages the execution of various DatasetTool instances, each responsible for a specific task. These tools are configurable through properties and can interact with language models for tasks like translation and text generation. **Key Features:** -* **Modularity:** The program is built with reusable components, allowing for easy customization and extension. -* **Concurrency:** It utilizes multi-threading to process data efficiently. -* **Language Model Integration:** It leverages language models for translation, rewriting, and prompt generation. -* **Customizability:** Users can configure various parameters, such as the number of stories to generate, prompt templates, and data formats. +* **Modularity:** The program is built with a modular design, allowing for easy extension and customization. +* **Concurrency:** It leverages multi-threading to process tasks concurrently, improving efficiency. +* **Configuration:** The program is highly configurable, allowing users to tailor its behavior to their specific needs. +* **Language Model Integration:** It seamlessly integrates with language models, enabling advanced text processing capabilities. -Overall, this program provides a powerful and flexible toolkit for researchers and developers working with text data and language models. +Overall, this program provides a powerful and versatile toolkit for researchers and developers working with text data and language models. -*gemma-2-27b-it-Q4_K_M.gguf, requests=13, requestDuration=PT2M5.366501869S, inTokens=16996, outTokens=1229, tps=9.832* +*gemma-2-27b-it-Q4_K_M.gguf, requests=15, requestDuration=PT2M12.702353502S, inTokens=19033, outTokens=1280, tps=9.7*