exo/lm-dataset-preparer
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Updated description;

Browse Source
This commit is contained in:
extio-de
2024-12-02 23:15:44 +01:00
parent f0872fb7ee
commit 6648ce43ef
1 changed files with 12 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files

20
README.md
View File

@@ -1,19 +1,23 @@
# lm-dataset-preparer
This program is designed for processing and rewriting text files using a language model.
This program is a comprehensive text processing pipeline designed for preparing and enhancing textual data.
Here's a breakdown of its functionality:
**Here's a breakdown of its functionality:**
1. **Text Preparation:** The program utilizes several classes to prepare the text for processing. It can translate text files to English, normalize text, split text into paragraphs, and convert paragraphs into JSON format.
1. **Translation:** The program can translate text files from various languages to English using a language model client.
2. **Contextual Prompt Generation:** For each paragraph, the program generates contextual prompts using an external AI service. These prompts help guide the language model in understanding the context of the paragraph.
2. **Text Processing and Manipulation:** The `Utils` class provides tools for normalizing text, splitting it into paragraphs, and managing files and filenames.
3. **Text Rewriting:** The program leverages a language model to rewrite text paragraphs. It supports two types of rewriting: "improvement" and "enhancement," each with its own set of prompts. Multiple enhancement variations are possible by randomly selecting from a list of prompts.
3. **Text Rewriting:** This component leverages a language model to improve and enhance the quality of text. It splits text into paragraphs, sends them to the model with specific prompts, and combines the rewritten paragraphs into a new file.
4. **Output:** The rewritten text, along with the generated prompts, is saved in a JSONL file.
4. **JSON Conversion:** The program can convert text files into JSON Lines (JSONL) format, with each paragraph represented as a JSON object.
5. **Contextual Prompt Generation:** This module generates contextual prompts based on text paragraphs, extracts names from the text, and creates variations of those names. The generated prompts and paragraphs are stored in JSONL files.
6. **File Merging:** The program can merge multiple JSONL files into a single output file, facilitating data aggregation.
**Purpose:**
The overall purpose of this program is to facilitate the rewriting and improvement of text files using a language model. The contextual prompts and different rewriting modes allow for flexible and nuanced text manipulation. This could be useful for tasks such as fine-tuning of large language models (LLMs).
The overall purpose of this program is to provide a versatile toolkit for preparing text data for various downstream tasks, such as machine learning model training, text analysis, and content creation. The program's capabilities in translation, rewriting, JSON conversion, prompt generation, and file merging make it suitable for a wide range of text processing applications.
*[gemma-2-27b-it-Q4_K_L.gguf; requests=6, requestDuration=PT1M36.661188325S, inTokens=6473, outTokens=506, tps=5.270833333333333]*
*[gemma-2-27b-it-Q4_K_L.gguf; requests=7, requestDuration=PT2M40.547239104S, inTokens=8437, outTokens=605, tps=3.78125]*