30 lines
2.2 KiB
Markdown
30 lines
2.2 KiB
Markdown
# lm-dataset-preparer
|
|
|
|
This program is a comprehensive toolkit for generating and manipulating text data, designed for use with large language models.
|
|
|
|
**Purpose:**
|
|
|
|
The program aims to facilitate the creation of diverse text-based datasets for training and evaluating language models. It achieves this through a modular system of tools that perform various tasks:
|
|
|
|
* **Text Processing:** The program offers tools for translating, rewriting, splitting, and converting text into different formats (JSON, JSONL).
|
|
* **Dataset Creation:** It enables the generation of various types of datasets, including:
|
|
* Story datasets: Using a chain of agents, the program can analyze input text, extract plot points, generate new storylines, and write complete stories.
|
|
* Instruction completion datasets: It can create question-answer pairs, instruction-text pairs, and conversation data based on input text.
|
|
* Contextual prompt datasets: It generates prompts based on text, encouraging users to write stories or complete instructions.
|
|
* **Dataset Merging:** The program can merge multiple JSONL files into a single output file, allowing for the aggregation of datasets from different sources.
|
|
|
|
**Workflow:**
|
|
|
|
The program utilizes a Spring Boot application as its entry point. This application manages the execution of various DatasetTool instances, each responsible for a specific task. These tools are configurable through properties and can interact with language models for tasks like translation and text generation.
|
|
|
|
**Key Features:**
|
|
|
|
* **Modularity:** The program is built with a modular design, allowing for easy extension and customization.
|
|
* **Concurrency:** It leverages multi-threading to process tasks concurrently, improving efficiency.
|
|
* **Configuration:** The program is highly configurable, allowing users to tailor its behavior to their specific needs.
|
|
* **Language Model Integration:** It seamlessly integrates with language models, enabling advanced text processing capabilities.
|
|
|
|
Overall, this program provides a powerful and versatile toolkit for researchers and developers working with text data and language models.
|
|
|
|
*gemma-2-27b-it-Q4_K_M.gguf, requests=15, requestDuration=PT2M12.702353502S, inTokens=19033, outTokens=1280, tps=9.7*
|