2.2 KiB
lm-dataset-preparer
This program is a comprehensive toolkit for generating and manipulating text data, designed for use with large language models.
Purpose:
The program aims to facilitate the creation of diverse text-based datasets for training and evaluating language models. It achieves this through a modular system of tools that perform various tasks:
- Text Processing: The program offers tools for translating, rewriting, splitting, and converting text into different formats (JSON, JSONL).
- Dataset Creation: It enables the generation of various types of datasets, including:
- Story datasets: Using a chain of agents, the program can analyze input text, extract plot points, generate new storylines, and write complete stories.
- Instruction completion datasets: It can create question-answer pairs, instruction-text pairs, and conversation data based on input text.
- Contextual prompt datasets: It generates prompts based on text, encouraging users to write stories or complete instructions.
- Dataset Merging: The program can merge multiple JSONL files into a single output file, allowing for the aggregation of datasets from different sources.
Workflow:
The program utilizes a Spring Boot application as its entry point. This application manages the execution of various DatasetTool instances, each responsible for a specific task. These tools are configurable through properties and can interact with language models for tasks like translation and text generation.
Key Features:
- Modularity: The program is built with a modular design, allowing for easy extension and customization.
- Concurrency: It leverages multi-threading to process tasks concurrently, improving efficiency.
- Configuration: The program is highly configurable, allowing users to tailor its behavior to their specific needs.
- Language Model Integration: It seamlessly integrates with language models, enabling advanced text processing capabilities.
Overall, this program provides a powerful and versatile toolkit for researchers and developers working with text data and language models.
gemma-2-27b-it-Q4_K_M.gguf, requests=15, requestDuration=PT2M12.702353502S, inTokens=19033, outTokens=1280, tps=9.7