From 78a814a64e719027aa991392edcf5d8e4d996a3d Mon Sep 17 00:00:00 2001 From: extio-de Date: Sun, 20 Oct 2024 22:01:19 +0200 Subject: [PATCH] Readme --- README.md | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/README.md b/README.md index 9386419..7fdc9aa 100644 --- a/README.md +++ b/README.md @@ -1 +1,26 @@ # lm-dataset-preparer + +**Program Name:** Text Processing and Translation Program + +**Purpose:** The program is designed to process and translate text files, generating new paragraphs and converting them into various formats. It utilizes a client to make API calls to a model, applies text normalization and transformation techniques, and writes the results to new files. + +**Key Features:** + +1. **Text Translation:** Translates text files from various languages to English using a client-provided completion API. +2. **Model-based Rewriting:** Generates new paragraphs based on input text and model categories, applying multiple models and enhancements. +3. **Text Normalization:** Removes excessive newlines, spaces, and tabs, and replaces multiple consecutive newlines with a single newline. +4. **Paragraph Splitting:** Splits text into paragraphs based on a specified chunk size and tolerance range. +5. **Directory Transformation:** Applies a function to each file in a directory in parallel using a fixed thread pool. +6. **File Conversion:** Converts text files into JSONL (JSON Lines) format, splitting them into paragraphs and chunks based on provided normalization and variation values. + +**Components:** + +1. A Spring component that translates text files to English. +2. A Spring component that rewrites text files by generating new paragraphs based on input text and model categories. +3. A Spring component that converts text files into JSONL format. +4. A utility class with methods for text normalization, paragraph splitting, and directory transformation. + +Overall, the program is designed to efficiently process and translate text files, applying various techniques to generate new paragraphs and convert them into different formats. + +*[requests=5, requestDuration=PT14.64756857S, inTokens=3966, outTokens=801, tps=57.214285714285715]* +