Readme

README.md

@@ -1 +1,26 @@
 # lm-dataset-preparer
+
+**Program Name:** Text Processing and Translation Program
+
+**Purpose:** The program is designed to process and translate text files, generating new paragraphs and converting them into various formats. It utilizes a client to make API calls to a model, applies text normalization and transformation techniques, and writes the results to new files.
+
+**Key Features:**
+
+1. **Text Translation:** Translates text files from various languages to English using a client-provided completion API.
+2. **Model-based Rewriting:** Generates new paragraphs based on input text and model categories, applying multiple models and enhancements.
+3. **Text Normalization:** Removes excessive newlines, spaces, and tabs, and replaces multiple consecutive newlines with a single newline.
+4. **Paragraph Splitting:** Splits text into paragraphs based on a specified chunk size and tolerance range.
+5. **Directory Transformation:** Applies a function to each file in a directory in parallel using a fixed thread pool.
+6. **File Conversion:** Converts text files into JSONL (JSON Lines) format, splitting them into paragraphs and chunks based on provided normalization and variation values.
+
+**Components:**
+
+1. A Spring component that translates text files to English.
+2. A Spring component that rewrites text files by generating new paragraphs based on input text and model categories.
+3. A Spring component that converts text files into JSONL format.
+4. A utility class with methods for text normalization, paragraph splitting, and directory transformation.
+
+Overall, the program is designed to efficiently process and translate text files, applying various techniques to generate new paragraphs and convert them into different formats.
+
+*[requests=5, requestDuration=PT14.64756857S, inTokens=3966, outTokens=801, tps=57.214285714285715]*
+