diff --git a/README.md b/README.md index 86f4d2f..174da4a 100644 --- a/README.md +++ b/README.md @@ -1,25 +1,33 @@ # lm-dataset-preparer -This program is a comprehensive text processing and data generation pipeline designed for training and evaluating language models. - -**Here's a breakdown of its functionality:** - -1. **Data Ingestion and Preparation:** The program reads text files, splits them into paragraphs, and performs various preprocessing steps like normalization and cleaning. It can also extract names from the text and replace them with randomly selected alternatives. - -2. **Prompt Generation:** The program generates different types of prompts based on the input text: - * **Contextual Prompts:** It creates question-answer pairs that encourage summarization or continuation of the text, incorporating context from previous paragraphs. - * **Instruction-Completion Pairs:** It uses predefined templates to generate instructions based on text chunks, suitable for training instruction-following models. - -3. **Text Rewriting and Translation:** The program can rewrite text using a language model, applying different rewriting strategies. It also offers text translation capabilities, converting text from a source language to English. - -4. **Data Formatting and Output:** The program formats the processed data into JSON Lines (JSONL) format, a common standard for machine learning datasets. It can merge multiple JSONL files, shuffle data, and write output to specified files. - -5. **Language Model Interaction:** The program provides a standardized way to interact with different language models, allowing users to configure and access them based on predefined categories. - -6. **Execution and Task Management:** The program utilizes a thread pool for efficient parallel processing of tasks, managing queues and scheduling for optimal performance. +This program is a comprehensive system for generating and processing text datasets for training language models. **Purpose:** -The primary purpose of this program is to generate high-quality, diverse datasets for training and evaluating language models. By providing various text processing, prompt generation, and data formatting capabilities, the program enables researchers and developers to create customized datasets tailored to their specific needs. +The program aims to create diverse and high-quality text datasets by leveraging various text processing techniques and language models. It enables users to: -*gemma-2-27b-it-Q4_K_M.gguf; requests=12, requestDuration=PT1M28.244075765S, inTokens=12090, outTokens=880, tps=10.0* +* **Prepare and Transform Data:** The program can load, clean, translate, rewrite, and convert text data into different formats (JSON, JSONL). +* **Generate Contextual Prompts:** It can generate question-answer pairs and prompts based on input text, encouraging models to summarize, continue, or analyze the given content. +* **Create Synthetic Stories:** A dedicated tool uses agents to analyze input text, extract plot points, generate new storylines, and write complete stories. + +**Workflow:** + +The program utilizes a modular design with different classes responsible for specific tasks: + +1. **Data Loading and Execution:** The `Execution` class manages file processing, threading, and task execution, allowing for concurrent processing of multiple tools. +2. **Text Processing:** Classes like `TextProcessor` handle text normalization, splitting, and preamble removal. +3. **Translation and Rewriting:** Tools translate text to English and rewrite it using language models for improvement or enhancement. +4. **Prompt Generation:** Classes generate contextual prompts and question-answer pairs based on input text, leveraging character mapping and predefined templates. +5. **Story Generation:** The `Create` class uses agents to analyze input text, extract plot points, generate storylines, and write complete stories. +6. **Data Merging and Output:** Tools merge multiple JSONL files and output processed data in various formats. + +**Key Features:** + +* **Modularity:** The program is built with reusable components, allowing for easy customization and extension. +* **Concurrency:** It utilizes multi-threading to process data efficiently. +* **Language Model Integration:** It leverages language models for translation, rewriting, and prompt generation. +* **Customizability:** Users can configure various parameters, such as the number of stories to generate, prompt templates, and data formats. + +Overall, this program provides a powerful and flexible toolkit for researchers and developers working with text data and language models. + +*gemma-2-27b-it-Q4_K_M.gguf, requests=13, requestDuration=PT2M5.366501869S, inTokens=16996, outTokens=1229, tps=9.832* diff --git a/pom.xml b/pom.xml index 4a5c7f3..d31a663 100755 --- a/pom.xml +++ b/pom.xml @@ -8,13 +8,25 @@ 21 - 21 - 21 + 21 + 21 2.5.0 - 3.4.0 - 1.0.0-SNAPSHOT + 3.4.1 + 1.0.0 + + + spacecraft-tactics-releases + Spacecraft Tactics Repository + https://spacecraft-tactics.com/repository/releases + + + central + https://repo1.maven.org/maven2/ + + + @@ -36,17 +48,17 @@ com.github.ben-manes.caffeine caffeine - - org.apache.commons - commons-lang3 - - - - de.extio - lm-lib - ${lmlib.version} - - + + org.apache.commons + commons-lang3 + + + + de.extio + lm-lib + ${lmlib.version} + + org.springframework.boot spring-boot-starter-test diff --git a/src/main/java/de/extio/lmdatasetprep/Execution.java b/src/main/java/de/extio/lmdatasetprep/Execution.java index cf54ed6..b4d3733 100644 --- a/src/main/java/de/extio/lmdatasetprep/Execution.java +++ b/src/main/java/de/extio/lmdatasetprep/Execution.java @@ -10,13 +10,12 @@ import java.nio.file.Paths; import java.util.List; import java.util.Properties; import java.util.concurrent.BlockingQueue; -import java.util.concurrent.CompletableFuture; import java.util.concurrent.ConcurrentHashMap; import java.util.concurrent.ConcurrentMap; import java.util.concurrent.ExecutorService; import java.util.concurrent.Executors; import java.util.concurrent.LinkedBlockingQueue; -import java.util.concurrent.ScheduledExecutorService; +import java.util.concurrent.Semaphore; import java.util.concurrent.TimeUnit; import java.util.concurrent.atomic.AtomicBoolean; import java.util.concurrent.atomic.AtomicInteger; @@ -38,25 +37,20 @@ public class Execution { public static final ConcurrentMap>> work = new ConcurrentHashMap<>(); - public static final BlockingQueue> tasks; - - public static final ScheduledExecutorService scheduler; - public static final ExecutorService executorService; + public static final AtomicInteger tasksRunning = new AtomicInteger(); + + private static final Semaphore taskSemaphore; + private static final Logger LOGGER = LoggerFactory.getLogger(Execution.class); static { final int EXECUTORS = Integer.parseInt(LmDatasetPreparerApplication.applicationContext.getEnvironment().getProperty("agent.threads")); - tasks = new LinkedBlockingQueue<>(EXECUTORS * 2); - executorService = Executors.newFixedThreadPool(EXECUTORS); - scheduler = Executors.newScheduledThreadPool(1); - scheduler.scheduleAtFixedRate(() -> { - tasks.removeIf(future -> future.isDone()); - }, 0, 1, TimeUnit.SECONDS); + taskSemaphore = new Semaphore(EXECUTORS * 2); } public static void transform(final String source, final Function> createTasks) { @@ -83,12 +77,7 @@ public class Execution { }) .forEach(p -> { for (final Runnable r : createTasks.apply(p)) { - try { - tasks.put(CompletableFuture.runAsync(r, executorService)); - } - catch (final InterruptedException e) { - throw new RuntimeException("Interrupted", e); - } + executeTask(r); } }); } @@ -120,16 +109,30 @@ public class Execution { WorkPacket next; while ((next = getNext.get()) != null) { for (final Runnable r : createTasks.apply(next)) { - try { - tasks.put(CompletableFuture.runAsync(r, executorService)); - } - catch (final InterruptedException e) { - throw new RuntimeException("Interrupted", e); - } + executeTask(r); } } } + private static void executeTask(final Runnable r) { + try { + taskSemaphore.acquire(); + tasksRunning.incrementAndGet(); + executorService.execute(() -> { + try { + r.run(); + } + finally { + tasksRunning.decrementAndGet(); + taskSemaphore.release(); + } + }); + } + catch (final InterruptedException e) { + throw new RuntimeException("Interrupted", e); + } + } + public static Path suffixFilename(final Path f, final String... suffixes) { final StringBuilder sb = new StringBuilder(); for (final String suffix : suffixes) { diff --git a/src/main/java/de/extio/lmdatasetprep/LmDatasetPreparerApplication.java b/src/main/java/de/extio/lmdatasetprep/LmDatasetPreparerApplication.java index 6379a3c..633661a 100755 --- a/src/main/java/de/extio/lmdatasetprep/LmDatasetPreparerApplication.java +++ b/src/main/java/de/extio/lmdatasetprep/LmDatasetPreparerApplication.java @@ -20,6 +20,7 @@ import org.springframework.boot.autoconfigure.SpringBootApplication; import org.springframework.cache.annotation.EnableCaching; import org.springframework.context.ApplicationContext; import org.springframework.context.ApplicationContextAware; +import org.springframework.context.ConfigurableApplicationContext; import org.springframework.context.annotation.ComponentScan; import de.extio.lmdatasetprep.tools.DatasetTool; @@ -77,9 +78,9 @@ public class LmDatasetPreparerApplication implements CommandLineRunner, Applicat LOGGER.info("All file consumers are finished"); int cnt = 0; - while (cnt < 3) { + while (cnt < 5) { Thread.sleep(1000); - if (!Execution.tasks.isEmpty() || Execution.work.values().stream().anyMatch(q -> q.values().stream().anyMatch(bq -> !bq.isEmpty()))) { + if (Execution.tasksRunning.get() != 0 || Execution.work.values().stream().anyMatch(q -> q.values().stream().anyMatch(bq -> !bq.isEmpty()))) { cnt = 0; } else { @@ -94,8 +95,8 @@ public class LmDatasetPreparerApplication implements CommandLineRunner, Applicat } LOGGER.info("Shutdown"); - Execution.scheduler.shutdown(); Execution.executorService.shutdown(); + ((ConfigurableApplicationContext) applicationContext).close(); } catch (final Exception e) { LOGGER.error("Error executing the component", e); diff --git a/src/main/java/de/extio/lmdatasetprep/tools/AbstractContextualPrompts.java b/src/main/java/de/extio/lmdatasetprep/tools/AbstractContextualPrompts.java index 6f4bd36..7d75845 100644 --- a/src/main/java/de/extio/lmdatasetprep/tools/AbstractContextualPrompts.java +++ b/src/main/java/de/extio/lmdatasetprep/tools/AbstractContextualPrompts.java @@ -48,8 +48,12 @@ abstract class AbstractContextualPrompts implements InitializingBean, DatasetToo @Override public void afterPropertiesSet() throws Exception { - this.maleNames = new BufferedReader(new InputStreamReader(new ClassPathResource("male.txt").getInputStream())).lines().collect(Collectors.toList()); - this.femaleNames = new BufferedReader(new InputStreamReader(new ClassPathResource("female.txt").getInputStream())).lines().collect(Collectors.toList()); + try (var in = new ClassPathResource("male.txt").getInputStream()) { + this.maleNames = new BufferedReader(new InputStreamReader(in)).lines().collect(Collectors.toList()); + } + try (var in = new ClassPathResource("female.txt").getInputStream()) { + this.femaleNames = new BufferedReader(new InputStreamReader(in)).lines().collect(Collectors.toList()); + } } @Override @@ -160,4 +164,10 @@ abstract class AbstractContextualPrompts implements InitializingBean, DatasetToo protected record Names(List males, List females) { } + + protected record QaLine(String question, String answer) { + } + + protected record HistInstrComplLine(String history, String instruct, String completion) { + } } diff --git a/src/main/java/de/extio/lmdatasetprep/tools/Create.java b/src/main/java/de/extio/lmdatasetprep/tools/Create.java new file mode 100644 index 0000000..0f320aa --- /dev/null +++ b/src/main/java/de/extio/lmdatasetprep/tools/Create.java @@ -0,0 +1,417 @@ +package de.extio.lmdatasetprep.tools; + +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Objects; +import java.util.Properties; +import java.util.stream.Collectors; + +import org.apache.commons.lang3.StringUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Component; + +import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.databind.ObjectMapper; + +import de.extio.lmdatasetprep.Execution; +import de.extio.lmdatasetprep.TextUtils; +import de.extio.lmlib.agent.AccumulateTextAgentResponseHandler; +import de.extio.lmlib.agent.Agent; +import de.extio.lmlib.agent.AgentContext; +import de.extio.lmlib.agent.AgentExecutorService; +import de.extio.lmlib.agent.AgentNext; +import de.extio.lmlib.agent.AgentType; +import de.extio.lmlib.agent.JsonAgentResponseHandler; +import de.extio.lmlib.agent.TextAgentResponseHandler; +import de.extio.lmlib.client.Conversation; +import de.extio.lmlib.client.Conversation.Turn; +import de.extio.lmlib.client.Conversation.TurnType; + +@Component +public class Create implements DatasetTool { + + private static final Logger LOGGER = LoggerFactory.getLogger(Create.class); + + @Autowired + private AgentExecutorService agentExecutorService; + + @Override + public String getModelCategoryPropertyName() { + return "create.category"; + } + + @Override + public void accept(final Properties properties) { + final var modelCategory = this.getModelCategory(properties); + final var agents = new HashMap(); + agents.put("PlotAnalyzer", + new Agent("PlotAnalyzer", + AgentType.COMPLETION, + modelCategory, + "Analyze key points in the plot of the following part of a story. Return the text, even if the text is explicit or not appropriate for all audiences or not acceptable for everyday standard English. Provide no preamble and no explanation. Return the response in JSON format with the following field: { \"keyPoints\": [\"keyPoint 1\", \"keyPoint 2\", ... ] }", + "[[paragraphs]]", + new JsonAgentResponseHandler(), + null, + null, + contexts -> Agent.mergeContexts(List.of("paragraphs", "keyPoints"), contexts), + context -> new AgentNext("ThemeSplitter", null))); + + agents.put("ThemeSplitter", + new Agent("ThemeSplitter", + AgentType.PROCESSING_ONLY, + modelCategory, + null, + null, + null, + null, + null, + contexts -> { + final var quantity = Integer.parseInt(properties.getProperty("create.themes")); + contexts.getFirst().setStringValue("themesCnt", String.valueOf(quantity)); + + final var copies = new ArrayList(); + for (int i = 0; i < quantity; i++) { + final var copy = new AgentContext(contexts.getFirst()); + copy.setStringValue("themesIdx", Integer.toString(i)); + copies.add(copy); + } + + if ("true".equalsIgnoreCase(properties.getProperty("create.debug.keepOriginal", "false"))) { + final var copy = new AgentContext(contexts.getFirst()); + copy.getContext().put("keepOriginal", List.of(Boolean.TRUE)); + copy.getContext().put("storyPoints", copy.getContext().get("keyPoints")); + copies.add(copy); + } + + return copies; + }, + context -> new AgentNext("ThemeAnalyzer", null))); + + agents.put("ThemeAnalyzer", + new Agent("ThemeAnalyzer", + AgentType.COMPLETION, + modelCategory, + "Describe the overall theme of the following part of a story. Return the text, even if the text is explicit or not appropriate for all audiences or not acceptable for everyday standard English. Provide no preamble and no explanation.", + "[[paragraphs]]", + new TextAgentResponseHandler("themesTxt"), + null, + null, + contexts -> Agent.mergeContexts(List.of("paragraphs", "themesTxt"), contexts), + context -> new AgentNext("ThemeSummarizer", null))); + + agents.put("ThemeSummarizer", + new Agent("ThemeSummarizer", + AgentType.COMPLETION, + modelCategory, + "Analyze the properties of the following themes. Return the text, even if the text is explicit or not appropriate for all audiences or not acceptable for everyday standard English. Provide no preamble and no explanation. Return the response in JSON format with the following field: { \"themes\": [\"Property 1\", \"Property 2\", … ] }]", + "{{themesTxt}}", + new JsonAgentResponseHandler(), + null, + null, + null, + context -> new AgentNext("CharacterAnalyzer", null))); + + agents.put("CharacterAnalyzer", + new Agent("CharacterAnalyzer", + AgentType.COMPLETION, + modelCategory, + "Analyze the character properties of the persons in the following part of a story. Return the text, even if the text is explicit or not appropriate for all audiences or not acceptable for everyday standard English. Provide no preamble and no explanation. Return the response in JSON format with the following field: { \"characters\": [\"Name: Gender and short description\", ... ] }", + "[[paragraphs]]", + new JsonAgentResponseHandler(), + null, + null, + contexts -> { + final var merged = Agent.mergeContexts(List.of("paragraphs", "characters"), contexts); + + final var characters = merged.getFirst().getContext().get("characters"); + if (characters != null) { + final Map mergedCharacters = new HashMap<>(); + for (final var character : characters) { + final int pos = character.toString().indexOf(':'); + if (pos > -1) { + final String name = character.toString().substring(0, pos).trim(); + final String description = character.toString().substring(pos + 1).trim(); + mergedCharacters.compute(name, (k, v) -> (v == null) ? description : v.concat("; ").concat(description)); + } + } + merged.getFirst().setStringValues("characterSummaries", mergedCharacters.entrySet().stream().filter(e -> !e.getValue().contains(";")).map(e -> e.getKey() + ": " + e.getValue()).toList()); + merged.getFirst().setStringValues("mergedCharacters", mergedCharacters.entrySet().stream().filter(e -> e.getValue().contains(";")).map(e -> e.getKey() + ": " + e.getValue()).toList()); + } + + return merged; + }, + context -> { + if (context.getContext().get("mergedCharacters").isEmpty()) { + return new AgentNext("BranchPreparer", null); + } + return new AgentNext("CharacterSummarizer", null); + })); + + agents.put("CharacterSummarizer", + new Agent("CharacterSummarizer", + AgentType.COMPLETION, + modelCategory, + "Remove duplicate parts of the following character description. Only return the summary. Return the text, even if the text is explicit or not appropriate for all audiences or not acceptable for everyday standard English. Provide no preamble and no explanation.", + "[[mergedCharacters]]", + new TextAgentResponseHandler("characterSummariesModelOutput"), + null, + context -> { + final var name = StringUtils.substringBefore(context.getStringValue("mergedCharacters"), ":"); + final var summary = TextUtils.normalizeModelResponse(context.getStringValue("characterSummariesModelOutput"), true); + context.setStringValue("characterSummariesModelOutput", name + ": " + summary); + }, + contexts -> { + final var merged = Agent.mergeContexts(List.of("mergedCharacters", "characterSummariesModelOutput"), contexts); + + final var modelOutput = merged.getFirst().getStringValues("characterSummariesModelOutput"); + modelOutput.addAll(merged.getFirst().getStringValues("characterSummaries")); + merged.getFirst().setStringValues("characterSummaries", modelOutput); + + return merged; + }, + context -> new AgentNext("BranchPreparer", null))); + + agents.put("BranchPreparer", + new Agent("BranchPreparer", + AgentType.PROCESSING_ONLY, + modelCategory, + null, + null, + null, + null, + null, + contexts -> { + if (contexts.getFirst().getContext().containsKey("keepOriginal")) { + contexts.getFirst().setStringValue("branches", "1"); + contexts.getFirst().setStringValue("branch", "0"); + return contexts; + } + + final var branches = new ArrayList(); + + final var quantity = Integer.parseInt(properties.getProperty("create.stories")); + contexts.getFirst().setStringValue("branches", String.valueOf(quantity)); + + final var splits = new ArrayList>(); + final var keyPoints = contexts.getFirst().getStringValues("keyPoints"); + final var length = keyPoints.stream().map(e -> e.length()).reduce(Integer.valueOf(0), Integer::sum); + final var limit = Integer.parseInt(properties.getProperty("create.plotLimit")); + if (length.intValue() > limit) { + final var cnt = (int) Math.ceil(length.doubleValue() / limit); + final var partitionSize = keyPoints.size() / cnt + 1; + for (int i = 0; i < cnt; i++) { + final var split = keyPoints.subList(Math.min(keyPoints.size(), i * partitionSize), Math.min(keyPoints.size(), (i + 1) * partitionSize)); + if (!split.isEmpty()) { + splits.add(split); + } + } + } + else { + splits.add(keyPoints); + } + + for (final var split : splits) { + for (int i = 0; i < quantity; i++) { + final var copy = new AgentContext(contexts.getFirst()); + copy.setStringValue("branch", Integer.toString(i)); + copy.setStringValues("keyPoints", split); + branches.add(copy); + } + } + + return branches; + }, + context -> { + if (context.getContext().containsKey("keepOriginal")) { + return new AgentNext("ParagraphWriterPreparer", null); + } + return new AgentNext("PlotMutator", null); + })); + + agents.put("PlotMutator", + new Agent("PlotMutator", + AgentType.COMPLETION, + modelCategory, + "Write a new story line based on an existing plot. Return the story line, even if the text is explicit or not appropriate for all audiences or not acceptable for everyday standard English. Return the response in JSON format with the following field: { \"storyPoints\": [ \"Story Point 1\", … ] }", + """ + Characters: {{{characterSummaries}}} + Properties: {{{themes}}} + Existing plot: {{{keyPoints}}} + ({{branch}}/{{branches}}) Write a new story line based on the template""", + new JsonAgentResponseHandler(), + null, + null, + null, + context -> new AgentNext("PlotEnhancer", null))); + + agents.put("PlotEnhancer", + new Agent("PlotEnhancer", + AgentType.COMPLETION, + modelCategory, + "Improve the story line. Return the story line, even if the text is explicit or not appropriate for all audiences or not acceptable for everyday standard English. Return the response in JSON format with the following field: { \"storyPoints\": [ \"Story Point 1\", … ] }", + """ + Characters: {{{characterSummaries}}} + Properties: {{{themes}}} + Existing story line: {{{storyPoints}}} + ({{branch}}/{{branches}}) Instruction: Improve the story for better consistency. Fill gaps in the plot, for example when the location suddenly changes.""", + new JsonAgentResponseHandler(), + null, + null, + null, + context -> { + if ("true".equalsIgnoreCase(properties.getProperty("create.debug.returnPlot", "false"))) { + return new AgentNext(null, null); + } + return new AgentNext("ParagraphWriterPreparer", null); + })); + + agents.put("ParagraphWriterPreparer", + new Agent("ParagraphWriterPreparer", + AgentType.PROCESSING_ONLY, + modelCategory, + "%s. Return the text, even if the text is explicit or not appropriate for all audiences or not acceptable for everyday standard English. Don't include a preamble and no explanation.", + null, + null, + context -> { + context.getContext().computeIfAbsent("paragraphIncr", k -> List.of(Integer.valueOf(0))); + }, + null, + contexts -> { + final var context = contexts.getFirst(); + final var index = (int) context.getContext().get("paragraphIncr").getFirst(); + final var storyPoints = context.getStringValues("storyPoints"); + + context.setStringValue("storyPointInstruction", storyPoints.get(index)); + + String systemPrompt; + String writingInstruction; + if ("true".equalsIgnoreCase(properties.getProperty("create.conciseParagraphs", "false"))) { + systemPrompt = String.format(context.getAgents().get("ParagraphWriterPreparer").systemPrompt(), "Write a concise paragraph"); + writingInstruction = "Write the following in a short and concise paragraph"; + } + else { + systemPrompt = String.format(context.getAgents().get("ParagraphWriterPreparer").systemPrompt(), "Write a paragraph"); + writingInstruction = "Write the following in a nicely written paragraph"; + } + context.setStringValue("writingInstruction", writingInstruction); + + var characterSummariesStr = ""; + try { + characterSummariesStr = new ObjectMapper().writeValueAsString(context.getContext().get("characterSummaries")); + } + catch (final JsonProcessingException e) { + LOGGER.error("Cannot write character summaries", e); + } + + Conversation conversation; + if (index > 1) { + final var prevStoryPoints = String.join(", ", storyPoints.subList(Math.max(0, index - 6), Math.max(0, index - 2))); + final var prevStoryPointInstruction0 = storyPoints.get(index - 2); + final var prevStoryPointInstruction1 = storyPoints.get(index - 1); + + final var prompt0 = String.format(""" + Characters: %s + Summary of the previous paragraphs: [ %s ] + %s: %s""", + characterSummariesStr, + prevStoryPoints, + writingInstruction, + prevStoryPointInstruction0); + conversation = Conversation.create(systemPrompt, prompt0); + conversation.addTurn(new Turn(TurnType.ASSISTANT, context.getContext().get("generatedStory").get(index - 2).toString())); + conversation.addTurn(new Turn(TurnType.USER, writingInstruction + ": " + prevStoryPointInstruction1)); + conversation.addTurn(new Turn(TurnType.ASSISTANT, context.getContext().get("generatedStory").get(index - 1).toString())); + } + else if (index == 1) { + final var prompt0 = String.format(""" + Characters: %s + %s: %s""", + characterSummariesStr, + writingInstruction, + storyPoints.get(0).toString()); + conversation = Conversation.create(systemPrompt, prompt0); + conversation.addTurn(new Turn(TurnType.ASSISTANT, context.getStringValue("generatedStory"))); + } + else { + final var prompt0 = String.format("Characters: %s", characterSummariesStr); + conversation = Conversation.create(systemPrompt, prompt0); + conversation.addTurn(new Turn(TurnType.ASSISTANT, "")); + } + + context.setConversation(conversation); + return contexts; + }, + context -> new AgentNext("ParagraphWriter", null))); + + agents.put("ParagraphWriter", + new Agent("ParagraphWriter", + AgentType.CONVERSATION_WITH_SYSTEM_PROMPT, + modelCategory, + "", + "{{writingInstruction}}: {{storyPointInstruction}}", + new AccumulateTextAgentResponseHandler("generatedStory", completion -> TextUtils.normalizeModelResponse(completion, true)), + null, + null, + null, + context -> { + var index = (int) context.getContext().get("paragraphIncr").getFirst(); + if (++index >= context.getContext().get("storyPoints").size()) { + return new AgentNext(null, null); + } + context.getContext().put("paragraphIncr", List.of(index)); + return new AgentNext("ParagraphWriterPreparer", null); + })); + + Execution.transform(properties.getProperty("create.source"), + packet -> { + return List.of(() -> { + LOGGER.info("Template: {}", packet.file().getFileName()); + + final var context = new AgentContext(agents); + context.setStringValues("paragraphs", TextUtils.splitParagraphs(packet.text(), 2000, 300, false)); + + final var resultContexts = this.agentExecutorService.walkGraph(agents.get("PlotAnalyzer"), context); + + for (int i = 0; i < resultContexts.size(); i++) { + final var resultContext = resultContexts.get(i); + LOGGER.debug(resultContext.getGraph().toString()); + + if (!resultContext.isError()) { + final Path out = Execution.suffixFilename(packet.file().getFileName(), + "create", + properties.getProperty("create.model"), + String.valueOf(i)); + Execution.streamOut(out, "create.destination", properties, fos -> { + try { + final byte[] story; + if ("true".equalsIgnoreCase(properties.getProperty("create.debug.returnPlot", "false"))) { + story = resultContext.getContext().get("storyPoints").stream().map(Objects::toString).collect(Collectors.joining("\n")).getBytes(StandardCharsets.UTF_8); + } + else { + story = resultContext.getContext().get("generatedStory") + .stream() + .map(Objects::toString) + .collect(Collectors.joining("\n\n")) + .getBytes(StandardCharsets.UTF_8); + } + fos.write(story); + } + catch (final IOException e) { + LOGGER.error("IO exception", e); + } + }); + } + } + }); + }); + + } + +} diff --git a/src/main/java/de/extio/lmdatasetprep/tools/DatasetTool.java b/src/main/java/de/extio/lmdatasetprep/tools/DatasetTool.java index 730f9f6..bbdff47 100644 --- a/src/main/java/de/extio/lmdatasetprep/tools/DatasetTool.java +++ b/src/main/java/de/extio/lmdatasetprep/tools/DatasetTool.java @@ -10,7 +10,7 @@ import de.extio.lmlib.profile.ModelCategory; public interface DatasetTool extends Consumer { default String getModelCategoryPropertyName() { - return null; + throw new IllegalStateException("Tool does not set a modelCategoryPropertyName"); } default ModelCategory getModelCategory(final Properties properties) { diff --git a/src/main/java/de/extio/lmdatasetprep/tools/GenContextualInstructions.java b/src/main/java/de/extio/lmdatasetprep/tools/GenContextualInstructions.java new file mode 100644 index 0000000..2fa5fe6 --- /dev/null +++ b/src/main/java/de/extio/lmdatasetprep/tools/GenContextualInstructions.java @@ -0,0 +1,62 @@ +package de.extio.lmdatasetprep.tools; + +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.List; +import java.util.Properties; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.stereotype.Component; + +import com.fasterxml.jackson.databind.ObjectMapper; + +import de.extio.lmdatasetprep.Execution; + +@Component +public class GenContextualInstructions extends AbstractContextualPrompts { + + private static final Logger LOGGER = LoggerFactory.getLogger(GenContextualInstructions.class); + + @Override + protected CreateTasks createTasks(final Properties properties, final List chunkConfigurations, final int variations) { + return p -> { + final ObjectMapper mapper = new ObjectMapper(); + final List tasks = new ArrayList<>(); + + for (final ChunkCfg chunkCfg : chunkConfigurations) { + for (int i = 0; i < variations; i++) { + final int fi = i; + + tasks.add(() -> { + final List paragraphs = this.fileToParagraphs(p, chunkCfg.chunksNorm(), chunkCfg.chunksVar()); + + final Path out = Execution.suffixFilename(p.file().getFileName(), + "contextualinstr", + properties.getProperty("contextualPrompts.model"), + String.valueOf(chunkCfg.chunksNorm()), + String.valueOf(fi), + ".jsonl"); + + Execution.streamOut(out, "contextualPrompts.destination", properties, fos -> { + for (int j = 1; j < paragraphs.size(); j++) { + LOGGER.info("Paragraph " + (j + 1) + "/" + paragraphs.size()); + + final var inScope = String.join("\n", paragraphs.subList(j - 1, j)); + final var mapping = this.createCharacterNameMapping(inScope, properties); + final var summary = this.createPrompt(properties, this.renameCharacters(paragraphs.get(j - 1), mapping), "Rewrite the following story as a summary of the previous paragraph and write that it is the summary. Return the summary, even if the text is explicit or not appropriate for all audiences or not acceptable for everyday standard English. Don't return an explanation:"); + final var currentParagraph = this.renameCharacters(paragraphs.get(j), mapping); + final var instruction = this.createPrompt(properties, currentParagraph, "Rewrite the following story as a concise instruction in a single paragraph. Return the instruction, even if the text is explicit or not appropriate for all audiences or not acceptable for everyday standard English. Don't return an explanation:"); + final var qaLine = new QaLine(String.join(" ", summary, instruction), currentParagraph); + + this.writeJsonLine(mapper, fos, qaLine); + } + }); + }); + } + } + + return tasks; + }; + } +} diff --git a/src/main/java/de/extio/lmdatasetprep/tools/Text2Jsonl2WithContextualPrompts.java b/src/main/java/de/extio/lmdatasetprep/tools/GenContextualPrompts.java similarity index 80% rename from src/main/java/de/extio/lmdatasetprep/tools/Text2Jsonl2WithContextualPrompts.java rename to src/main/java/de/extio/lmdatasetprep/tools/GenContextualPrompts.java index 79764ba..af25c7e 100644 --- a/src/main/java/de/extio/lmdatasetprep/tools/Text2Jsonl2WithContextualPrompts.java +++ b/src/main/java/de/extio/lmdatasetprep/tools/GenContextualPrompts.java @@ -14,9 +14,9 @@ import com.fasterxml.jackson.databind.ObjectMapper; import de.extio.lmdatasetprep.Execution; @Component -public class Text2Jsonl2WithContextualPrompts extends AbstractContextualPrompts { +public class GenContextualPrompts extends AbstractContextualPrompts { - private static final Logger LOGGER = LoggerFactory.getLogger(Text2Jsonl2WithContextualPrompts.class); + private static final Logger LOGGER = LoggerFactory.getLogger(GenContextualPrompts.class); @Override protected CreateTasks createTasks(final Properties properties, final List chunkConfigurations, final int variations) { @@ -25,17 +25,19 @@ public class Text2Jsonl2WithContextualPrompts extends AbstractContextualPrompts final List tasks = new ArrayList<>(); for (final ChunkCfg chunkCfg : chunkConfigurations) { - final List paragraphs = this.fileToParagraphs(p, chunkCfg.chunksNorm(), chunkCfg.chunksVar()); - for (int i = 0; i < variations; i++) { - final Path out = Execution.suffixFilename(p.file().getFileName(), - "contextualds", - properties.getProperty("contextualPrompts.model"), - String.valueOf(chunkCfg.chunksNorm()), - String.valueOf(i), - ".jsonl"); + final int fi = i; tasks.add(() -> { + final List paragraphs = this.fileToParagraphs(p, chunkCfg.chunksNorm(), chunkCfg.chunksVar()); + + final Path out = Execution.suffixFilename(p.file().getFileName(), + "contextualds", + properties.getProperty("contextualPrompts.model"), + String.valueOf(chunkCfg.chunksNorm()), + String.valueOf(fi), + ".jsonl"); + Execution.streamOut(out, "contextualPrompts.destination", properties, fos -> { for (int j = 0; j < paragraphs.size(); j++) { LOGGER.info("Paragraph " + (j + 1) + "/" + paragraphs.size()); @@ -65,6 +67,4 @@ public class Text2Jsonl2WithContextualPrompts extends AbstractContextualPrompts }; } - protected record QaLine(String question, String answer) { - } } diff --git a/src/main/java/de/extio/lmdatasetprep/tools/Text2Jsonl2HistoryInstructCompletion.java b/src/main/java/de/extio/lmdatasetprep/tools/GenHistoryInstructCompletion.java similarity index 67% rename from src/main/java/de/extio/lmdatasetprep/tools/Text2Jsonl2HistoryInstructCompletion.java rename to src/main/java/de/extio/lmdatasetprep/tools/GenHistoryInstructCompletion.java index 6a45899..67159bc 100644 --- a/src/main/java/de/extio/lmdatasetprep/tools/Text2Jsonl2HistoryInstructCompletion.java +++ b/src/main/java/de/extio/lmdatasetprep/tools/GenHistoryInstructCompletion.java @@ -14,9 +14,9 @@ import com.fasterxml.jackson.databind.ObjectMapper; import de.extio.lmdatasetprep.Execution; @Component -public class Text2Jsonl2HistoryInstructCompletion extends AbstractContextualPrompts { +public class GenHistoryInstructCompletion extends AbstractContextualPrompts { - private static final Logger LOGGER = LoggerFactory.getLogger(Text2Jsonl2HistoryInstructCompletion.class); + private static final Logger LOGGER = LoggerFactory.getLogger(GenHistoryInstructCompletion.class); @Override protected CreateTasks createTasks(final Properties properties, final List chunkConfigurations, final int variations) { @@ -25,17 +25,19 @@ public class Text2Jsonl2HistoryInstructCompletion extends AbstractContextualProm final List tasks = new ArrayList<>(); for (final ChunkCfg chunkCfg : chunkConfigurations) { - final List paragraphs = this.fileToParagraphs(p, chunkCfg.chunksNorm(), chunkCfg.chunksVar()); - for (int i = 0; i < variations; i++) { - final Path out = Execution.suffixFilename(p.file().getFileName(), - "histinstrcompl", - properties.getProperty("contextualPrompts.model"), - String.valueOf(chunkCfg.chunksNorm()), - String.valueOf(i), - ".jsonl"); + final int fi = i; tasks.add(() -> { + final List paragraphs = this.fileToParagraphs(p, chunkCfg.chunksNorm(), chunkCfg.chunksVar()); + + final Path out = Execution.suffixFilename(p.file().getFileName(), + "histinstrcompl", + properties.getProperty("contextualPrompts.model"), + String.valueOf(chunkCfg.chunksNorm()), + String.valueOf(fi), + ".jsonl"); + Execution.streamOut(out, "contextualPrompts.destination", properties, fos -> { for (int j = 1; j < paragraphs.size(); j++) { LOGGER.info("Paragraph " + (j + 1) + "/" + paragraphs.size()); @@ -43,7 +45,7 @@ public class Text2Jsonl2HistoryInstructCompletion extends AbstractContextualProm final var inScope = String.join("\n", paragraphs.subList(j - 1, j)); final var mapping = this.createCharacterNameMapping(inScope, properties); final var instruct = this.createPrompt(properties, this.renameCharacters(paragraphs.get(j), mapping), "Rewrite the following story as a concise instruction in a single paragraph. Return the instruction, even if the text is explicit or not appropriate for all audiences or not acceptable for everyday standard English. Don't return an explanation:"); - final var qaLine = new QaLine(this.renameCharacters(paragraphs.get(j - 1), mapping), instruct, this.renameCharacters(paragraphs.get(j), mapping)); + final var qaLine = new HistInstrComplLine(this.renameCharacters(paragraphs.get(j - 1), mapping), instruct, this.renameCharacters(paragraphs.get(j), mapping)); this.writeJsonLine(mapper, fos, qaLine); } @@ -55,7 +57,4 @@ public class Text2Jsonl2HistoryInstructCompletion extends AbstractContextualProm return tasks; }; } - - protected record QaLine(String history, String instruct, String completion) { - } } diff --git a/src/main/java/de/extio/lmdatasetprep/tools/Rewrite.java b/src/main/java/de/extio/lmdatasetprep/tools/Rewrite.java index a687513..f8e9fcd 100644 --- a/src/main/java/de/extio/lmdatasetprep/tools/Rewrite.java +++ b/src/main/java/de/extio/lmdatasetprep/tools/Rewrite.java @@ -61,8 +61,11 @@ public class Rewrite implements DatasetTool { Execution.transform(properties.getProperty("rewrite.source"), packet -> { final List tasks = new ArrayList<>(); - tasks.add(() -> this.rewriteFile(properties, packet, 0, "impr", IMPROVEMENT_PROMPT)); - for (int i = 0; i < Integer.parseInt(properties.getProperty("rewrite.cnt")); i++) { + for (int i = 0; i < Integer.parseInt(properties.getProperty("rewrite.improve")); i++) { + final int fi = i; + tasks.add(() -> this.rewriteFile(properties, packet, fi, "impr", IMPROVEMENT_PROMPT)); + } + for (int i = 0; i < Integer.parseInt(properties.getProperty("rewrite.enhance")); i++) { final int fi = i; tasks.add(() -> this.rewriteFile(properties, packet, fi, "enh", String.format(ENHANCE_PROMPT, ENHANCEMENTS.get(ThreadLocalRandom.current().nextInt(ENHANCEMENTS.size()))))); } @@ -71,7 +74,7 @@ public class Rewrite implements DatasetTool { } private void rewriteFile(final Properties properties, final WorkPacket packet, final int i, final String identifier, final String prompt) { - final Path out = Execution.suffixFilename(packet.file(), + final Path out = Execution.suffixFilename(packet.file().getFileName(), "rewr", properties.getProperty("rewrite.model"), identifier, diff --git a/src/main/resources/application.properties b/src/main/resources/application.properties index 0c3b505..e0ef2e9 100755 --- a/src/main/resources/application.properties +++ b/src/main/resources/application.properties @@ -3,7 +3,7 @@ spring.main.web-application-type=NONE logging.level.de.extio=info profile.model.s=llama3-cold -profile.model.m=mixtral-cold +profile.model.m=mistral-cold profile.model.l=chatml-cold profile.model.hot=llama3-hot profile.model.cold=alpaca-cold diff --git a/src/main/resources/mixtral-cold.properties b/src/main/resources/mistral-cold.properties similarity index 81% rename from src/main/resources/mixtral-cold.properties rename to src/main/resources/mistral-cold.properties index 814991e..e194782 100755 --- a/src/main/resources/mixtral-cold.properties +++ b/src/main/resources/mistral-cold.properties @@ -1,8 +1,8 @@ -prompts=mixtral +prompts=mistralNemo tokenEncoding= maxTokens=2000 maxContextLength=4000 -temperature=0.2 +temperature=0.3 topP=1.0 maxContinuations=1 modelProvider=OAI_TEXT_COMPLETION diff --git a/src/main/resources/tool.properties.template b/src/main/resources/tool.properties.template index 4f70567..f0a75df 100644 --- a/src/main/resources/tool.properties.template +++ b/src/main/resources/tool.properties.template @@ -1,14 +1,17 @@ +translate.category=SMALL # Source and destination can be either a path... translate.source=/home/xxx/folder # ... or a queue. Queues are prefixed with "queue", after the prefix comes the queue name translate.destination=queue0 +rewrite.category=SMALL rewrite.source=queue0 rewrite.destination=queue1 -rewrite.cnt=2 +rewrite.improve=1 +rewrite.enhance=2 rewrite.model=l3dp -# .n indicates a sequence. Add as many keys as you need +# .n indicates that the key supports sequences. Add as many keys as you need mergeJsonl.source.0= mergeJsonl.destination= @@ -17,6 +20,7 @@ text2Jsonl.destination= text2Jsonl.chunkNorm=750 text2Jsonl.chunkVar=250 +contextualPrompts.category=SMALL contextualPrompts.source=queue1 contextualPrompts.destination=/home/xxx/outputdir contextualPrompts.model=l3dp @@ -25,3 +29,14 @@ contextualPrompts.chunkNorm.0=750 contextualPrompts.chunkVar.0=250 contextualPrompts.chunkNorm.1=1500 contextualPrompts.chunkVar.1=300 + +create.source= +create.destination= +create.category=SMALL +create.model=l3dp +create.plotLimit=6000 +create.conciseParagraphs=false +create.themes=2 +create.stories=2 +create.debug.keepOriginal=false +create.debug.returnPlot=false