exo/lm-dataset-preparer
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Commit Graph

  • 775bb8f1d0 Fix normalizeModelResponse main extio-de 2025-06-22 13:36:36 +02:00
  • dcaecaf52f Update spring boot extio-de 2025-06-22 13:36:13 +02:00
  • c2f50f27ba Upgraded to lm-lib 1.1.0; extio-de 2025-02-23 20:11:08 +01:00
  • 40d50f96ec New Tool: GenMergedText; Fixes; extio-de 2025-02-09 17:58:39 +01:00
  • c15e539e33 Optimized ProxyOutputStream; extio-de 2025-02-04 23:36:31 +01:00
  • 6e7db0c6d8 Reworked instruction generation tools; extio-de 2025-01-27 20:17:32 +01:00
  • 8be692c237 Fix Execution; Removed dep to commons-io; extio-de 2025-01-25 18:10:09 +01:00
  • 77637436f2 MergeJsonl: Separate outputs per filter extio-de 2025-01-25 17:36:05 +01:00
  • 36ba17ef23 GenConversation: Provide history in the prompt; extio-de 2025-01-12 11:36:53 +01:00
  • 538e92ca0a Randomize character names only once per conversation; extio-de 2025-01-12 10:57:37 +01:00
  • dad9915870 Readme extio-de 2025-01-11 23:20:53 +01:00
  • 2628cce48e mergeJsonl.filter; extio-de 2025-01-11 22:00:04 +01:00
  • 282b5ffb7f New tool: Contextual conversation; extio-de 2025-01-11 21:59:52 +01:00
  • c76b108c19 AbstractContextualPrompts minor refactoring; extio-de 2024-12-30 11:58:38 +01:00
  • ad001dae84 Misc improvements; extio-de 2024-12-29 12:05:15 +01:00
  • 4d320fa141 Output to multiple streams; extio-de 2024-12-27 12:27:10 +01:00
  • 436e996e5d Rewrite: Use different prompts for first and next continuations; extio-de 2024-12-27 12:25:59 +01:00
  • 674c47dfe1 Rewrite: Add previous split as history for better context; extio-de 2024-12-27 11:17:12 +01:00
  • 685c740c5c New tool: Create; New tool: Contextual instructions; Rewrite: Configure cnt for both impr and enh; Task Execution: Reimplemented task queuing approach; Fixes; extio-de 2024-12-26 21:49:51 +01:00
  • 07e5a4779a Update readme extio-de 2024-12-20 01:21:32 +01:00
  • dc5a4f3e25 Improve contextual prompt instruction; extio-de 2024-12-20 00:43:16 +01:00
  • d44fd53323 Configurable model categories; extio-de 2024-12-20 00:14:02 +01:00
  • 74eaa5c72b Tool pipelines; extio-de 2024-12-19 19:20:45 +01:00
  • bfb4e8be3b Text2Jsonl2HistoryInstructCompletion: Improved llm prompt; Utils: normalizeModelResponse(), streamOut(); extio-de 2024-12-19 14:04:01 +01:00
  • 7352f28b3c Readme updated; extio-de 2024-12-18 20:58:59 +01:00
  • 56b7dcb265 MergeJsonl tools accepts multiple directories; extio-de 2024-12-18 20:48:58 +01:00
  • 1c94b97b92 DatasetTool marker interface; extio-de 2024-12-18 20:42:12 +01:00
  • 54ebdb2e51 New tool: Text2Jsonl2HistoryInstructCompletion; extio-de 2024-12-18 20:07:07 +01:00
  • 75c2109d6d Switch dataset preparer to lmlib; extio-de 2024-12-18 20:05:03 +01:00
  • fb34561bd5 Client: Collect and show more statistics; MergeJsonl: Shuffle merged json; Rewrite: Fix progress indicator; extio-de 2024-12-10 18:11:48 +01:00
  • 6648ce43ef Updated description; extio-de 2024-12-02 23:15:44 +01:00
  • f0872fb7ee createCharacterNameMapping: Changed log level; extio-de 2024-12-02 11:36:03 +01:00
  • 57f7ec2e48 Text2Jsonl2WithContextualPrompts: Prompt improved; Name mapping requires at least 5 characters (otherwise we may replace random syllables); extio-de 2024-12-01 22:50:53 +01:00
  • b027c2a892 Text2Jsonl2WithContextualPrompts: Rename persons; Prompt conversation contexts; extio-de 2024-12-01 18:31:39 +01:00
  • dda9400cc1 Text2Jsonl2WithContextualPrompts: Overlap chunks; extio-de 2024-11-16 10:54:06 +01:00
  • 70822207dd splitParagraphs: Split by the closest occurence to chunk_norm instead of chunk_min; extio-de 2024-11-16 10:52:28 +01:00
  • 82f969d666 Fix paragraphToContextualPrompt response post-processing; extio-de 2024-11-03 22:53:35 +01:00
  • cb8ac57580 Improved prompt for Text2Jsonl2WithContextualPrompts; extio-de 2024-11-03 20:37:22 +01:00
  • 509f36238d Fix MixtralPromptStrategy; Refactoring; extio-de 2024-11-02 12:24:28 +01:00
  • 1813ab8928 Client refactoring; MergeJsonl tool; extio-de 2024-11-01 19:41:03 +01:00
  • 94a20bd4ed Merge pull request #1 from stephan-birkl/patch-1 extio-de 2024-11-01 19:18:35 +01:00
  • 3b6a0ee74a Update Client.java stephan-birkl 2024-11-01 17:43:40 +01:00
  • e40e648557 splitParagraphs extio-de 2024-11-01 15:48:28 +01:00
  • 14bd1c1915 splitParagraphs extio-de 2024-11-01 15:31:24 +01:00
  • 65e50e1dcc Sliding contexts for contextual prompts; extio-de 2024-11-01 15:08:26 +01:00
  • 7232387134 Switched remaining tools to streaming output; Generate contextual prompt datasets as single files per input; Added Alpaca prompt template; Generate multiple contextual prompt lengths; extio-de 2024-11-01 13:43:59 +01:00
  • 2087405c64 Readme; extio-de 2024-10-30 00:22:54 +01:00
  • dca4aa0a29 Refactoring; Readme; extio-de 2024-10-30 00:21:18 +01:00
  • efb520fc07 New generator: Jsonl QA style contextual prompts; JsonL: Add sliding windows; Streamed jsonl output; Memory optimization: Buffered, blocking task creation; extio-de 2024-10-29 23:51:41 +01:00
  • 4b93ed97be Fix splitParagraphs; Improved prompts; extio-de 2024-10-29 21:00:23 +01:00
  • 030bc5bee1 Improved rewriting; extio-de 2024-10-27 09:54:34 +01:00
  • 78a814a64e Readme extio-de 2024-10-20 22:01:19 +02:00
  • 59dc27a8ef initial commit extio-de 2024-10-20 21:04:32 +02:00