exo/lm-lib
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Streaming Client

Browse Source
This commit is contained in:
extio-de
2025-06-28 14:36:26 +02:00
parent 2350740136
commit 365043b5a0
23 changed files with 473 additions and 83 deletions
Download Patch File Download Diff File Expand all files Collapse all files

44
README.md
View File

@@ -51,7 +51,9 @@ These profiles must be mapped to fixed categories with the following application
The value is the name of the model profile .properties file, for example "llama3.3-70b-azure".
Profiles must be located in the main resources location.
Agent and completion requests will be invoked with these categories and will load the corresponding model profile.
Profiles must be located in the classpath resources location (e.g. src/main/resources).
Examples can be found in test resources.
@@ -75,8 +77,8 @@ The library supports both chat completions and (legacy) text completions. Choose
modelProvider=OAI_CHAT_COMPLETION
modelProvider=OAI_TEXT_COMPLETION
Chat completions are recommended for most use cases and easy to use, since the library only needs to send the actual conversation to the server, whereas for text completions, the library needs to create a raw prompt that includes the correct instruction formatting (e.g. special tokens). Text completions are still preferred if you need more control over the raw prompt or if you don't use an instruct model. For text completions, it is required to select the right prompt template. See package .lmlib.prompt for available prompt templates or implement your own PromptStrategy (must be a bean in the spring context).
To activate a prompt template, set the following property in your model profile:
Chat completions are recommended for most use cases and easy to use, since the library only needs to send the actual conversation to the server, whereas for text completions, the library needs to create a raw prompt that includes the correct instruction formatting (e.g. special tokens). Text completions are still preferred if you need more control over the raw prompt or if you don't use an instruct model. For text completions, it is required to select the right prompt template. See package .lmlib.prompt for available prompt templates or implement your own PromptStrategy (must be a bean in the spring context). There is also the special NoPromptStrategy ("no") that does not add any special tokens.
To activate a prompt template, set the following property in your model profile (corresponds to getPromptName()):
prompts=llama3
@@ -109,6 +111,28 @@ Example profile: src/test/resources/llama3.3-70b-azure.properties
<artifactId>caffeine</artifactId>
</dependency>
# Usage
1. Setup dependencies, see above
2. Configure model profiles, see above
3. Implement a CachedClientRepository if you need your requests cached
4. For chats or completions, use ClientService service bean
5. For agentic flows, create an AgentContext and use AgentExecutorService service bean
# Usage Examples
Client service: clientService.getClient(ModelCategory.MEDIUM);
Client usage details: CompletionClientTest and AzureAiClientTest
Agentic flow: AgentTest#agenticFlow
## Completions and Streaming
The library supports both streaming and non-streaming completions.
Streaming methods accept a Consumer to receive the next chunk (direct Client usage) or a callback on the next AgentContext update (Agents). They also return a full Completion object or the final AgentContext at the end of the execution that also contains the statistics.
Non-streaming methods only return the final objects at the end of the execution. This approach is much more resource-friendly and therefore recommended if you don't need to display or process the results in real-time.
# Completions Cache
All language model requests will be cached if you add a CachedClientRepository bean to your context.
@@ -131,20 +155,6 @@ Example:
assertTrue(Grader.assessScoreBinary("Does the following blog post mention the feature " + feature, resultContext.getStringValue("blogPost"), this.clientService));
# Usage
1. Setup dependencies, see above
2. Configure model profiles, see above
3. Implement a CachedClientRepository if you need your requests cached
4. For chats or completions, use ClientService service bean
5. For agentic flows, create an AgentContext and use AgentExecutorService service bean
# Usage Examples
Text completions: AzureAiClientTest#completion
Chat: AzureAiClientTest#chat
Agentic flow: AgentTest#agenticFlow
# Announcement generated by AgentTest#agenticFlow
## Introducing Advanced Agent Orchestration: Dynamic Conversations & Parallel Processing