feat: Enhance README with reasoning model configuration and clarify streaming behavior
This commit is contained in:
34
README.md
34
README.md
@@ -4,11 +4,11 @@ The purpose of lm-lib is to provide a modular, extensible Java framework for int
|
|||||||
|
|
||||||
The libraries's key features include:
|
The libraries's key features include:
|
||||||
|
|
||||||
- Unified API for multiple LLM providers (OpenAI, Azure, local models, etc.)
|
|
||||||
- Streaming and non-streaming completions
|
|
||||||
- Agentic flow execution
|
- Agentic flow execution
|
||||||
|
- Dynamic prompt templates with placeholder substitution for injecting agent context data
|
||||||
- Branching & merging (depth-1 parallel splits) with contextual graph tracing
|
- Branching & merging (depth-1 parallel splits) with contextual graph tracing
|
||||||
- Retry & parse validation loop per agent response
|
- Streaming and non-streaming chat and text completions with reasoning support
|
||||||
|
- Unified API for multiple LLM providers (OpenAI, Azure, local models, etc.)
|
||||||
- Model profile configuration and switching
|
- Model profile configuration and switching
|
||||||
- Prompt and conversation management utilities
|
- Prompt and conversation management utilities
|
||||||
- Request caching and statistics tracking
|
- Request caching and statistics tracking
|
||||||
@@ -70,6 +70,15 @@ Profiles must be located in the classpath resources location (e.g. src/main/reso
|
|||||||
|
|
||||||
Examples can be found in test resources.
|
Examples can be found in test resources.
|
||||||
|
|
||||||
|
### Reasoning Models
|
||||||
|
|
||||||
|
For models that support reasoning (e.g., OpenAI's o1 series), you can configure reasoning parameters in the model profile:
|
||||||
|
|
||||||
|
reasoningEffort=low|medium|high
|
||||||
|
reasoningSummaryDetails=auto|concise|detailed
|
||||||
|
|
||||||
|
These settings control how the model generates and presents its reasoning process. The reasoning content is captured separately from the main response and can be accessed via the `reasoning` field in `Completion` objects or the `<key>_reasoning` context key in agents.
|
||||||
|
|
||||||
## OpenAi compatible client
|
## OpenAi compatible client
|
||||||
|
|
||||||
Use tokenizer strategy jtokkit or llamaServer (if the model is hosted on a llama.cpp server)
|
Use tokenizer strategy jtokkit or llamaServer (if the model is hosted on a llama.cpp server)
|
||||||
@@ -138,9 +147,9 @@ Example profile: src/test/resources/llama3.3-70b-azure.properties
|
|||||||
|
|
||||||
The library supports both streaming and non-streaming completions.
|
The library supports both streaming and non-streaming completions.
|
||||||
|
|
||||||
Streaming methods accept a Consumer to receive the next chunk (direct Client usage) or a callback on the next AgentContext update (Agents). They also return a full Completion object or the final AgentContext at the end of the execution that also contains the statistics.
|
Streaming methods accept a `Consumer<Chunk>` to receive streaming updates. Each `Chunk` contains both `content()` and `reasoningContent()` fields, allowing models that support reasoning to stream their thought process separately from the final response. They also return a full `Completion` object or the final `AgentContext` at the end of the execution that also contains the statistics.
|
||||||
|
|
||||||
Non-streaming methods only return the final objects at the end of the execution. This approach is much more resource-friendly and therefore recommended if you don't need to display or process the results in real-time.
|
Non-streaming methods only return the final objects at the end of the execution. This approach is much more resource-friendly and therefore recommended if you don't need to display or process the results in real-time. The `Completion` record includes both `response()` and `reasoning()` fields.
|
||||||
|
|
||||||
# Completions Cache
|
# Completions Cache
|
||||||
|
|
||||||
@@ -262,22 +271,25 @@ Each agent completion is parsed by a fresh `responseHandler()` instance (created
|
|||||||
Response handlers implement `AgentResponseHandler` (or `StreamedAgentResponseHandler`) and transform raw model output into structured context keys. A handler returning `false` triggers a retry (up to one additional attempt).
|
Response handlers implement `AgentResponseHandler` (or `StreamedAgentResponseHandler`) and transform raw model output into structured context keys. A handler returning `false` triggers a retry (up to one additional attempt).
|
||||||
|
|
||||||
### TextAgentResponseHandler
|
### TextAgentResponseHandler
|
||||||
Purpose: Capture the full response (optionally transformed) under a single key. Implements streaming.
|
Purpose: Capture the full response (optionally transformed) under a single key. Implements streaming with support for reasoning content.
|
||||||
Constructor variants:
|
Constructor variants:
|
||||||
* `new TextAgentResponseHandler(key)`
|
* `new TextAgentResponseHandler(key)`
|
||||||
* `new TextAgentResponseHandler(key, transformer, beforeStream, chunkTransformer, afterChunkUpdate)`
|
* `new TextAgentResponseHandler(key, transformer, beforeStream, chunkTransformer, afterChunkUpdate)`
|
||||||
|
|
||||||
|
Note: `chunkTransformer` signature is `Function<Chunk, Chunk>` and `afterChunkUpdate` signature is `BiConsumer<AgentContext, Chunk>`.
|
||||||
|
|
||||||
Streaming lifecycle:
|
Streaming lifecycle:
|
||||||
1. `beforeStream()` sets the target key to empty string and invokes optional `beforeStream` callback.
|
1. `beforeStream()` sets the target key to empty string and invokes optional `beforeStream` callback.
|
||||||
2. For each chunk: `handleChunk()` optionally transforms chunk, sets `UPDATE_KEY` (= `chunkUpdateKey`) to the logical key, sets `<key>_chunk` to the last chunk, appends chunk to aggregated key, invokes `afterChunkUpdate`, then cleans temporary keys.
|
2. For each chunk: `handleChunk()` receives a `Chunk` object, optionally transforms it, appends `content()` to the aggregated key and `reasoningContent()` to `<key>_reasoning`, then invokes `afterChunkUpdate` with both context and chunk.
|
||||||
3. After stream completion the final `Completion` is passed to `handle()` (allowing a final transformer pass / overwrite).
|
3. After stream completion the final `Completion` is passed to `handle()` (allowing a final transformer pass / overwrite).
|
||||||
|
|
||||||
Context keys during streaming (transient):
|
Context keys during streaming:
|
||||||
* `<key>`: aggregated content so far
|
* `<key>`: aggregated content so far
|
||||||
* `<key>_chunk`: last received chunk (removed after callback)
|
* `<key>_reasoning`: aggregated reasoning content (if model provides it)
|
||||||
* `chunkUpdateKey`: signals which key received an update (removed post-callback)
|
|
||||||
|
|
||||||
Use cases: plain text generation, incremental UI updates, progressive summarization.
|
Both reasoning and response content are logged when available.
|
||||||
|
|
||||||
|
Use cases: plain text generation, incremental UI updates, progressive summarization, observing model reasoning process.
|
||||||
|
|
||||||
### AccumulateTextAgentResponseHandler
|
### AccumulateTextAgentResponseHandler
|
||||||
Purpose: Append each response to a list under the provided key (e.g. collecting multiple answers or iterative drafts).
|
Purpose: Append each response to a list under the provided key (e.g. collecting multiple answers or iterative drafts).
|
||||||
|
|||||||
Reference in New Issue
Block a user