feat: Add OAI_RESPONSES model provider and implement Responses API client tests
- Updated ModelProfile to include OAI_RESPONSES in the ModelProvider enum. - Created CompletionClientTest to validate text and chat completion functionalities. - Added ResponsesApiClientTest to test Responses API interactions, including model listing and conversation handling. - Implemented ResponsesApiDialectTestConfiguration for managing reasoning efforts in tests. - Updated application-test.properties to include a new profile for responses. - Added local-responses.properties configuration for the Responses API.
This commit is contained in:
extio-de
85
README.md
85
README.md
@@ -13,13 +13,13 @@ lm-lib is published under the Apache License 2.0.
|
||||
|
||||
lm-lib is designed for applications that need more than single prompt-response calls. Its center of gravity is the agent framework: structured multi-step flows such as classification, planning, fan-out analysis, synthesis, grading, feedback loops, and other graph-based execution patterns.
|
||||
|
||||
It also supports direct completion use cases through a client abstraction layer. The built-in clients currently cover OpenAI-compatible chat and text completion endpoints, OpenRouter's OpenAI-compatible routing API, and Ollama's native chat API, and the abstraction remains open for additional provider implementations over time. Applications can also provide their own client implementations by implementing the relevant interfaces.
|
||||
It also supports direct completion use cases through a client abstraction layer. The built-in clients currently cover OpenAI-compatible chat completion endpoints, the newer OpenAI-compatible Responses API, legacy text completion endpoints, OpenRouter's OpenAI-compatible routing API, and Ollama's native chat API, and the abstraction remains open for additional provider implementations over time. Applications can also provide their own client implementations by implementing the relevant interfaces.
|
||||
|
||||
The current built-in client set supports OpenAI-compatible chat completions, OpenRouter chat completions with cached metadata discovery, legacy text completions, and native Ollama chat completions, with model-specific prompt strategies for text completion models.
|
||||
The current built-in client set supports OpenAI-compatible chat completions, OpenAI-compatible Responses, OpenRouter chat completions with cached metadata discovery, legacy text completions, and native Ollama chat completions, with model-specific prompt strategies for text completion models.
|
||||
|
||||
## Key Capabilities
|
||||
|
||||
- OpenAI-compatible chat and text completions, OpenRouter chat completions, plus native Ollama chat completions
|
||||
- OpenAI-compatible chat completions, OpenAI-compatible Responses, legacy text completions, OpenRouter chat completions, plus native Ollama chat completions
|
||||
- Chat-completion tool calling with tool definitions, returned tool calls, and tool-result follow-up rounds
|
||||
- Streaming and non-streaming APIs
|
||||
- Separate reasoning capture for models that expose reasoning output
|
||||
@@ -242,7 +242,7 @@ Each profile file is a `.properties` resource on the classpath.
|
||||
| `maxContextLength` | Integer | Yes | Maximum total context length. | `16000`, `32768`, `128000` |
|
||||
| `temperature` | Double | Yes | Sampling temperature. Must be greater than zero. | `0.2`, `0.4`, `0.7` |
|
||||
| `topP` | Double | Yes | Nucleus sampling parameter. Must be greater than zero. | `1.0`, `0.9` |
|
||||
| `modelProvider` | Enum | Yes | Completion client type. | `OAI_CHAT_COMPLETION`, `OAI_TEXT_COMPLETION`, `OPENROUTER`, `OLLAMA` |
|
||||
| `modelProvider` | Enum | Yes | Completion client type. | `OAI_CHAT_COMPLETION`, `OAI_RESPONSES`, `OAI_TEXT_COMPLETION`, `OPENROUTER`, `OLLAMA` |
|
||||
| `modelName` | String | No | Model name or deployment. Can be empty for endpoints where the model should be auto-detected. | `gpt-4`, `Llama-3.3-70B-Instruct`, `` |
|
||||
| `url` | String | Yes | Base API URL. | `https://api.openai.com/v1`, `http://localhost:5001` |
|
||||
| `apiKey` | String | No | Bearer token or Spring placeholder. Can be empty for local servers. | `sk-...`, `${apikey}`, `` |
|
||||
@@ -300,6 +300,27 @@ cost1MInTokens=0
|
||||
cost1MOutTokens=0
|
||||
```
|
||||
|
||||
OpenAI Responses:
|
||||
|
||||
```properties
|
||||
category=M
|
||||
prompts=
|
||||
tokenizer=jTokkit
|
||||
tokenEncoding=cl100k_base
|
||||
maxTokens=2500
|
||||
maxContextLength=128000
|
||||
temperature=0.4
|
||||
topP=1.0
|
||||
modelProvider=OAI_RESPONSES
|
||||
modelName=gpt-4.1
|
||||
url=https://api.openai.com
|
||||
apiKey=${OPENAI_API_KEY:}
|
||||
cost1MInTokens=0
|
||||
cost1MCachedInTokens=0
|
||||
cost1MOutTokens=0
|
||||
cost1MReasoningOutTokens=0
|
||||
```
|
||||
|
||||
Local Ollama chat completion:
|
||||
|
||||
```properties
|
||||
@@ -329,6 +350,7 @@ The built-in provider implementations currently support these types:
|
||||
|
||||
```properties
|
||||
modelProvider=OAI_CHAT_COMPLETION
|
||||
modelProvider=OAI_RESPONSES
|
||||
modelProvider=OAI_TEXT_COMPLETION
|
||||
modelProvider=OPENROUTER
|
||||
modelProvider=OLLAMA
|
||||
@@ -336,10 +358,12 @@ modelProvider=OLLAMA
|
||||
|
||||
### OpenAI-Compatible Providers
|
||||
|
||||
`OAI_CHAT_COMPLETION` and `OAI_TEXT_COMPLETION` target the OpenAI-style API surface.
|
||||
`OAI_CHAT_COMPLETION`, `OAI_RESPONSES`, and `OAI_TEXT_COMPLETION` target the OpenAI-style API surface.
|
||||
|
||||
Use `OAI_CHAT_COMPLETION` for cloud APIs such as OpenAI itself and for local or self-hosted servers that expose a compatible `/v1/chat/completions` interface. This is the preferred default because lm-lib can send the conversation directly, supports streaming, and exposes tool calls through the standard chat-completion flow.
|
||||
|
||||
Use `OAI_RESPONSES` when you want to keep the same lm-lib `Client` and `Conversation` API while targeting `/v1/responses` instead of `/v1/chat/completions`. This provider translates lm-lib conversation turns into Responses input items, supports streaming, supports function tool calling, and stays stateless by replaying the full lm-lib conversation on each request.
|
||||
|
||||
Use `OAI_TEXT_COMPLETION` when you need explicit prompt templating or must target instruct-style models that still depend on raw prompt construction rather than structured chat messages. This is mainly useful for older compatible backends, local servers configured around prompt templates, or if you need full control over the raw prompt that is sent to the LLM.
|
||||
|
||||
OpenAI-compatible deployments commonly fall into two operational patterns:
|
||||
@@ -347,7 +371,9 @@ OpenAI-compatible deployments commonly fall into two operational patterns:
|
||||
- Cloud endpoints: hosted APIs with a fixed base URL such as `https://api.openai.com/v1`, normally with a Bearer API key.
|
||||
- Local or self-hosted endpoints: llama-server, vLLM-compatible gateways, or other OpenAI-style frontends exposed on an internal URL such as `http://localhost:5001`.
|
||||
|
||||
If you need to adapt request shaping for a specific OpenAI-compatible provider, you can optionally register an `OpenAiProviderDialect` bean. This allows applications to suppress or switch request fields that a provider does not support while keeping `ModelProfile` provider-agnostic and runtime-driven.
|
||||
If you need to adapt request shaping for a specific chat-completions compatible provider, you can optionally register an `OpenAiProviderDialect` bean. This applies to the `OAI_CHAT_COMPLETION` and `OAI_TEXT_COMPLETION` providers.
|
||||
|
||||
`OAI_RESPONSES` is implemented as a separate standalone client and uses its own optional `OpenAiResponsesApiDialect` hook.
|
||||
|
||||
The default OpenAI dialect behavior is:
|
||||
|
||||
@@ -356,12 +382,30 @@ The default OpenAI dialect behavior is:
|
||||
- `chatTokenLimitParameterMode(...) = ChatTokenLimitParameterMode.MAX_COMPLETION_TOKENS`
|
||||
- `reasoningEffort(...) = ReasoningEffort.MEDIUM`
|
||||
|
||||
The default Responses dialect behavior is:
|
||||
|
||||
- `store(...) = false`
|
||||
- `reasoning(...) = new ResponsesReasoning(ReasoningEffort.MEDIUM, null, null)`
|
||||
- `include(...) = []`, with `reasoning.encrypted_content` added automatically for stateless reasoning replay
|
||||
- `background(...) = null`
|
||||
- `maxToolCalls(...) = null`
|
||||
- `metadata(...) = {}`
|
||||
- `promptCacheKey(...) = null`
|
||||
- `promptCacheRetention(...) = null`
|
||||
- `safetyIdentifier(...) = null`
|
||||
- `serviceTier(...) = null`
|
||||
- `text(...) = null`
|
||||
- `topLogprobs(...) = null`
|
||||
- `truncation(...) = null`
|
||||
- `streamOptions(...) = null`
|
||||
|
||||
Notes:
|
||||
|
||||
- `usage` is a llama-server style extension and is therefore disabled unless a dialect explicitly enables it.
|
||||
- `reasoning_effort` is sent as a top-level request field when reasoning is enabled.
|
||||
- `max_completion_tokens` is the default because it matches the current OpenAI recommendation.
|
||||
- Some older OpenAI-compatible servers still require `max_tokens`, which can be selected through the dialect.
|
||||
- Responses reasoning requests are sent in the native `reasoning` object and the client preserves raw response output items so tool-call loops can replay reasoning items, encrypted reasoning payloads, and assistant `phase` values on the next stateless request.
|
||||
|
||||
Example:
|
||||
|
||||
@@ -392,6 +436,35 @@ OpenAiProviderDialect legacyOpenAiProviderDialect() {
|
||||
}
|
||||
```
|
||||
|
||||
Example Responses dialect:
|
||||
|
||||
```java
|
||||
@Bean
|
||||
OpenAiResponsesApiDialect openAiResponsesApiDialect() {
|
||||
return new OpenAiResponsesApiDialect() {
|
||||
@Override
|
||||
public ResponsesReasoning reasoning(final ModelProfile modelProfile) {
|
||||
return new ResponsesReasoning(ReasoningEffort.HIGH, ReasoningSummaryGenerateMode.AUTO, ReasoningSummary.CONCISE);
|
||||
}
|
||||
|
||||
@Override
|
||||
public List<String> include(final ModelProfile modelProfile) {
|
||||
return List.of(OpenAiResponsesApiDialect.INCLUDE_REASONING_ENCRYPTED_CONTENT);
|
||||
}
|
||||
|
||||
@Override
|
||||
public ServiceTier serviceTier(final ModelProfile modelProfile) {
|
||||
return ServiceTier.FLEX;
|
||||
}
|
||||
|
||||
@Override
|
||||
public Truncation truncation(final ModelProfile modelProfile) {
|
||||
return Truncation.AUTO;
|
||||
}
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
### OpenRouter Provider
|
||||
|
||||
`OPENROUTER` targets OpenRouter's chat-completions API and its richer model discovery endpoints.
|
||||
|
||||
@@ -461,10 +461,10 @@ public interface BaseAgent {
|
||||
return;
|
||||
}
|
||||
final var completion = context.getLastCompletion();
|
||||
if ((completion.response() == null || completion.response().isEmpty()) && completion.toolCalls().isEmpty()) {
|
||||
if (completion.response().isEmpty() && completion.toolCalls().isEmpty() && completion.outputItems().isEmpty()) {
|
||||
return;
|
||||
}
|
||||
context.getConversation().addTurn(new Conversation.Turn(TurnType.ASSISTANT, completion.response() == null ? "" : completion.response(), completion.toolCalls(), null));
|
||||
context.getConversation().addTurn(new Conversation.Turn(TurnType.ASSISTANT, completion.response(), completion.toolCalls(), null, null, completion.outputItems()));
|
||||
}
|
||||
|
||||
private void rememberToolCallData(final AgentContext context, final ToolCallData toolCallData) {
|
||||
|
||||
@@ -11,6 +11,7 @@ import de.extio.lmlib.client.openrouter.OpenRouterModelDiscovery;
|
||||
import de.extio.lmlib.client.ollama.OllamaClient;
|
||||
import de.extio.lmlib.client.oai.completion.chat.ChatCompletionClient;
|
||||
import de.extio.lmlib.client.oai.completion.text.TextCompletionClient;
|
||||
import de.extio.lmlib.client.oai.responses.ResponsesApiClient;
|
||||
import de.extio.lmlib.profile.ProfileAutoConfiguration;
|
||||
import de.extio.lmlib.token.TokenizerAutoConfiguration;
|
||||
|
||||
@@ -37,6 +38,12 @@ public class ClientAutoConfiguration {
|
||||
return new ChatCompletionClient();
|
||||
}
|
||||
|
||||
@Bean
|
||||
@ConditionalOnMissingBean
|
||||
ResponsesApiClient responsesApiClient() {
|
||||
return new ResponsesApiClient();
|
||||
}
|
||||
|
||||
@Bean
|
||||
@ConditionalOnMissingBean
|
||||
OpenRouterModelDiscovery openRouterModelDiscovery() {
|
||||
|
||||
@@ -3,8 +3,22 @@ package de.extio.lmlib.client;
|
||||
import java.math.BigDecimal;
|
||||
import java.time.Duration;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
|
||||
public final record Completion(String response, String reasoning, CompletionFinishReason finishReason, CompletionStatistics statistics, List<ToolCall> toolCalls) {
|
||||
/**
|
||||
* Normalized result returned by lm-lib clients.
|
||||
*
|
||||
* @param response the assistant-visible text output
|
||||
* @param reasoning the separately exposed reasoning summary or reasoning text
|
||||
* @param finishReason why generation stopped
|
||||
* @param statistics token, duration, cache, and cost accounting for the request
|
||||
* @param toolCalls tool calls returned by the provider when the response stopped
|
||||
* for tool execution
|
||||
* @param outputItems provider-specific structured output items that need to
|
||||
* survive follow-up rounds, primarily for stateless OpenAI Responses tool
|
||||
* loops
|
||||
*/
|
||||
public final record Completion(String response, String reasoning, CompletionFinishReason finishReason, CompletionStatistics statistics, List<ToolCall> toolCalls, List<Map<String, Object>> outputItems) {
|
||||
|
||||
public Completion {
|
||||
response = response == null ? "" : response;
|
||||
@@ -12,10 +26,22 @@ public final record Completion(String response, String reasoning, CompletionFini
|
||||
finishReason = finishReason == null ? CompletionFinishReason.DONE : finishReason;
|
||||
statistics = statistics == null ? new CompletionStatistics(0, Duration.ZERO, 0, 0, 0, 0, BigDecimal.ZERO, false) : statistics;
|
||||
toolCalls = toolCalls == null ? List.of() : List.copyOf(toolCalls);
|
||||
outputItems = outputItems == null ? List.of() : outputItems.stream().map(outputItem -> outputItem == null ? Map.<String, Object>of() : Map.copyOf(outputItem)).toList();
|
||||
}
|
||||
|
||||
/**
|
||||
* Creates a completion without provider-specific output items.
|
||||
*/
|
||||
public Completion(final String response, final String reasoning, final CompletionFinishReason finishReason, final CompletionStatistics statistics, final List<ToolCall> toolCalls) {
|
||||
this(response, reasoning, finishReason, statistics, toolCalls, List.of());
|
||||
}
|
||||
|
||||
/**
|
||||
* Creates a plain text completion without tool calls or provider-specific
|
||||
* output items.
|
||||
*/
|
||||
public Completion(final String response, final String reasoning, final CompletionFinishReason finishReason, final CompletionStatistics statistics) {
|
||||
this(response, reasoning, finishReason, statistics, List.of());
|
||||
this(response, reasoning, finishReason, statistics, List.of(), List.of());
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
@@ -3,7 +3,11 @@ package de.extio.lmlib.client;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Collections;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
|
||||
/**
|
||||
* Mutable conversation history shared across lm-lib clients and agents.
|
||||
*/
|
||||
public final class Conversation {
|
||||
|
||||
public static Conversation create(final String system, final String user) {
|
||||
@@ -57,21 +61,53 @@ public final class Conversation {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Appends the assistant tool-call round and the corresponding tool results.
|
||||
* <p>
|
||||
* When the completion contains structured output items, they are preserved on
|
||||
* the assistant turn so Responses-style stateless replay can round-trip the
|
||||
* provider payload untouched.
|
||||
*
|
||||
* @param completion the completion that requested tool execution
|
||||
* @param toolResults the tool outputs to append after the assistant turn
|
||||
*/
|
||||
public void appendToolCallRound(final Completion completion, final List<ToolResult> toolResults) {
|
||||
if (completion == null) {
|
||||
throw new IllegalArgumentException("completion must not be null");
|
||||
}
|
||||
final var assistantText = completion.toolCalls().isEmpty() ? completion.response() : "";
|
||||
this.appendToolCallRound(assistantText, completion.toolCalls(), toolResults);
|
||||
this.appendToolCallRound(assistantText, completion.toolCalls(), toolResults, null, completion.outputItems());
|
||||
}
|
||||
|
||||
/**
|
||||
* Appends an assistant tool-call round using only normalized lm-lib fields.
|
||||
*
|
||||
* @param assistantText assistant text to append before the tool results
|
||||
* @param toolCalls tool calls returned by the provider
|
||||
* @param toolResults tool outputs to append after the assistant turn
|
||||
*/
|
||||
public void appendToolCallRound(final String assistantText, final List<ToolCall> toolCalls, final List<ToolResult> toolResults) {
|
||||
this.addTurn(new Turn(TurnType.ASSISTANT, assistantText == null ? "" : assistantText, toolCalls, null));
|
||||
this.appendToolCallRound(assistantText, toolCalls, toolResults, null, List.of());
|
||||
}
|
||||
|
||||
/**
|
||||
* Appends an assistant tool-call round including provider-specific replay
|
||||
* metadata.
|
||||
*
|
||||
* @param assistantText assistant text to append before the tool results
|
||||
* @param toolCalls tool calls returned by the provider
|
||||
* @param toolResults tool outputs to append after the assistant turn
|
||||
* @param phase optional assistant phase metadata for providers that emit it
|
||||
* @param outputItems provider-specific structured output items that should be
|
||||
* preserved for follow-up requests
|
||||
*/
|
||||
public void appendToolCallRound(final String assistantText, final List<ToolCall> toolCalls, final List<ToolResult> toolResults, final String phase, final List<Map<String, Object>> outputItems) {
|
||||
this.addTurn(new Turn(TurnType.ASSISTANT, assistantText == null ? "" : assistantText, toolCalls, null, phase, outputItems));
|
||||
if (toolResults == null || toolResults.isEmpty()) {
|
||||
return;
|
||||
}
|
||||
for (final var toolResult : toolResults) {
|
||||
this.addTurn(new Turn(TurnType.TOOL, toolResult.text(), null, toolResult.toolCallId()));
|
||||
this.addTurn(new Turn(TurnType.TOOL, toolResult.text(), null, toolResult.toolCallId(), null, List.of()));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -103,29 +139,75 @@ public final class Conversation {
|
||||
return builder.toString();
|
||||
}
|
||||
|
||||
/**
|
||||
* Optional metadata associated with the whole conversation, for example a user
|
||||
* id or an upstream conversation id.
|
||||
*/
|
||||
public String getMetadata() {
|
||||
return metadata;
|
||||
}
|
||||
|
||||
/**
|
||||
* Sets optional conversation-level metadata.
|
||||
*/
|
||||
public void setMetadata(final String metadata) {
|
||||
this.metadata = metadata;
|
||||
}
|
||||
|
||||
/**
|
||||
* Turn roles supported by lm-lib conversations.
|
||||
*/
|
||||
public enum TurnType {
|
||||
SYSTEM, USER, ASSISTANT, TOOL
|
||||
}
|
||||
|
||||
public record Turn(TurnType type, String text, List<ToolCall> toolCalls, String toolCallId) {
|
||||
/**
|
||||
* Single conversation turn.
|
||||
*
|
||||
* @param type logical role of the turn
|
||||
* @param text normalized text payload for the turn
|
||||
* @param toolCalls tool calls attached to assistant turns
|
||||
* @param toolCallId tool-call id referenced by tool result turns
|
||||
* @param phase optional assistant phase metadata used by Responses-style replay
|
||||
* @param outputItems provider-specific structured output items that should be
|
||||
* replayed verbatim in later requests
|
||||
*/
|
||||
public record Turn(TurnType type, String text, List<ToolCall> toolCalls, String toolCallId, String phase, List<Map<String, Object>> outputItems) {
|
||||
|
||||
public Turn {
|
||||
toolCalls = toolCalls == null ? List.of() : List.copyOf(toolCalls);
|
||||
outputItems = outputItems == null ? List.of() : outputItems.stream().map(outputItem -> outputItem == null ? Map.<String, Object>of() : Map.copyOf(outputItem)).toList();
|
||||
}
|
||||
|
||||
/**
|
||||
* Creates a turn without provider-specific replay metadata.
|
||||
*/
|
||||
public Turn(final TurnType type, final String text, final List<ToolCall> toolCalls, final String toolCallId) {
|
||||
this(type, text, toolCalls, toolCallId, null, List.of());
|
||||
}
|
||||
|
||||
/**
|
||||
* Creates a plain turn with an explicit assistant phase and no tool-call or
|
||||
* replay payload.
|
||||
*/
|
||||
public Turn(final TurnType type, final String text, final String phase) {
|
||||
this(type, text, List.of(), null, phase, List.of());
|
||||
}
|
||||
|
||||
/**
|
||||
* Creates a plain turn without tool-call or replay metadata.
|
||||
*/
|
||||
public Turn(final TurnType type, final String text) {
|
||||
this(type, text, List.of(), null);
|
||||
this(type, text, List.of(), null, null, List.of());
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Result payload returned by a caller after executing a tool.
|
||||
*
|
||||
* @param text serialized tool output
|
||||
* @param toolCallId tool-call id the output belongs to
|
||||
*/
|
||||
public record ToolResult(String text, String toolCallId) {
|
||||
|
||||
public ToolResult {
|
||||
|
||||
@@ -92,12 +92,12 @@ public class CachedClient implements Client {
|
||||
|
||||
final var cachedCompletion = skipCache ? null : this.cacheRepository.get(key);
|
||||
if (cachedCompletion != null) {
|
||||
return new Completion(cachedCompletion.response(), cachedCompletion.reasoning(), cachedCompletion.finishReason(), new CompletionStatistics(0, Duration.ZERO, cachedCompletion.inTokens(), cachedCompletion.cachedInTokens(), cachedCompletion.outTokens(), cachedCompletion.reasoningOutTokens(), BigDecimal.ZERO, true), cachedCompletion.toolCalls());
|
||||
return new Completion(cachedCompletion.response(), cachedCompletion.reasoning(), cachedCompletion.finishReason(), new CompletionStatistics(0, Duration.ZERO, cachedCompletion.inTokens(), cachedCompletion.cachedInTokens(), cachedCompletion.outTokens(), cachedCompletion.reasoningOutTokens(), BigDecimal.ZERO, true), cachedCompletion.toolCalls(), cachedCompletion.outputItems());
|
||||
}
|
||||
|
||||
final var completion = supplier.get();
|
||||
if (completion != null) {
|
||||
this.cacheRepository.put(key, new CachedCompletion(completion.response(), completion.reasoning(), completion.finishReason(), completion.toolCalls(), completion.statistics().inTokens(), completion.statistics().cachedInTokens(), completion.statistics().outTokens(), completion.statistics().reasoningOutTokens(), OffsetDateTime.now()));
|
||||
this.cacheRepository.put(key, new CachedCompletion(completion.response(), completion.reasoning(), completion.finishReason(), completion.toolCalls(), completion.outputItems(), completion.statistics().inTokens(), completion.statistics().cachedInTokens(), completion.statistics().outTokens(), completion.statistics().reasoningOutTokens(), OffsetDateTime.now()));
|
||||
}
|
||||
return completion;
|
||||
}
|
||||
|
||||
@@ -2,10 +2,19 @@ package de.extio.lmlib.client.cached;
|
||||
|
||||
import java.time.OffsetDateTime;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
|
||||
import de.extio.lmlib.client.CompletionFinishReason;
|
||||
import de.extio.lmlib.client.ToolCall;
|
||||
|
||||
public record CachedCompletion(String response, String reasoning, CompletionFinishReason finishReason, List<ToolCall> toolCalls, long inTokens, long cachedInTokens, long outTokens, long reasoningOutTokens, OffsetDateTime created) {
|
||||
|
||||
/**
|
||||
* Serializable cache entry for normalized lm-lib completions.
|
||||
*/
|
||||
public record CachedCompletion(String response, String reasoning, CompletionFinishReason finishReason, List<ToolCall> toolCalls, List<Map<String, Object>> outputItems, long inTokens, long cachedInTokens, long outTokens, long reasoningOutTokens, OffsetDateTime created) {
|
||||
|
||||
public CachedCompletion {
|
||||
toolCalls = toolCalls == null ? List.of() : List.copyOf(toolCalls);
|
||||
outputItems = outputItems == null ? List.of() : outputItems.stream().map(outputItem -> outputItem == null ? Map.<String, Object>of() : Map.copyOf(outputItem)).toList();
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
@@ -0,0 +1,388 @@
|
||||
package de.extio.lmlib.client.oai.responses;
|
||||
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
|
||||
import com.fasterxml.jackson.annotation.JsonValue;
|
||||
|
||||
import de.extio.lmlib.profile.ModelProfile;
|
||||
|
||||
/**
|
||||
* Optional request-shaping hook for the standalone OpenAI Responses client.
|
||||
* <p>
|
||||
* Applications can register this as a Spring bean to opt into Responses-specific
|
||||
* request fields without affecting the chat-completions clients.
|
||||
*/
|
||||
public interface OpenAiResponsesApiDialect {
|
||||
|
||||
/**
|
||||
* Include value that asks the Responses API to return encrypted reasoning
|
||||
* payloads. This is required when reasoning items should survive stateless
|
||||
* tool-call loops.
|
||||
*/
|
||||
String INCLUDE_REASONING_ENCRYPTED_CONTENT = "reasoning.encrypted_content";
|
||||
|
||||
/**
|
||||
* Controls whether OpenAI should persist the server-side response state for
|
||||
* later retrieval or follow-up requests.
|
||||
* <p>
|
||||
* lm-lib defaults this to {@code false} because the Responses client replays the
|
||||
* full conversation statelessly.
|
||||
*
|
||||
* @param modelProfile the resolved model profile for the request
|
||||
* @return {@code true} to store responses server-side, otherwise {@code false}
|
||||
*/
|
||||
default boolean store(final ModelProfile modelProfile) {
|
||||
return false;
|
||||
}
|
||||
|
||||
/**
|
||||
* Controls the optional {@code background} request flag.
|
||||
* <p>
|
||||
* When enabled, the provider may continue processing the response after the
|
||||
* initial request has been accepted.
|
||||
*
|
||||
* @param modelProfile the resolved model profile for the request
|
||||
* @return {@code true} or {@code false} to send the flag, or {@code null} to
|
||||
* omit it
|
||||
*/
|
||||
default Boolean background(final ModelProfile modelProfile) {
|
||||
return null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Additional output sections to request via the Responses {@code include}
|
||||
* field.
|
||||
* <p>
|
||||
* Use this for optional payloads such as encrypted reasoning, tool results, or
|
||||
* output logprobs.
|
||||
*
|
||||
* @param modelProfile the resolved model profile for the request
|
||||
* @return includable field names to send with the request
|
||||
*/
|
||||
default List<String> include(final ModelProfile modelProfile) {
|
||||
return List.of();
|
||||
}
|
||||
|
||||
/**
|
||||
* Caps the number of built-in tool calls the Responses API may execute during a
|
||||
* single request.
|
||||
*
|
||||
* @param modelProfile the resolved model profile for the request
|
||||
* @return the maximum number of built-in tool calls, or {@code null} to omit
|
||||
* the field
|
||||
*/
|
||||
default Integer maxToolCalls(final ModelProfile modelProfile) {
|
||||
return null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Supplies request metadata that OpenAI stores alongside the generated
|
||||
* response.
|
||||
* <p>
|
||||
* This is useful for provider-side filtering, trace correlation, or dashboard
|
||||
* inspection.
|
||||
*
|
||||
* @param modelProfile the resolved model profile for the request
|
||||
* @return metadata entries to attach to the request
|
||||
*/
|
||||
default Map<String, String> metadata(final ModelProfile modelProfile) {
|
||||
return Map.of();
|
||||
}
|
||||
|
||||
/**
|
||||
* Sets the optional {@code prompt_cache_key} used by OpenAI prompt caching.
|
||||
*
|
||||
* @param modelProfile the resolved model profile for the request
|
||||
* @return the cache key, or {@code null} to omit it
|
||||
*/
|
||||
default String promptCacheKey(final ModelProfile modelProfile) {
|
||||
return null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Selects how long OpenAI may retain cached prompt prefixes for this request.
|
||||
*
|
||||
* @param modelProfile the resolved model profile for the request
|
||||
* @return the prompt-cache retention policy, or {@code null} to omit it
|
||||
*/
|
||||
default PromptCacheRetention promptCacheRetention(final ModelProfile modelProfile) {
|
||||
return null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Configures the Responses {@code reasoning} object for reasoning-capable
|
||||
* models.
|
||||
* <p>
|
||||
* This controls effort level and optional summary generation settings.
|
||||
*
|
||||
* @param modelProfile the resolved model profile for the request
|
||||
* @return the reasoning configuration to send, or {@code null} to omit it
|
||||
*/
|
||||
default ResponsesReasoning reasoning(final ModelProfile modelProfile) {
|
||||
return new ResponsesReasoning(ReasoningEffort.MEDIUM, null, null);
|
||||
}
|
||||
|
||||
/**
|
||||
* Supplies the optional {@code safety_identifier} field.
|
||||
* <p>
|
||||
* This should be a stable, non-identifying user or tenant token that helps the
|
||||
* provider detect abuse patterns.
|
||||
*
|
||||
* @param modelProfile the resolved model profile for the request
|
||||
* @return the safety identifier, or {@code null} to omit it
|
||||
*/
|
||||
default String safetyIdentifier(final ModelProfile modelProfile) {
|
||||
return null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Selects the OpenAI processing tier for the request when the provider supports
|
||||
* tiered serving.
|
||||
*
|
||||
* @param modelProfile the resolved model profile for the request
|
||||
* @return the requested service tier, or {@code null} to omit it
|
||||
*/
|
||||
default ServiceTier serviceTier(final ModelProfile modelProfile) {
|
||||
return null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Configures the optional Responses {@code stream_options} object.
|
||||
*
|
||||
* @param modelProfile the resolved model profile for the request
|
||||
* @return stream options, or {@code null} to omit them
|
||||
*/
|
||||
default ResponsesStreamOptions streamOptions(final ModelProfile modelProfile) {
|
||||
return null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Configures the optional Responses {@code text} object.
|
||||
* <p>
|
||||
* This can be used to influence text-only output shaping such as verbosity.
|
||||
*
|
||||
* @param modelProfile the resolved model profile for the request
|
||||
* @return text output options, or {@code null} to omit them
|
||||
*/
|
||||
default ResponsesText text(final ModelProfile modelProfile) {
|
||||
return null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Requests token-level log probability data for text outputs.
|
||||
*
|
||||
* @param modelProfile the resolved model profile for the request
|
||||
* @return the number of top logprobs to request, or {@code null} to omit the
|
||||
* field
|
||||
*/
|
||||
default Integer topLogprobs(final ModelProfile modelProfile) {
|
||||
return null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Selects how the provider should react when the input exceeds the model's
|
||||
* context window.
|
||||
*
|
||||
* @param modelProfile the resolved model profile for the request
|
||||
* @return the truncation mode, or {@code null} to omit it
|
||||
*/
|
||||
default Truncation truncation(final ModelProfile modelProfile) {
|
||||
return null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Request payload for the Responses {@code reasoning} object.
|
||||
*
|
||||
* @param effort the reasoning budget requested from the model
|
||||
* @param generateSummary optional instruction for whether and how the provider
|
||||
* should generate reasoning summaries
|
||||
* @param summary optional summary detail level for reasoning output
|
||||
*/
|
||||
record ResponsesReasoning(ReasoningEffort effort, ReasoningSummaryGenerateMode generateSummary, ReasoningSummary summary) {
|
||||
}
|
||||
|
||||
/**
|
||||
* Request payload for the Responses {@code stream_options} object.
|
||||
*
|
||||
* @param includeObfuscation whether streaming responses should include the
|
||||
* provider's obfuscation metadata when available
|
||||
*/
|
||||
record ResponsesStreamOptions(Boolean includeObfuscation) {
|
||||
}
|
||||
|
||||
/**
|
||||
* Request payload for the Responses {@code text} object.
|
||||
*
|
||||
* @param verbosity desired verbosity level for text output
|
||||
*/
|
||||
record ResponsesText(TextVerbosity verbosity) {
|
||||
}
|
||||
|
||||
/**
|
||||
* Retention policy for prompt caching.
|
||||
*/
|
||||
enum PromptCacheRetention {
|
||||
/** Retain cached prompt prefixes in volatile memory only. */
|
||||
IN_MEMORY("in_memory"),
|
||||
/** Retain cached prompt prefixes for up to 24 hours. */
|
||||
DAY_24H("24h");
|
||||
|
||||
private final String wireValue;
|
||||
|
||||
PromptCacheRetention(final String wireValue) {
|
||||
this.wireValue = wireValue;
|
||||
}
|
||||
|
||||
@JsonValue
|
||||
public String getWireValue() {
|
||||
return this.wireValue;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Reasoning effort levels supported by the Responses API.
|
||||
*/
|
||||
enum ReasoningEffort {
|
||||
/** Disable extra reasoning tokens when the model supports that mode. */
|
||||
NONE("none"),
|
||||
/** Request the smallest possible reasoning budget. */
|
||||
MINIMAL("minimal"),
|
||||
/** Request a low reasoning budget. */
|
||||
LOW("low"),
|
||||
/** Request the default medium reasoning budget. */
|
||||
MEDIUM("medium"),
|
||||
/** Request a high reasoning budget. */
|
||||
HIGH("high"),
|
||||
/** Request the highest extended reasoning budget. */
|
||||
XHIGH("xhigh");
|
||||
|
||||
private final String wireValue;
|
||||
|
||||
ReasoningEffort(final String wireValue) {
|
||||
this.wireValue = wireValue;
|
||||
}
|
||||
|
||||
@JsonValue
|
||||
public String getWireValue() {
|
||||
return this.wireValue;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Summary detail levels for returned reasoning summaries.
|
||||
*/
|
||||
enum ReasoningSummary {
|
||||
/** Let the provider pick the summary detail level automatically. */
|
||||
AUTO("auto"),
|
||||
/** Request a short reasoning summary. */
|
||||
CONCISE("concise"),
|
||||
/** Request a more detailed reasoning summary. */
|
||||
DETAILED("detailed");
|
||||
|
||||
private final String wireValue;
|
||||
|
||||
ReasoningSummary(final String wireValue) {
|
||||
this.wireValue = wireValue;
|
||||
}
|
||||
|
||||
@JsonValue
|
||||
public String getWireValue() {
|
||||
return this.wireValue;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Modes for asking the provider to generate reasoning summaries.
|
||||
*/
|
||||
enum ReasoningSummaryGenerateMode {
|
||||
/** Let the provider decide whether and how to generate a summary. */
|
||||
AUTO("auto"),
|
||||
/** Generate a short reasoning summary. */
|
||||
CONCISE("concise"),
|
||||
/** Generate a detailed reasoning summary. */
|
||||
DETAILED("detailed"),
|
||||
/** Do not generate a reasoning summary. */
|
||||
NONE("none");
|
||||
|
||||
private final String wireValue;
|
||||
|
||||
ReasoningSummaryGenerateMode(final String wireValue) {
|
||||
this.wireValue = wireValue;
|
||||
}
|
||||
|
||||
@JsonValue
|
||||
public String getWireValue() {
|
||||
return this.wireValue;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* OpenAI processing tiers supported by the Responses API.
|
||||
*/
|
||||
enum ServiceTier {
|
||||
/** Let the provider choose the service tier automatically. */
|
||||
AUTO("auto"),
|
||||
/** Use the standard service tier. */
|
||||
DEFAULT("default"),
|
||||
/** Use flex processing when available. */
|
||||
FLEX("flex"),
|
||||
/** Use priority processing when available. */
|
||||
PRIORITY("priority");
|
||||
|
||||
private final String wireValue;
|
||||
|
||||
ServiceTier(final String wireValue) {
|
||||
this.wireValue = wireValue;
|
||||
}
|
||||
|
||||
@JsonValue
|
||||
public String getWireValue() {
|
||||
return this.wireValue;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Verbosity levels for text output.
|
||||
*/
|
||||
enum TextVerbosity {
|
||||
/** Prefer shorter text outputs. */
|
||||
LOW("low"),
|
||||
/** Prefer balanced text outputs. */
|
||||
MEDIUM("medium"),
|
||||
/** Prefer more detailed text outputs. */
|
||||
HIGH("high");
|
||||
|
||||
private final String wireValue;
|
||||
|
||||
TextVerbosity(final String wireValue) {
|
||||
this.wireValue = wireValue;
|
||||
}
|
||||
|
||||
@JsonValue
|
||||
public String getWireValue() {
|
||||
return this.wireValue;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Input truncation strategies for oversized requests.
|
||||
*/
|
||||
enum Truncation {
|
||||
/** Allow the provider to drop older context automatically. */
|
||||
AUTO("auto"),
|
||||
/** Reject oversized requests instead of truncating them. */
|
||||
DISABLED("disabled");
|
||||
|
||||
private final String wireValue;
|
||||
|
||||
Truncation(final String wireValue) {
|
||||
this.wireValue = wireValue;
|
||||
}
|
||||
|
||||
@JsonValue
|
||||
public String getWireValue() {
|
||||
return this.wireValue;
|
||||
}
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -42,6 +42,6 @@ public record ModelProfile(
|
||||
}
|
||||
|
||||
public static enum ModelProvider {
|
||||
OAI_TEXT_COMPLETION, OAI_CHAT_COMPLETION, OPENROUTER, OLLAMA;
|
||||
OAI_TEXT_COMPLETION, OAI_CHAT_COMPLETION, OAI_RESPONSES, OPENROUTER, OLLAMA;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
package de.extio.lmlib.client.oai;
|
||||
package de.extio.lmlib.client.oai.completion;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.assertEquals;
|
||||
import static org.junit.jupiter.api.Assertions.assertFalse;
|
||||
@@ -0,0 +1,264 @@
|
||||
package de.extio.lmlib.client.oai.responses;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.assertEquals;
|
||||
import static org.junit.jupiter.api.Assertions.assertFalse;
|
||||
import static org.junit.jupiter.api.Assertions.assertTrue;
|
||||
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
|
||||
import org.junit.jupiter.api.Disabled;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.boot.SpringBootConfiguration;
|
||||
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;
|
||||
import org.springframework.boot.test.context.SpringBootTest;
|
||||
import org.springframework.boot.test.context.SpringBootTest.WebEnvironment;
|
||||
import org.springframework.context.annotation.Import;
|
||||
import org.springframework.test.context.TestPropertySource;
|
||||
|
||||
import de.extio.lmlib.client.ClientService;
|
||||
import de.extio.lmlib.client.CompletionFinishReason;
|
||||
import de.extio.lmlib.client.Conversation;
|
||||
import de.extio.lmlib.client.ToolCall;
|
||||
import de.extio.lmlib.client.ToolCallData;
|
||||
import de.extio.lmlib.client.ToolDefinition;
|
||||
import de.extio.lmlib.client.ToolParameters;
|
||||
import de.extio.lmlib.grader.Grader2;
|
||||
import de.extio.lmlib.profile.ModelProfile;
|
||||
import de.extio.lmlib.profile.ModelProfileService;
|
||||
|
||||
@Disabled("This test requires a Responses API compatible endpoint")
|
||||
@SpringBootTest(webEnvironment = WebEnvironment.NONE)
|
||||
@SpringBootConfiguration
|
||||
@EnableAutoConfiguration
|
||||
@Import(ResponsesApiDialectTestConfiguration.class)
|
||||
@TestPropertySource("classpath:/application-test.properties")
|
||||
public class ResponsesApiClientTest {
|
||||
|
||||
private static final Logger LOGGER = LoggerFactory.getLogger(ResponsesApiClientTest.class);
|
||||
|
||||
@Autowired
|
||||
private ResponsesApiClient responsesApiClient;
|
||||
|
||||
@Autowired
|
||||
private ModelProfileService modelProfileService;
|
||||
|
||||
@Autowired
|
||||
private ClientService clientService;
|
||||
|
||||
@Autowired
|
||||
private ResponsesApiDialectTestConfiguration.MutableOpenAiResponsesApiDialect openAiResponsesApiDialect;
|
||||
|
||||
@Test
|
||||
void listModels() throws Exception {
|
||||
final var modelProfile = this.modelProfile();
|
||||
final var modelNames = this.responsesApiClient.getModelNames(modelProfile, true);
|
||||
LOGGER.info("Models: {}", modelNames);
|
||||
assertFalse(modelNames.isEmpty());
|
||||
assertFalse(modelNames.getFirst().isBlank());
|
||||
assertEquals(ModelProfile.ModelProvider.OAI_RESPONSES, this.clientService.getClient(modelProfile).getModelProvider());
|
||||
}
|
||||
|
||||
@Test
|
||||
void conversation() throws Exception {
|
||||
this.openAiResponsesApiDialect.setReasoning(OpenAiResponsesApiDialect.ReasoningEffort.NONE);
|
||||
runCompletion();
|
||||
}
|
||||
|
||||
@Test
|
||||
void conversationWithReasoning() throws Exception {
|
||||
this.openAiResponsesApiDialect.setReasoning(OpenAiResponsesApiDialect.ReasoningEffort.MEDIUM);
|
||||
runCompletion();
|
||||
}
|
||||
|
||||
private void runCompletion() {
|
||||
final var modelProfile = this.modelProfile();
|
||||
final var completion = this.responsesApiClient.conversation(
|
||||
modelProfile,
|
||||
Conversation.create("You are a helpful assistant", "Say the color is green"),
|
||||
null,
|
||||
false);
|
||||
|
||||
LOGGER.info(completion.reasoning());
|
||||
LOGGER.info(completion.response());
|
||||
LOGGER.info(completion.statistics().toString());
|
||||
|
||||
assertTrue(Grader2.assessScoreBinary("Does the text mention the color green?", completion.response(), modelProfile, this.clientService));
|
||||
}
|
||||
|
||||
@Test
|
||||
void streamConversation() throws Exception {
|
||||
this.openAiResponsesApiDialect.setReasoning(OpenAiResponsesApiDialect.ReasoningEffort.NONE);
|
||||
runStreamedCompletion();
|
||||
}
|
||||
|
||||
@Test
|
||||
void streamConversationWithReasoning() throws Exception {
|
||||
this.openAiResponsesApiDialect.setReasoning(OpenAiResponsesApiDialect.ReasoningEffort.MEDIUM);
|
||||
runStreamedCompletion();
|
||||
}
|
||||
|
||||
private void runStreamedCompletion() {
|
||||
final var modelProfile = this.modelProfile();
|
||||
final var conversation = Conversation.create("You are a helpful assistant", "I would like to play a game of chess.");
|
||||
conversation.addTurn(new Conversation.Turn(Conversation.TurnType.ASSISTANT, "Which opening do you choose?"));
|
||||
conversation.addTurn(new Conversation.Turn(Conversation.TurnType.USER, "You choose, but explain the opening to me."));
|
||||
final var responseBuilder = new StringBuilder();
|
||||
final var reasoningBuilder = new StringBuilder();
|
||||
final var completion = this.responsesApiClient.streamConversation(modelProfile, conversation, chunk -> {
|
||||
if (chunk.reasoningContent() != null) {
|
||||
System.out.print(chunk.reasoningContent());
|
||||
reasoningBuilder.append(chunk.reasoningContent());
|
||||
}
|
||||
if (chunk.content() != null) {
|
||||
System.out.print(chunk.content());
|
||||
responseBuilder.append(chunk.content());
|
||||
}
|
||||
}, null, false);
|
||||
|
||||
LOGGER.info(completion.statistics().toString());
|
||||
|
||||
assertEquals(completion.response(), responseBuilder.toString());
|
||||
if (completion.reasoning() != null && !completion.reasoning().isBlank()) {
|
||||
assertEquals(completion.reasoning(), reasoningBuilder.toString());
|
||||
}
|
||||
assertTrue(Grader2.assessScoreBinary("Does the text explain a chess opening?", completion.response(), modelProfile, this.clientService));
|
||||
}
|
||||
|
||||
@Test
|
||||
void toolCallingConversation() throws Exception {
|
||||
this.openAiResponsesApiDialect.setReasoning(OpenAiResponsesApiDialect.ReasoningEffort.MEDIUM);
|
||||
final var modelProfile = this.modelProfile();
|
||||
final var weatherTool = new ToolDefinition(
|
||||
"get_weather",
|
||||
"Gets the current weather for a given location.",
|
||||
ToolParameters.create(Map.of("location", "City and country, for example Berlin, Germany")),
|
||||
true);
|
||||
final var conversation = Conversation.create(
|
||||
"You are a helpful assistant. Use the provided tools when the user asks for location information. After receiving tool results, answer the question from the tool output.",
|
||||
"What is the weather in Berlin, Germany? Use the tool.");
|
||||
final var completion = this.responsesApiClient.conversation(
|
||||
modelProfile,
|
||||
conversation,
|
||||
ToolCallData.required(List.of(weatherTool)).withParallelToolCalls(true),
|
||||
false);
|
||||
|
||||
LOGGER.info(completion.reasoning());
|
||||
LOGGER.info(completion.response());
|
||||
LOGGER.info("Tool calls: {}", completion.toolCalls());
|
||||
LOGGER.info(completion.statistics().toString());
|
||||
|
||||
assertEquals(CompletionFinishReason.TOOL_CALLS, completion.finishReason());
|
||||
assertFalse(completion.outputItems().isEmpty());
|
||||
assertFalse(completion.toolCalls().isEmpty());
|
||||
final var weatherToolCall = this.findToolCall(completion.toolCalls(), "get_weather");
|
||||
assertTrue(weatherToolCall.arguments().contains("Berlin"));
|
||||
|
||||
conversation.appendToolCallRound(completion, List.of(
|
||||
new Conversation.ToolResult(ToolParameters.create()
|
||||
.add("forecast", "Berlin is rainy and 21C")
|
||||
.json(), weatherToolCall.id())));
|
||||
|
||||
final var finalCompletion = this.responsesApiClient.conversation(modelProfile, conversation, ToolCallData.auto(List.of(weatherTool)), false);
|
||||
LOGGER.info(finalCompletion.reasoning());
|
||||
LOGGER.info(finalCompletion.response());
|
||||
LOGGER.info(finalCompletion.statistics().toString());
|
||||
assertEquals(CompletionFinishReason.DONE, finalCompletion.finishReason());
|
||||
assertTrue(Grader2.assessScoreBinary("Does the text say that Berlin is rainy and 21C?", finalCompletion.response(), modelProfile, this.clientService));
|
||||
}
|
||||
|
||||
@Test
|
||||
void streamToolCallingConversationBothRounds() throws Exception {
|
||||
this.openAiResponsesApiDialect.setReasoning(OpenAiResponsesApiDialect.ReasoningEffort.MEDIUM);
|
||||
final var modelProfile = this.modelProfile();
|
||||
final var weatherTool = new ToolDefinition(
|
||||
"get_weather",
|
||||
"Gets the current weather for a given location.",
|
||||
ToolParameters.create(Map.of("location", "City and country, for example Berlin, Germany")),
|
||||
true);
|
||||
final var conversation = Conversation.create(
|
||||
"You are a helpful assistant. Use the provided tools when the user asks for location information. After receiving tool results, answer the question from the tool output.",
|
||||
"What is the weather in Berlin, Germany? Use the tool.");
|
||||
final var firstRoundResponseBuilder = new StringBuilder();
|
||||
final var firstRoundReasoningBuilder = new StringBuilder();
|
||||
final var completion = this.responsesApiClient.streamConversation(
|
||||
modelProfile,
|
||||
conversation,
|
||||
chunk -> {
|
||||
if (chunk.reasoningContent() != null) {
|
||||
System.out.print(chunk.reasoningContent());
|
||||
firstRoundReasoningBuilder.append(chunk.reasoningContent());
|
||||
}
|
||||
if (chunk.content() != null) {
|
||||
System.out.print(chunk.content());
|
||||
firstRoundResponseBuilder.append(chunk.content());
|
||||
}
|
||||
},
|
||||
ToolCallData.required(List.of(weatherTool)).withParallelToolCalls(true),
|
||||
false);
|
||||
|
||||
LOGGER.info(completion.reasoning());
|
||||
LOGGER.info(completion.response());
|
||||
LOGGER.info("Tool calls: {}", completion.toolCalls());
|
||||
LOGGER.info(completion.statistics().toString());
|
||||
|
||||
assertEquals(CompletionFinishReason.TOOL_CALLS, completion.finishReason());
|
||||
assertEquals(completion.response(), firstRoundResponseBuilder.toString());
|
||||
if (completion.reasoning() != null && !completion.reasoning().isBlank()) {
|
||||
assertEquals(completion.reasoning(), firstRoundReasoningBuilder.toString());
|
||||
}
|
||||
assertFalse(completion.outputItems().isEmpty());
|
||||
assertFalse(completion.toolCalls().isEmpty());
|
||||
final var weatherToolCall = this.findToolCall(completion.toolCalls(), "get_weather");
|
||||
assertTrue(weatherToolCall.arguments().contains("Berlin"));
|
||||
|
||||
conversation.appendToolCallRound(completion, List.of(
|
||||
new Conversation.ToolResult(ToolParameters.create()
|
||||
.add("forecast", "Berlin is rainy and 21C")
|
||||
.json(), weatherToolCall.id())));
|
||||
|
||||
final var secondRoundResponseBuilder = new StringBuilder();
|
||||
final var secondRoundReasoningBuilder = new StringBuilder();
|
||||
final var finalCompletion = this.responsesApiClient.streamConversation(
|
||||
modelProfile,
|
||||
conversation,
|
||||
chunk -> {
|
||||
if (chunk.reasoningContent() != null) {
|
||||
System.out.print(chunk.reasoningContent());
|
||||
secondRoundReasoningBuilder.append(chunk.reasoningContent());
|
||||
}
|
||||
if (chunk.content() != null) {
|
||||
System.out.print(chunk.content());
|
||||
secondRoundResponseBuilder.append(chunk.content());
|
||||
}
|
||||
},
|
||||
ToolCallData.auto(List.of(weatherTool)),
|
||||
false);
|
||||
|
||||
LOGGER.info(finalCompletion.reasoning());
|
||||
LOGGER.info(finalCompletion.response());
|
||||
LOGGER.info(finalCompletion.statistics().toString());
|
||||
|
||||
assertEquals(CompletionFinishReason.DONE, finalCompletion.finishReason());
|
||||
assertEquals(finalCompletion.response(), secondRoundResponseBuilder.toString());
|
||||
if (finalCompletion.reasoning() != null && !finalCompletion.reasoning().isBlank()) {
|
||||
assertEquals(finalCompletion.reasoning(), secondRoundReasoningBuilder.toString());
|
||||
}
|
||||
assertTrue(Grader2.assessScoreBinary("Does the text say that Berlin is rainy and 21C?", finalCompletion.response(), modelProfile, this.clientService));
|
||||
}
|
||||
|
||||
private ModelProfile modelProfile() {
|
||||
return this.modelProfileService.getModelProfile("profile.model.responses");
|
||||
}
|
||||
|
||||
private ToolCall findToolCall(final List<ToolCall> toolCalls, final String toolName) {
|
||||
return toolCalls.stream()
|
||||
.filter(toolCall -> toolName.equals(toolCall.name()))
|
||||
.findFirst()
|
||||
.orElseThrow(() -> new IllegalStateException("Missing tool call: " + toolName));
|
||||
}
|
||||
|
||||
}
|
||||
@@ -0,0 +1,29 @@
|
||||
package de.extio.lmlib.client.oai.responses;
|
||||
|
||||
import org.springframework.boot.test.context.TestConfiguration;
|
||||
import org.springframework.context.annotation.Bean;
|
||||
|
||||
import de.extio.lmlib.profile.ModelProfile;
|
||||
|
||||
@TestConfiguration
|
||||
public class ResponsesApiDialectTestConfiguration {
|
||||
|
||||
@Bean
|
||||
MutableOpenAiResponsesApiDialect openAiResponsesApiDialect() {
|
||||
return new MutableOpenAiResponsesApiDialect();
|
||||
}
|
||||
|
||||
public static class MutableOpenAiResponsesApiDialect implements OpenAiResponsesApiDialect {
|
||||
|
||||
private volatile ReasoningEffort reasoning = ReasoningEffort.MEDIUM;
|
||||
|
||||
@Override
|
||||
public ResponsesReasoning reasoning(final ModelProfile modelProfile) {
|
||||
return this.reasoning == null ? null : new ResponsesReasoning(this.reasoning, null, null);
|
||||
}
|
||||
|
||||
public void setReasoning(final ReasoningEffort reasoning) {
|
||||
this.reasoning = reasoning;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -6,6 +6,7 @@ profile.model.cold=local-chatcompletion
|
||||
|
||||
profile.model.textcompletion=local-completion
|
||||
profile.model.chatcompletion=local-chatcompletion
|
||||
profile.model.responses=local-responses
|
||||
profile.model.ollama=local-ollama-chatcompletion
|
||||
profile.model.openrouter=local-openrouter-chatcompletion
|
||||
#profile.model.azure=llama3.3-70b-azure
|
||||
|
||||
16
src/test/resources/local-responses.properties
Normal file
16
src/test/resources/local-responses.properties
Normal file
@@ -0,0 +1,16 @@
|
||||
category=RESPONSES
|
||||
prompts=
|
||||
tokenizer=llamaServer
|
||||
tokenEncoding=cl100k_base
|
||||
maxTokens=2500
|
||||
maxContextLength=16000
|
||||
temperature=0.4
|
||||
topP=1.0
|
||||
modelProvider=OAI_RESPONSES
|
||||
modelName=
|
||||
url=http://localhost:5001
|
||||
apiKey=
|
||||
cost1MInTokens=0
|
||||
cost1MCachedInTokens=0
|
||||
cost1MOutTokens=0
|
||||
cost1MReasoningOutTokens=0
|
||||
Reference in New Issue
Block a user