diff --git a/README.md b/README.md index 42a4eb7..3996d3d 100644 --- a/README.md +++ b/README.md @@ -13,13 +13,13 @@ lm-lib is published under the Apache License 2.0. lm-lib is designed for applications that need more than single prompt-response calls. Its center of gravity is the agent framework: structured multi-step flows such as classification, planning, fan-out analysis, synthesis, grading, feedback loops, and other graph-based execution patterns. -It also supports direct completion use cases through a client abstraction layer. The built-in clients currently cover OpenAI-compatible chat and text completion endpoints, OpenRouter's OpenAI-compatible routing API, and Ollama's native chat API, and the abstraction remains open for additional provider implementations over time. Applications can also provide their own client implementations by implementing the relevant interfaces. +It also supports direct completion use cases through a client abstraction layer. The built-in clients currently cover OpenAI-compatible chat completion endpoints, the newer OpenAI-compatible Responses API, legacy text completion endpoints, OpenRouter's OpenAI-compatible routing API, and Ollama's native chat API, and the abstraction remains open for additional provider implementations over time. Applications can also provide their own client implementations by implementing the relevant interfaces. -The current built-in client set supports OpenAI-compatible chat completions, OpenRouter chat completions with cached metadata discovery, legacy text completions, and native Ollama chat completions, with model-specific prompt strategies for text completion models. +The current built-in client set supports OpenAI-compatible chat completions, OpenAI-compatible Responses, OpenRouter chat completions with cached metadata discovery, legacy text completions, and native Ollama chat completions, with model-specific prompt strategies for text completion models. ## Key Capabilities -- OpenAI-compatible chat and text completions, OpenRouter chat completions, plus native Ollama chat completions +- OpenAI-compatible chat completions, OpenAI-compatible Responses, legacy text completions, OpenRouter chat completions, plus native Ollama chat completions - Chat-completion tool calling with tool definitions, returned tool calls, and tool-result follow-up rounds - Streaming and non-streaming APIs - Separate reasoning capture for models that expose reasoning output @@ -242,7 +242,7 @@ Each profile file is a `.properties` resource on the classpath. | `maxContextLength` | Integer | Yes | Maximum total context length. | `16000`, `32768`, `128000` | | `temperature` | Double | Yes | Sampling temperature. Must be greater than zero. | `0.2`, `0.4`, `0.7` | | `topP` | Double | Yes | Nucleus sampling parameter. Must be greater than zero. | `1.0`, `0.9` | -| `modelProvider` | Enum | Yes | Completion client type. | `OAI_CHAT_COMPLETION`, `OAI_TEXT_COMPLETION`, `OPENROUTER`, `OLLAMA` | +| `modelProvider` | Enum | Yes | Completion client type. | `OAI_CHAT_COMPLETION`, `OAI_RESPONSES`, `OAI_TEXT_COMPLETION`, `OPENROUTER`, `OLLAMA` | | `modelName` | String | No | Model name or deployment. Can be empty for endpoints where the model should be auto-detected. | `gpt-4`, `Llama-3.3-70B-Instruct`, `` | | `url` | String | Yes | Base API URL. | `https://api.openai.com/v1`, `http://localhost:5001` | | `apiKey` | String | No | Bearer token or Spring placeholder. Can be empty for local servers. | `sk-...`, `${apikey}`, `` | @@ -300,6 +300,27 @@ cost1MInTokens=0 cost1MOutTokens=0 ``` +OpenAI Responses: + +```properties +category=M +prompts= +tokenizer=jTokkit +tokenEncoding=cl100k_base +maxTokens=2500 +maxContextLength=128000 +temperature=0.4 +topP=1.0 +modelProvider=OAI_RESPONSES +modelName=gpt-4.1 +url=https://api.openai.com +apiKey=${OPENAI_API_KEY:} +cost1MInTokens=0 +cost1MCachedInTokens=0 +cost1MOutTokens=0 +cost1MReasoningOutTokens=0 +``` + Local Ollama chat completion: ```properties @@ -329,6 +350,7 @@ The built-in provider implementations currently support these types: ```properties modelProvider=OAI_CHAT_COMPLETION +modelProvider=OAI_RESPONSES modelProvider=OAI_TEXT_COMPLETION modelProvider=OPENROUTER modelProvider=OLLAMA @@ -336,10 +358,12 @@ modelProvider=OLLAMA ### OpenAI-Compatible Providers -`OAI_CHAT_COMPLETION` and `OAI_TEXT_COMPLETION` target the OpenAI-style API surface. +`OAI_CHAT_COMPLETION`, `OAI_RESPONSES`, and `OAI_TEXT_COMPLETION` target the OpenAI-style API surface. Use `OAI_CHAT_COMPLETION` for cloud APIs such as OpenAI itself and for local or self-hosted servers that expose a compatible `/v1/chat/completions` interface. This is the preferred default because lm-lib can send the conversation directly, supports streaming, and exposes tool calls through the standard chat-completion flow. +Use `OAI_RESPONSES` when you want to keep the same lm-lib `Client` and `Conversation` API while targeting `/v1/responses` instead of `/v1/chat/completions`. This provider translates lm-lib conversation turns into Responses input items, supports streaming, supports function tool calling, and stays stateless by replaying the full lm-lib conversation on each request. + Use `OAI_TEXT_COMPLETION` when you need explicit prompt templating or must target instruct-style models that still depend on raw prompt construction rather than structured chat messages. This is mainly useful for older compatible backends, local servers configured around prompt templates, or if you need full control over the raw prompt that is sent to the LLM. OpenAI-compatible deployments commonly fall into two operational patterns: @@ -347,7 +371,9 @@ OpenAI-compatible deployments commonly fall into two operational patterns: - Cloud endpoints: hosted APIs with a fixed base URL such as `https://api.openai.com/v1`, normally with a Bearer API key. - Local or self-hosted endpoints: llama-server, vLLM-compatible gateways, or other OpenAI-style frontends exposed on an internal URL such as `http://localhost:5001`. -If you need to adapt request shaping for a specific OpenAI-compatible provider, you can optionally register an `OpenAiProviderDialect` bean. This allows applications to suppress or switch request fields that a provider does not support while keeping `ModelProfile` provider-agnostic and runtime-driven. +If you need to adapt request shaping for a specific chat-completions compatible provider, you can optionally register an `OpenAiProviderDialect` bean. This applies to the `OAI_CHAT_COMPLETION` and `OAI_TEXT_COMPLETION` providers. + +`OAI_RESPONSES` is implemented as a separate standalone client and uses its own optional `OpenAiResponsesApiDialect` hook. The default OpenAI dialect behavior is: @@ -356,12 +382,30 @@ The default OpenAI dialect behavior is: - `chatTokenLimitParameterMode(...) = ChatTokenLimitParameterMode.MAX_COMPLETION_TOKENS` - `reasoningEffort(...) = ReasoningEffort.MEDIUM` +The default Responses dialect behavior is: + +- `store(...) = false` +- `reasoning(...) = new ResponsesReasoning(ReasoningEffort.MEDIUM, null, null)` +- `include(...) = []`, with `reasoning.encrypted_content` added automatically for stateless reasoning replay +- `background(...) = null` +- `maxToolCalls(...) = null` +- `metadata(...) = {}` +- `promptCacheKey(...) = null` +- `promptCacheRetention(...) = null` +- `safetyIdentifier(...) = null` +- `serviceTier(...) = null` +- `text(...) = null` +- `topLogprobs(...) = null` +- `truncation(...) = null` +- `streamOptions(...) = null` + Notes: - `usage` is a llama-server style extension and is therefore disabled unless a dialect explicitly enables it. - `reasoning_effort` is sent as a top-level request field when reasoning is enabled. - `max_completion_tokens` is the default because it matches the current OpenAI recommendation. - Some older OpenAI-compatible servers still require `max_tokens`, which can be selected through the dialect. +- Responses reasoning requests are sent in the native `reasoning` object and the client preserves raw response output items so tool-call loops can replay reasoning items, encrypted reasoning payloads, and assistant `phase` values on the next stateless request. Example: @@ -392,6 +436,35 @@ OpenAiProviderDialect legacyOpenAiProviderDialect() { } ``` +Example Responses dialect: + +```java +@Bean +OpenAiResponsesApiDialect openAiResponsesApiDialect() { + return new OpenAiResponsesApiDialect() { + @Override + public ResponsesReasoning reasoning(final ModelProfile modelProfile) { + return new ResponsesReasoning(ReasoningEffort.HIGH, ReasoningSummaryGenerateMode.AUTO, ReasoningSummary.CONCISE); + } + + @Override + public List include(final ModelProfile modelProfile) { + return List.of(OpenAiResponsesApiDialect.INCLUDE_REASONING_ENCRYPTED_CONTENT); + } + + @Override + public ServiceTier serviceTier(final ModelProfile modelProfile) { + return ServiceTier.FLEX; + } + + @Override + public Truncation truncation(final ModelProfile modelProfile) { + return Truncation.AUTO; + } + }; +} +``` + ### OpenRouter Provider `OPENROUTER` targets OpenRouter's chat-completions API and its richer model discovery endpoints. diff --git a/src/main/java/de/extio/lmlib/agent/BaseAgent.java b/src/main/java/de/extio/lmlib/agent/BaseAgent.java index 5950392..89c198f 100644 --- a/src/main/java/de/extio/lmlib/agent/BaseAgent.java +++ b/src/main/java/de/extio/lmlib/agent/BaseAgent.java @@ -461,10 +461,10 @@ public interface BaseAgent { return; } final var completion = context.getLastCompletion(); - if ((completion.response() == null || completion.response().isEmpty()) && completion.toolCalls().isEmpty()) { + if (completion.response().isEmpty() && completion.toolCalls().isEmpty() && completion.outputItems().isEmpty()) { return; } - context.getConversation().addTurn(new Conversation.Turn(TurnType.ASSISTANT, completion.response() == null ? "" : completion.response(), completion.toolCalls(), null)); + context.getConversation().addTurn(new Conversation.Turn(TurnType.ASSISTANT, completion.response(), completion.toolCalls(), null, null, completion.outputItems())); } private void rememberToolCallData(final AgentContext context, final ToolCallData toolCallData) { diff --git a/src/main/java/de/extio/lmlib/client/ClientAutoConfiguration.java b/src/main/java/de/extio/lmlib/client/ClientAutoConfiguration.java index ea2d19e..865ee51 100644 --- a/src/main/java/de/extio/lmlib/client/ClientAutoConfiguration.java +++ b/src/main/java/de/extio/lmlib/client/ClientAutoConfiguration.java @@ -11,6 +11,7 @@ import de.extio.lmlib.client.openrouter.OpenRouterModelDiscovery; import de.extio.lmlib.client.ollama.OllamaClient; import de.extio.lmlib.client.oai.completion.chat.ChatCompletionClient; import de.extio.lmlib.client.oai.completion.text.TextCompletionClient; +import de.extio.lmlib.client.oai.responses.ResponsesApiClient; import de.extio.lmlib.profile.ProfileAutoConfiguration; import de.extio.lmlib.token.TokenizerAutoConfiguration; @@ -37,6 +38,12 @@ public class ClientAutoConfiguration { return new ChatCompletionClient(); } + @Bean + @ConditionalOnMissingBean + ResponsesApiClient responsesApiClient() { + return new ResponsesApiClient(); + } + @Bean @ConditionalOnMissingBean OpenRouterModelDiscovery openRouterModelDiscovery() { diff --git a/src/main/java/de/extio/lmlib/client/Completion.java b/src/main/java/de/extio/lmlib/client/Completion.java index 3b522f5..096a3c3 100755 --- a/src/main/java/de/extio/lmlib/client/Completion.java +++ b/src/main/java/de/extio/lmlib/client/Completion.java @@ -3,8 +3,22 @@ package de.extio.lmlib.client; import java.math.BigDecimal; import java.time.Duration; import java.util.List; +import java.util.Map; -public final record Completion(String response, String reasoning, CompletionFinishReason finishReason, CompletionStatistics statistics, List toolCalls) { +/** + * Normalized result returned by lm-lib clients. + * + * @param response the assistant-visible text output + * @param reasoning the separately exposed reasoning summary or reasoning text + * @param finishReason why generation stopped + * @param statistics token, duration, cache, and cost accounting for the request + * @param toolCalls tool calls returned by the provider when the response stopped + * for tool execution + * @param outputItems provider-specific structured output items that need to + * survive follow-up rounds, primarily for stateless OpenAI Responses tool + * loops + */ +public final record Completion(String response, String reasoning, CompletionFinishReason finishReason, CompletionStatistics statistics, List toolCalls, List> outputItems) { public Completion { response = response == null ? "" : response; @@ -12,10 +26,22 @@ public final record Completion(String response, String reasoning, CompletionFini finishReason = finishReason == null ? CompletionFinishReason.DONE : finishReason; statistics = statistics == null ? new CompletionStatistics(0, Duration.ZERO, 0, 0, 0, 0, BigDecimal.ZERO, false) : statistics; toolCalls = toolCalls == null ? List.of() : List.copyOf(toolCalls); + outputItems = outputItems == null ? List.of() : outputItems.stream().map(outputItem -> outputItem == null ? Map.of() : Map.copyOf(outputItem)).toList(); } + /** + * Creates a completion without provider-specific output items. + */ + public Completion(final String response, final String reasoning, final CompletionFinishReason finishReason, final CompletionStatistics statistics, final List toolCalls) { + this(response, reasoning, finishReason, statistics, toolCalls, List.of()); + } + + /** + * Creates a plain text completion without tool calls or provider-specific + * output items. + */ public Completion(final String response, final String reasoning, final CompletionFinishReason finishReason, final CompletionStatistics statistics) { - this(response, reasoning, finishReason, statistics, List.of()); + this(response, reasoning, finishReason, statistics, List.of(), List.of()); } } diff --git a/src/main/java/de/extio/lmlib/client/Conversation.java b/src/main/java/de/extio/lmlib/client/Conversation.java index 9deca58..714c912 100755 --- a/src/main/java/de/extio/lmlib/client/Conversation.java +++ b/src/main/java/de/extio/lmlib/client/Conversation.java @@ -3,7 +3,11 @@ package de.extio.lmlib.client; import java.util.ArrayList; import java.util.Collections; import java.util.List; +import java.util.Map; +/** + * Mutable conversation history shared across lm-lib clients and agents. + */ public final class Conversation { public static Conversation create(final String system, final String user) { @@ -57,21 +61,53 @@ public final class Conversation { } } + /** + * Appends the assistant tool-call round and the corresponding tool results. + *

+ * When the completion contains structured output items, they are preserved on + * the assistant turn so Responses-style stateless replay can round-trip the + * provider payload untouched. + * + * @param completion the completion that requested tool execution + * @param toolResults the tool outputs to append after the assistant turn + */ public void appendToolCallRound(final Completion completion, final List toolResults) { if (completion == null) { throw new IllegalArgumentException("completion must not be null"); } final var assistantText = completion.toolCalls().isEmpty() ? completion.response() : ""; - this.appendToolCallRound(assistantText, completion.toolCalls(), toolResults); + this.appendToolCallRound(assistantText, completion.toolCalls(), toolResults, null, completion.outputItems()); } + /** + * Appends an assistant tool-call round using only normalized lm-lib fields. + * + * @param assistantText assistant text to append before the tool results + * @param toolCalls tool calls returned by the provider + * @param toolResults tool outputs to append after the assistant turn + */ public void appendToolCallRound(final String assistantText, final List toolCalls, final List toolResults) { - this.addTurn(new Turn(TurnType.ASSISTANT, assistantText == null ? "" : assistantText, toolCalls, null)); + this.appendToolCallRound(assistantText, toolCalls, toolResults, null, List.of()); + } + + /** + * Appends an assistant tool-call round including provider-specific replay + * metadata. + * + * @param assistantText assistant text to append before the tool results + * @param toolCalls tool calls returned by the provider + * @param toolResults tool outputs to append after the assistant turn + * @param phase optional assistant phase metadata for providers that emit it + * @param outputItems provider-specific structured output items that should be + * preserved for follow-up requests + */ + public void appendToolCallRound(final String assistantText, final List toolCalls, final List toolResults, final String phase, final List> outputItems) { + this.addTurn(new Turn(TurnType.ASSISTANT, assistantText == null ? "" : assistantText, toolCalls, null, phase, outputItems)); if (toolResults == null || toolResults.isEmpty()) { return; } for (final var toolResult : toolResults) { - this.addTurn(new Turn(TurnType.TOOL, toolResult.text(), null, toolResult.toolCallId())); + this.addTurn(new Turn(TurnType.TOOL, toolResult.text(), null, toolResult.toolCallId(), null, List.of())); } } @@ -103,29 +139,75 @@ public final class Conversation { return builder.toString(); } + /** + * Optional metadata associated with the whole conversation, for example a user + * id or an upstream conversation id. + */ public String getMetadata() { return metadata; } + /** + * Sets optional conversation-level metadata. + */ public void setMetadata(final String metadata) { this.metadata = metadata; } + /** + * Turn roles supported by lm-lib conversations. + */ public enum TurnType { SYSTEM, USER, ASSISTANT, TOOL } - public record Turn(TurnType type, String text, List toolCalls, String toolCallId) { + /** + * Single conversation turn. + * + * @param type logical role of the turn + * @param text normalized text payload for the turn + * @param toolCalls tool calls attached to assistant turns + * @param toolCallId tool-call id referenced by tool result turns + * @param phase optional assistant phase metadata used by Responses-style replay + * @param outputItems provider-specific structured output items that should be + * replayed verbatim in later requests + */ + public record Turn(TurnType type, String text, List toolCalls, String toolCallId, String phase, List> outputItems) { public Turn { toolCalls = toolCalls == null ? List.of() : List.copyOf(toolCalls); + outputItems = outputItems == null ? List.of() : outputItems.stream().map(outputItem -> outputItem == null ? Map.of() : Map.copyOf(outputItem)).toList(); } + /** + * Creates a turn without provider-specific replay metadata. + */ + public Turn(final TurnType type, final String text, final List toolCalls, final String toolCallId) { + this(type, text, toolCalls, toolCallId, null, List.of()); + } + + /** + * Creates a plain turn with an explicit assistant phase and no tool-call or + * replay payload. + */ + public Turn(final TurnType type, final String text, final String phase) { + this(type, text, List.of(), null, phase, List.of()); + } + + /** + * Creates a plain turn without tool-call or replay metadata. + */ public Turn(final TurnType type, final String text) { - this(type, text, List.of(), null); + this(type, text, List.of(), null, null, List.of()); } } + /** + * Result payload returned by a caller after executing a tool. + * + * @param text serialized tool output + * @param toolCallId tool-call id the output belongs to + */ public record ToolResult(String text, String toolCallId) { public ToolResult { diff --git a/src/main/java/de/extio/lmlib/client/cached/CachedClient.java b/src/main/java/de/extio/lmlib/client/cached/CachedClient.java index eca2654..3a38615 100644 --- a/src/main/java/de/extio/lmlib/client/cached/CachedClient.java +++ b/src/main/java/de/extio/lmlib/client/cached/CachedClient.java @@ -92,12 +92,12 @@ public class CachedClient implements Client { final var cachedCompletion = skipCache ? null : this.cacheRepository.get(key); if (cachedCompletion != null) { - return new Completion(cachedCompletion.response(), cachedCompletion.reasoning(), cachedCompletion.finishReason(), new CompletionStatistics(0, Duration.ZERO, cachedCompletion.inTokens(), cachedCompletion.cachedInTokens(), cachedCompletion.outTokens(), cachedCompletion.reasoningOutTokens(), BigDecimal.ZERO, true), cachedCompletion.toolCalls()); + return new Completion(cachedCompletion.response(), cachedCompletion.reasoning(), cachedCompletion.finishReason(), new CompletionStatistics(0, Duration.ZERO, cachedCompletion.inTokens(), cachedCompletion.cachedInTokens(), cachedCompletion.outTokens(), cachedCompletion.reasoningOutTokens(), BigDecimal.ZERO, true), cachedCompletion.toolCalls(), cachedCompletion.outputItems()); } final var completion = supplier.get(); if (completion != null) { - this.cacheRepository.put(key, new CachedCompletion(completion.response(), completion.reasoning(), completion.finishReason(), completion.toolCalls(), completion.statistics().inTokens(), completion.statistics().cachedInTokens(), completion.statistics().outTokens(), completion.statistics().reasoningOutTokens(), OffsetDateTime.now())); + this.cacheRepository.put(key, new CachedCompletion(completion.response(), completion.reasoning(), completion.finishReason(), completion.toolCalls(), completion.outputItems(), completion.statistics().inTokens(), completion.statistics().cachedInTokens(), completion.statistics().outTokens(), completion.statistics().reasoningOutTokens(), OffsetDateTime.now())); } return completion; } diff --git a/src/main/java/de/extio/lmlib/client/cached/CachedCompletion.java b/src/main/java/de/extio/lmlib/client/cached/CachedCompletion.java index be5466b..bfa593c 100644 --- a/src/main/java/de/extio/lmlib/client/cached/CachedCompletion.java +++ b/src/main/java/de/extio/lmlib/client/cached/CachedCompletion.java @@ -2,10 +2,19 @@ package de.extio.lmlib.client.cached; import java.time.OffsetDateTime; import java.util.List; +import java.util.Map; import de.extio.lmlib.client.CompletionFinishReason; import de.extio.lmlib.client.ToolCall; -public record CachedCompletion(String response, String reasoning, CompletionFinishReason finishReason, List toolCalls, long inTokens, long cachedInTokens, long outTokens, long reasoningOutTokens, OffsetDateTime created) { - +/** + * Serializable cache entry for normalized lm-lib completions. + */ +public record CachedCompletion(String response, String reasoning, CompletionFinishReason finishReason, List toolCalls, List> outputItems, long inTokens, long cachedInTokens, long outTokens, long reasoningOutTokens, OffsetDateTime created) { + + public CachedCompletion { + toolCalls = toolCalls == null ? List.of() : List.copyOf(toolCalls); + outputItems = outputItems == null ? List.of() : outputItems.stream().map(outputItem -> outputItem == null ? Map.of() : Map.copyOf(outputItem)).toList(); + } + } diff --git a/src/main/java/de/extio/lmlib/client/oai/responses/OpenAiResponsesApiDialect.java b/src/main/java/de/extio/lmlib/client/oai/responses/OpenAiResponsesApiDialect.java new file mode 100644 index 0000000..cdd037d --- /dev/null +++ b/src/main/java/de/extio/lmlib/client/oai/responses/OpenAiResponsesApiDialect.java @@ -0,0 +1,388 @@ +package de.extio.lmlib.client.oai.responses; + +import java.util.List; +import java.util.Map; + +import com.fasterxml.jackson.annotation.JsonValue; + +import de.extio.lmlib.profile.ModelProfile; + +/** + * Optional request-shaping hook for the standalone OpenAI Responses client. + *

+ * Applications can register this as a Spring bean to opt into Responses-specific + * request fields without affecting the chat-completions clients. + */ +public interface OpenAiResponsesApiDialect { + + /** + * Include value that asks the Responses API to return encrypted reasoning + * payloads. This is required when reasoning items should survive stateless + * tool-call loops. + */ + String INCLUDE_REASONING_ENCRYPTED_CONTENT = "reasoning.encrypted_content"; + + /** + * Controls whether OpenAI should persist the server-side response state for + * later retrieval or follow-up requests. + *

+ * lm-lib defaults this to {@code false} because the Responses client replays the + * full conversation statelessly. + * + * @param modelProfile the resolved model profile for the request + * @return {@code true} to store responses server-side, otherwise {@code false} + */ + default boolean store(final ModelProfile modelProfile) { + return false; + } + + /** + * Controls the optional {@code background} request flag. + *

+ * When enabled, the provider may continue processing the response after the + * initial request has been accepted. + * + * @param modelProfile the resolved model profile for the request + * @return {@code true} or {@code false} to send the flag, or {@code null} to + * omit it + */ + default Boolean background(final ModelProfile modelProfile) { + return null; + } + + /** + * Additional output sections to request via the Responses {@code include} + * field. + *

+ * Use this for optional payloads such as encrypted reasoning, tool results, or + * output logprobs. + * + * @param modelProfile the resolved model profile for the request + * @return includable field names to send with the request + */ + default List include(final ModelProfile modelProfile) { + return List.of(); + } + + /** + * Caps the number of built-in tool calls the Responses API may execute during a + * single request. + * + * @param modelProfile the resolved model profile for the request + * @return the maximum number of built-in tool calls, or {@code null} to omit + * the field + */ + default Integer maxToolCalls(final ModelProfile modelProfile) { + return null; + } + + /** + * Supplies request metadata that OpenAI stores alongside the generated + * response. + *

+ * This is useful for provider-side filtering, trace correlation, or dashboard + * inspection. + * + * @param modelProfile the resolved model profile for the request + * @return metadata entries to attach to the request + */ + default Map metadata(final ModelProfile modelProfile) { + return Map.of(); + } + + /** + * Sets the optional {@code prompt_cache_key} used by OpenAI prompt caching. + * + * @param modelProfile the resolved model profile for the request + * @return the cache key, or {@code null} to omit it + */ + default String promptCacheKey(final ModelProfile modelProfile) { + return null; + } + + /** + * Selects how long OpenAI may retain cached prompt prefixes for this request. + * + * @param modelProfile the resolved model profile for the request + * @return the prompt-cache retention policy, or {@code null} to omit it + */ + default PromptCacheRetention promptCacheRetention(final ModelProfile modelProfile) { + return null; + } + + /** + * Configures the Responses {@code reasoning} object for reasoning-capable + * models. + *

+ * This controls effort level and optional summary generation settings. + * + * @param modelProfile the resolved model profile for the request + * @return the reasoning configuration to send, or {@code null} to omit it + */ + default ResponsesReasoning reasoning(final ModelProfile modelProfile) { + return new ResponsesReasoning(ReasoningEffort.MEDIUM, null, null); + } + + /** + * Supplies the optional {@code safety_identifier} field. + *

+ * This should be a stable, non-identifying user or tenant token that helps the + * provider detect abuse patterns. + * + * @param modelProfile the resolved model profile for the request + * @return the safety identifier, or {@code null} to omit it + */ + default String safetyIdentifier(final ModelProfile modelProfile) { + return null; + } + + /** + * Selects the OpenAI processing tier for the request when the provider supports + * tiered serving. + * + * @param modelProfile the resolved model profile for the request + * @return the requested service tier, or {@code null} to omit it + */ + default ServiceTier serviceTier(final ModelProfile modelProfile) { + return null; + } + + /** + * Configures the optional Responses {@code stream_options} object. + * + * @param modelProfile the resolved model profile for the request + * @return stream options, or {@code null} to omit them + */ + default ResponsesStreamOptions streamOptions(final ModelProfile modelProfile) { + return null; + } + + /** + * Configures the optional Responses {@code text} object. + *

+ * This can be used to influence text-only output shaping such as verbosity. + * + * @param modelProfile the resolved model profile for the request + * @return text output options, or {@code null} to omit them + */ + default ResponsesText text(final ModelProfile modelProfile) { + return null; + } + + /** + * Requests token-level log probability data for text outputs. + * + * @param modelProfile the resolved model profile for the request + * @return the number of top logprobs to request, or {@code null} to omit the + * field + */ + default Integer topLogprobs(final ModelProfile modelProfile) { + return null; + } + + /** + * Selects how the provider should react when the input exceeds the model's + * context window. + * + * @param modelProfile the resolved model profile for the request + * @return the truncation mode, or {@code null} to omit it + */ + default Truncation truncation(final ModelProfile modelProfile) { + return null; + } + + /** + * Request payload for the Responses {@code reasoning} object. + * + * @param effort the reasoning budget requested from the model + * @param generateSummary optional instruction for whether and how the provider + * should generate reasoning summaries + * @param summary optional summary detail level for reasoning output + */ + record ResponsesReasoning(ReasoningEffort effort, ReasoningSummaryGenerateMode generateSummary, ReasoningSummary summary) { + } + + /** + * Request payload for the Responses {@code stream_options} object. + * + * @param includeObfuscation whether streaming responses should include the + * provider's obfuscation metadata when available + */ + record ResponsesStreamOptions(Boolean includeObfuscation) { + } + + /** + * Request payload for the Responses {@code text} object. + * + * @param verbosity desired verbosity level for text output + */ + record ResponsesText(TextVerbosity verbosity) { + } + + /** + * Retention policy for prompt caching. + */ + enum PromptCacheRetention { + /** Retain cached prompt prefixes in volatile memory only. */ + IN_MEMORY("in_memory"), + /** Retain cached prompt prefixes for up to 24 hours. */ + DAY_24H("24h"); + + private final String wireValue; + + PromptCacheRetention(final String wireValue) { + this.wireValue = wireValue; + } + + @JsonValue + public String getWireValue() { + return this.wireValue; + } + } + + /** + * Reasoning effort levels supported by the Responses API. + */ + enum ReasoningEffort { + /** Disable extra reasoning tokens when the model supports that mode. */ + NONE("none"), + /** Request the smallest possible reasoning budget. */ + MINIMAL("minimal"), + /** Request a low reasoning budget. */ + LOW("low"), + /** Request the default medium reasoning budget. */ + MEDIUM("medium"), + /** Request a high reasoning budget. */ + HIGH("high"), + /** Request the highest extended reasoning budget. */ + XHIGH("xhigh"); + + private final String wireValue; + + ReasoningEffort(final String wireValue) { + this.wireValue = wireValue; + } + + @JsonValue + public String getWireValue() { + return this.wireValue; + } + } + + /** + * Summary detail levels for returned reasoning summaries. + */ + enum ReasoningSummary { + /** Let the provider pick the summary detail level automatically. */ + AUTO("auto"), + /** Request a short reasoning summary. */ + CONCISE("concise"), + /** Request a more detailed reasoning summary. */ + DETAILED("detailed"); + + private final String wireValue; + + ReasoningSummary(final String wireValue) { + this.wireValue = wireValue; + } + + @JsonValue + public String getWireValue() { + return this.wireValue; + } + } + + /** + * Modes for asking the provider to generate reasoning summaries. + */ + enum ReasoningSummaryGenerateMode { + /** Let the provider decide whether and how to generate a summary. */ + AUTO("auto"), + /** Generate a short reasoning summary. */ + CONCISE("concise"), + /** Generate a detailed reasoning summary. */ + DETAILED("detailed"), + /** Do not generate a reasoning summary. */ + NONE("none"); + + private final String wireValue; + + ReasoningSummaryGenerateMode(final String wireValue) { + this.wireValue = wireValue; + } + + @JsonValue + public String getWireValue() { + return this.wireValue; + } + } + + /** + * OpenAI processing tiers supported by the Responses API. + */ + enum ServiceTier { + /** Let the provider choose the service tier automatically. */ + AUTO("auto"), + /** Use the standard service tier. */ + DEFAULT("default"), + /** Use flex processing when available. */ + FLEX("flex"), + /** Use priority processing when available. */ + PRIORITY("priority"); + + private final String wireValue; + + ServiceTier(final String wireValue) { + this.wireValue = wireValue; + } + + @JsonValue + public String getWireValue() { + return this.wireValue; + } + } + + /** + * Verbosity levels for text output. + */ + enum TextVerbosity { + /** Prefer shorter text outputs. */ + LOW("low"), + /** Prefer balanced text outputs. */ + MEDIUM("medium"), + /** Prefer more detailed text outputs. */ + HIGH("high"); + + private final String wireValue; + + TextVerbosity(final String wireValue) { + this.wireValue = wireValue; + } + + @JsonValue + public String getWireValue() { + return this.wireValue; + } + } + + /** + * Input truncation strategies for oversized requests. + */ + enum Truncation { + /** Allow the provider to drop older context automatically. */ + AUTO("auto"), + /** Reject oversized requests instead of truncating them. */ + DISABLED("disabled"); + + private final String wireValue; + + Truncation(final String wireValue) { + this.wireValue = wireValue; + } + + @JsonValue + public String getWireValue() { + return this.wireValue; + } + } +} diff --git a/src/main/java/de/extio/lmlib/client/oai/responses/ResponsesApiClient.java b/src/main/java/de/extio/lmlib/client/oai/responses/ResponsesApiClient.java new file mode 100644 index 0000000..f49507b --- /dev/null +++ b/src/main/java/de/extio/lmlib/client/oai/responses/ResponsesApiClient.java @@ -0,0 +1,1034 @@ +package de.extio.lmlib.client.oai.responses; + +import java.io.BufferedReader; +import java.io.IOException; +import java.io.InputStream; +import java.io.InputStreamReader; +import java.math.BigDecimal; +import java.nio.charset.StandardCharsets; +import java.time.Duration; +import java.time.LocalDateTime; +import java.util.ArrayList; +import java.util.List; +import java.util.LinkedHashSet; +import java.util.LinkedHashMap; +import java.util.Map; +import java.util.concurrent.ConcurrentHashMap; +import java.util.function.Consumer; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.beans.factory.annotation.Qualifier; +import org.springframework.http.HttpMethod; +import org.springframework.web.client.RestClient; +import org.springframework.web.client.RestClientException; +import org.springframework.web.client.RestClientResponseException; + +import com.fasterxml.jackson.annotation.JsonInclude; + +import tools.jackson.core.JacksonException; +import tools.jackson.core.json.JsonReadFeature; +import tools.jackson.databind.DeserializationFeature; +import tools.jackson.databind.JsonNode; +import tools.jackson.databind.ObjectMapper; +import tools.jackson.databind.json.JsonMapper; + +import de.extio.lmlib.client.Chunk; +import de.extio.lmlib.client.Client; +import de.extio.lmlib.client.Completion; +import de.extio.lmlib.client.CompletionFinishReason; +import de.extio.lmlib.client.CompletionStatistics; +import de.extio.lmlib.client.Conversation; +import de.extio.lmlib.client.Conversation.Turn; +import de.extio.lmlib.client.ToolCallData; +import de.extio.lmlib.client.ToolCall; +import de.extio.lmlib.client.ToolDefinition; +import de.extio.lmlib.profile.ModelCategory; +import de.extio.lmlib.profile.ModelProfile; +import de.extio.lmlib.profile.ModelProfile.ModelProvider; +import de.extio.lmlib.profile.ModelProfileService; +import de.extio.lmlib.token.TokenizerResolver; + +public class ResponsesApiClient implements Client{ + + private static final Logger LOGGER = LoggerFactory.getLogger(ResponsesApiClient.class); + + private static final String ASSISTANT_PHASE_COMMENTARY = "commentary"; + + private static final String ASSISTANT_PHASE_FINAL_ANSWER = "final_answer"; + + private static final int TOKEN_MARGIN = 15; + + private static final OpenAiResponsesApiDialect DEFAULT_DIALECT = new OpenAiResponsesApiDialect() { + }; + + @Autowired + @Qualifier("lmLibRestClientBuilder") + private RestClient.Builder restClientBuilder; + + @Autowired + private TokenizerResolver tokenizerResolver; + + @Autowired + private ModelProfileService modelProfileService; + + @Autowired(required = false) + private OpenAiResponsesApiDialect openAiResponsesApiDialect; + + private final Map> resolvedModelNames = new ConcurrentHashMap<>(); + + private final ObjectMapper objectMapper; + + public ResponsesApiClient() { + this.objectMapper = JsonMapper.builder() + .configure(DeserializationFeature.FAIL_ON_IGNORED_PROPERTIES, false) + .configure(DeserializationFeature.FAIL_ON_INVALID_SUBTYPE, false) + .configure(DeserializationFeature.FAIL_ON_MISSING_EXTERNAL_TYPE_ID_PROPERTY, false) + .configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false) + .configure(DeserializationFeature.FAIL_ON_UNRESOLVED_OBJECT_IDS, false) + .changeDefaultPropertyInclusion(incl -> incl.withValueInclusion(JsonInclude.Include.NON_NULL)) + .changeDefaultPropertyInclusion(incl -> incl.withContentInclusion(JsonInclude.Include.NON_NULL)) + .enable(JsonReadFeature.ALLOW_BACKSLASH_ESCAPING_ANY_CHARACTER) + .enable(JsonReadFeature.ALLOW_JAVA_COMMENTS) + .enable(JsonReadFeature.ALLOW_MISSING_VALUES) + .enable(JsonReadFeature.ALLOW_NON_NUMERIC_NUMBERS) + .enable(JsonReadFeature.ALLOW_SINGLE_QUOTES) + .enable(JsonReadFeature.ALLOW_TRAILING_COMMA) + .enable(JsonReadFeature.ALLOW_UNESCAPED_CONTROL_CHARS) + .enable(JsonReadFeature.ALLOW_YAML_COMMENTS) + .enable(JsonReadFeature.ALLOW_LEADING_DECIMAL_POINT_FOR_NUMBERS) + .enable(JsonReadFeature.ALLOW_LEADING_PLUS_SIGN_FOR_NUMBERS) + .enable(JsonReadFeature.ALLOW_TRAILING_DECIMAL_POINT_FOR_NUMBERS) + .enable(JsonReadFeature.ALLOW_UNQUOTED_PROPERTY_NAMES) + .build(); + } + + @Override + public Completion conversation(final ModelCategory modelCategory, final Conversation conversation, final ToolCallData toolCallData, final boolean skipCache) { + return this.streamConversation(modelCategory, conversation, null, toolCallData, skipCache); + } + + @Override + public Completion conversation(final ModelProfile modelProfile, final Conversation conversation, final ToolCallData toolCallData, final boolean skipCache) { + return this.streamConversation(modelProfile, conversation, null, toolCallData, skipCache); + } + + @Override + public Completion streamConversation(final ModelCategory modelCategory, final Conversation conversation, final Consumer chunkConsumer, final ToolCallData toolCallData, final boolean skipCache) { + final var resolvedModelCategory = modelCategory == null ? ModelCategory.MEDIUM : modelCategory; + final var modelProfile = this.modelProfileService.getModelProfile(resolvedModelCategory.getModelProfile(), resolvedModelCategory); + return this.streamConversation(modelProfile, conversation, chunkConsumer, toolCallData, skipCache); + } + + @Override + public Completion streamConversation(final ModelProfile modelProfile, final Conversation conversation, final Consumer chunkConsumer, final ToolCallData toolCallData, final boolean skipCache) { + if (modelProfile == null || modelProfile.modelProvider() != ModelProvider.OAI_RESPONSES) { + throw new IllegalArgumentException("Invalid ModelProfile"); + } + return this.requestCompletion(conversation, modelProfile, chunkConsumer, toolCallData); + } + + @Override + public List getModelNames(final ModelProfile modelProfile, final boolean forceReload) { + if (modelProfile == null || modelProfile.modelProvider() != ModelProvider.OAI_RESPONSES) { + return List.of(); + } + if (modelProfile.modelName() != null && !modelProfile.modelName().isBlank()) { + return List.of(modelProfile.modelName()); + } + if (modelProfile.url() == null || modelProfile.url().isBlank()) { + return List.of(); + } + + final var cacheKey = modelProfile.category() + "|" + modelProfile.url() + "|" + modelProfile.apiKey(); + if (forceReload) { + final var modelNames = this.loadModelNames(modelProfile); + this.resolvedModelNames.put(cacheKey, modelNames); + return modelNames; + } + return this.resolvedModelNames.computeIfAbsent(cacheKey, key -> this.loadModelNames(modelProfile)); + } + + @Override + public ModelProvider getModelProvider() { + return ModelProvider.OAI_RESPONSES; + } + + @Override + public boolean supportsToolCalling(final ModelCategory modelCategory) { + final var resolvedModelCategory = modelCategory == null ? ModelCategory.MEDIUM : modelCategory; + final var modelProfile = this.modelProfileService.getModelProfile(resolvedModelCategory.getModelProfile(), resolvedModelCategory); + return this.supportsToolCalling(modelProfile); + } + + @Override + public boolean supportsToolCalling(final ModelProfile modelProfile) { + return modelProfile != null && modelProfile.modelProvider() == ModelProvider.OAI_RESPONSES; + } + + private Completion requestCompletion(final Conversation conversation, final ModelProfile modelProfile, final Consumer chunkConsumer, final ToolCallData toolCallData) { + final var inputTextBuilder = new StringBuilder(); + final var inputItems = this.createInputItems(conversation, modelProfile, inputTextBuilder); + final var request = this.createResponseRequest(inputItems, modelProfile, chunkConsumer != null, toolCallData); + + final String requestBody; + try { + requestBody = this.objectMapper.writeValueAsString(request); + } + catch (final JacksonException e) { + throw new IllegalStateException("Cannot serialize request body", e); + } + + LOGGER.debug("Requesting response at {}", modelProfile.url()); + final var start = LocalDateTime.now(); + var requestSpec = this.restClientBuilder.baseUrl(modelProfile.url()).build() + .method(HttpMethod.POST) + .uri("/v1/responses") + .header("Content-Type", "application/json"); + if (modelProfile.apiKey() != null && !modelProfile.apiKey().isBlank()) { + requestSpec = requestSpec.header("Authorization", "Bearer " + modelProfile.apiKey()); + } + + return requestSpec.body(requestBody).exchange((clientRequest, clientResponse) -> { + if (clientResponse.getStatusCode().isError()) { + final var errorBody = new String(clientResponse.getBody().readAllBytes(), StandardCharsets.UTF_8); + final var errorMessage = this.getErrorMessage(errorBody); + LOGGER.warn("Error response from server: {} {}", clientResponse.getStatusCode(), errorMessage); + throw new IllegalStateException("Error response from server: " + clientResponse.getStatusCode() + " " + errorMessage); + } + + if (chunkConsumer == null) { + return this.parseCompletion(clientResponse.getBody(), modelProfile, start, inputTextBuilder.toString()); + } + return this.parseStreamingCompletion(clientResponse.getBody(), modelProfile, start, inputTextBuilder.toString(), chunkConsumer); + }); + } + + private Map createResponseRequest(final List> inputItems, final ModelProfile modelProfile, final boolean streaming, final ToolCallData toolCallData) { + final var dialect = this.responsesDialect(); + final var store = dialect.store(modelProfile); + final var request = new LinkedHashMap(); + request.put("model", this.getPrimaryModelName(modelProfile)); + request.put("input", inputItems); + request.put("store", store); + request.put("stream", streaming); + request.put("temperature", modelProfile.temperature()); + request.put("top_p", modelProfile.topP()); + if (modelProfile.maxTokens() > 0) { + request.put("max_output_tokens", modelProfile.maxTokens()); + } + final var reasoning = this.createReasoningRequest(dialect.reasoning(modelProfile)); + if (reasoning != null) { + request.put("reasoning", reasoning); + } + final var include = this.createIncludeList(modelProfile, store); + if (!include.isEmpty()) { + request.put("include", include); + } + final var background = dialect.background(modelProfile); + if (background != null) { + request.put("background", background); + } + final var maxToolCalls = dialect.maxToolCalls(modelProfile); + if (maxToolCalls != null) { + request.put("max_tool_calls", maxToolCalls); + } + final var metadata = dialect.metadata(modelProfile); + if (metadata != null && !metadata.isEmpty()) { + request.put("metadata", metadata); + } + final var promptCacheKey = dialect.promptCacheKey(modelProfile); + if (promptCacheKey != null && !promptCacheKey.isBlank()) { + request.put("prompt_cache_key", promptCacheKey); + } + final var promptCacheRetention = dialect.promptCacheRetention(modelProfile); + if (promptCacheRetention != null) { + request.put("prompt_cache_retention", promptCacheRetention); + } + final var safetyIdentifier = dialect.safetyIdentifier(modelProfile); + if (safetyIdentifier != null && !safetyIdentifier.isBlank()) { + request.put("safety_identifier", safetyIdentifier); + } + final var serviceTier = dialect.serviceTier(modelProfile); + if (serviceTier != null) { + request.put("service_tier", serviceTier); + } + final var text = this.createTextRequest(dialect.text(modelProfile)); + if (text != null) { + request.put("text", text); + } + final var topLogprobs = dialect.topLogprobs(modelProfile); + if (topLogprobs != null) { + request.put("top_logprobs", topLogprobs); + } + final var truncation = dialect.truncation(modelProfile); + if (truncation != null) { + request.put("truncation", truncation); + } + final var streamOptions = this.createStreamOptionsRequest(dialect.streamOptions(modelProfile)); + if (streaming && streamOptions != null) { + request.put("stream_options", streamOptions); + } + if (toolCallData != null && toolCallData.hasTools()) { + request.put("tools", this.createTools(toolCallData.tools())); + final var toolChoice = this.createToolChoice(toolCallData); + if (toolChoice != null) { + request.put("tool_choice", toolChoice); + } + if (toolCallData.parallelToolCalls() != null) { + request.put("parallel_tool_calls", toolCallData.parallelToolCalls()); + } + } + return request; + } + + private List> createInputItems(final Conversation conversation, final ModelProfile modelProfile, final StringBuilder inputTextBuilder) { + final var inputItems = new ArrayList>(); + var tokens = 0; + final var maxLength = Math.max(0, modelProfile.maxContextLength() - modelProfile.maxTokens() - TOKEN_MARGIN); + + for (final Turn turn : conversation.getConversation()) { + switch (turn.type()) { + case SYSTEM -> { + if (this.addInputMessageItem("developer", turn.text(), null, inputItems, inputTextBuilder, modelProfile, maxLength, tokens)) { + return inputItems; + } + tokens = this.countTokens(inputTextBuilder.toString(), modelProfile); + } + case USER -> { + if (this.addInputMessageItem("user", turn.text(), null, inputItems, inputTextBuilder, modelProfile, maxLength, tokens)) { + return inputItems; + } + tokens = this.countTokens(inputTextBuilder.toString(), modelProfile); + } + case ASSISTANT -> { + if (!turn.outputItems().isEmpty()) { + if (this.addAssistantOutputItems(turn.outputItems(), inputItems, inputTextBuilder, modelProfile, maxLength)) { + return inputItems; + } + tokens = this.countTokens(inputTextBuilder.toString(), modelProfile); + break; + } + if (turn.text() != null && !turn.text().isBlank()) { + if (this.addInputMessageItem("assistant", turn.text(), this.resolveAssistantPhase(turn.phase(), turn.toolCalls()), inputItems, inputTextBuilder, modelProfile, maxLength, tokens)) { + return inputItems; + } + tokens = this.countTokens(inputTextBuilder.toString(), modelProfile); + } + for (final var toolCall : turn.toolCalls()) { + final var outputItem = new LinkedHashMap(); + outputItem.put("type", "function_call"); + outputItem.put("call_id", toolCall.id()); + outputItem.put("name", toolCall.name()); + outputItem.put("arguments", toolCall.arguments()); + outputItem.put("status", "completed"); + inputItems.add(outputItem); + this.appendInputItemText(inputTextBuilder, outputItem); + tokens = this.countTokens(inputTextBuilder.toString(), modelProfile); + if (tokens > maxLength && maxLength > 0) { + LOGGER.warn("Prompt too long: {} tokens / {} max. Conversation will be cut!", tokens, maxLength); + return inputItems; + } + } + } + case TOOL -> { + final var toolOutput = turn.text() == null ? "" : turn.text(); + var normalizedOutput = toolOutput; + if (maxLength > 0) { + final var currentTokens = this.countTokens(inputTextBuilder.toString(), modelProfile); + final var remaining = maxLength - currentTokens; + if (remaining <= 0) { + LOGGER.warn("Prompt too long: {} tokens / {} max. Conversation will be cut!", currentTokens, maxLength); + return inputItems; + } + normalizedOutput = this.trimToTokenBudget(toolOutput, remaining, modelProfile); + } + final var item = new LinkedHashMap(); + item.put("type", "function_call_output"); + item.put("call_id", turn.toolCallId()); + item.put("output", normalizedOutput); + inputItems.add(item); + this.appendInputItemText(inputTextBuilder, item); + tokens = this.countTokens(inputTextBuilder.toString(), modelProfile); + if (tokens > maxLength) { + LOGGER.warn("Prompt too long: {} tokens / {} max. Conversation will be cut!", tokens, maxLength); + return inputItems; + } + } + } + } + + return inputItems; + } + + private boolean addInputMessageItem(final String role, final String text, final String phase, final List> inputItems, final StringBuilder inputTextBuilder, final ModelProfile modelProfile, final int maxLength, final int currentTokens) { + var normalizedText = text == null ? "" : text; + if (maxLength > 0) { + final var remaining = maxLength - currentTokens; + if (remaining <= 0) { + LOGGER.warn("Prompt too long: {} tokens / {} max. Conversation will be cut!", currentTokens, maxLength); + return true; + } + normalizedText = this.trimToTokenBudget(normalizedText, remaining, modelProfile); + } + final Map item; + if ("assistant".equals(role)) { + item = this.createAssistantMessageInputItem(normalizedText, phase); + } + else { + final var easyInputMessage = new LinkedHashMap(); + easyInputMessage.put("role", role); + easyInputMessage.put("content", normalizedText); + item = easyInputMessage; + } + inputItems.add(item); + this.appendInputText(inputTextBuilder, normalizedText); + return this.countTokens(inputTextBuilder.toString(), modelProfile) > maxLength && maxLength > 0; + } + + private Map createAssistantMessageInputItem(final String normalizedText, final String phase) { + final var messageItem = new LinkedHashMap(); + messageItem.put("type", "message"); + messageItem.put("role", "assistant"); + messageItem.put("status", "completed"); + messageItem.put("content", List.of(Map.of("type", "output_text", "text", normalizedText))); + if (phase != null && !phase.isBlank()) { + messageItem.put("phase", phase); + } + return Map.copyOf(messageItem); + } + + private boolean addAssistantOutputItems(final List> outputItems, final List> inputItems, final StringBuilder inputTextBuilder, final ModelProfile modelProfile, final int maxLength) { + final var containsFunctionCalls = outputItems.stream().anyMatch(outputItem -> "function_call".equals(outputItem.get("type"))); + for (final var outputItem : outputItems) { + final var normalizedOutputItem = this.normalizeAssistantOutputItemPhase(outputItem, containsFunctionCalls); + inputItems.add(normalizedOutputItem); + this.appendInputItemText(inputTextBuilder, normalizedOutputItem); + final var tokens = this.countTokens(inputTextBuilder.toString(), modelProfile); + if (tokens > maxLength && maxLength > 0) { + LOGGER.warn("Prompt too long: {} tokens / {} max. Conversation will be cut!", tokens, maxLength); + return true; + } + } + return false; + } + + private String trimToTokenBudget(final String text, final int remainingTokens, final ModelProfile modelProfile) { + if (text == null || text.isEmpty()) { + return ""; + } + final var textTokens = this.countTokens(text, modelProfile); + if (textTokens <= remainingTokens) { + return text; + } + var tokenized = this.tokenizerResolver.tokenize(text, modelProfile); + if (remainingTokens <= 0 || tokenized.isEmpty()) { + return ""; + } + final var endIndex = Math.max(1, Math.min(tokenized.size(), remainingTokens)); + return this.tokenizerResolver.detokenize(tokenized.subList(0, endIndex), modelProfile); + } + + private List> createTools(final List toolDefinitions) { + return toolDefinitions.stream().map(toolDefinition -> { + final var tool = new LinkedHashMap(); + tool.put("type", "function"); + tool.put("name", toolDefinition.name()); + tool.put("description", toolDefinition.description()); + tool.put("parameters", toolDefinition.parameters()); + if (toolDefinition.strict() != null) { + tool.put("strict", toolDefinition.strict()); + } + return Map.copyOf(tool); + }).toList(); + } + + private Object createToolChoice(final ToolCallData toolCallData) { + if (toolCallData == null) { + return null; + } + if (toolCallData.forcedToolName() != null && !toolCallData.forcedToolName().isBlank()) { + final var toolChoice = new LinkedHashMap(); + toolChoice.put("type", "function"); + toolChoice.put("name", toolCallData.forcedToolName()); + return toolChoice; + } + if (toolCallData.toolChoice() != null && !toolCallData.toolChoice().isBlank()) { + return toolCallData.toolChoice(); + } + return null; + } + + private Completion parseCompletion(final InputStream body, final ModelProfile modelProfile, final LocalDateTime start, final String prompt) throws IOException { + final var response = this.objectMapper.readTree(body); + final var output = response.path("output"); + final var responseText = this.extractAssistantMessageText(output); + final var reasoningText = this.extractReasoningSummaryText(output); + final var toolCalls = this.extractToolCalls(output); + final var outputItems = this.extractOutputItems(output); + final var finishReason = this.determineFinishReason(response, toolCalls); + final var statistics = this.createCompletionStatistics(modelProfile, start, response.path("usage"), prompt, responseText, reasoningText); + return new Completion(responseText, reasoningText, finishReason, statistics, toolCalls, outputItems); + } + + private Completion parseStreamingCompletion(final InputStream body, final ModelProfile modelProfile, final LocalDateTime start, final String prompt, final Consumer chunkConsumer) throws IOException { + var response = this.objectMapper.createObjectNode(); + final var responseText = new StringBuilder(); + final var reasoningText = new StringBuilder(); + final var toolCalls = new ArrayList(); + + try (BufferedReader reader = new BufferedReader(new InputStreamReader(body, StandardCharsets.UTF_8))) { + String line; + while ((line = reader.readLine()) != null) { + if (line.startsWith(":")) { + continue; + } + if (line.startsWith("event:")) { + continue; + } + if (line.startsWith("data:")) { + line = line.substring(5).trim(); + } + if (line.isEmpty()) { + continue; + } + if ("[DONE]".equals(line)) { + break; + } + + final JsonNode event; + try { + event = this.objectMapper.readTree(line); + } + catch (final JacksonException e) { + LOGGER.warn("Streamed chunk is not parseable: {}", line, e); + continue; + } + + final var eventType = this.text(event, "type"); + switch (eventType) { + case "response.output_text.delta" -> { + final var delta = this.text(event, "delta"); + if (!delta.isEmpty()) { + responseText.append(delta); + chunkConsumer.accept(new Chunk(delta, null)); + } + } + case "response.reasoning_text.delta" -> { + final var delta = this.extractReasoningDelta(eventType, event); + if (!delta.isEmpty()) { + reasoningText.append(delta); + chunkConsumer.accept(new Chunk(null, delta)); + } + } + case "response.output_item.added" -> this.mergeFunctionCallItem(toolCalls, event.path("item"), this.integerValue(event.path("output_index"), -1)); + case "response.output_item.done" -> this.mergeFunctionCallItem(toolCalls, event.path("item"), this.integerValue(event.path("output_index"), -1)); + case "response.function_call_arguments.delta" -> this.mergeFunctionCallArguments(toolCalls, this.integerValue(event.path("output_index"), -1), this.text(event, "delta")); + case "response.function_call_arguments.done" -> this.setFunctionCallArguments(toolCalls, this.integerValue(event.path("output_index"), -1), this.text(event, "arguments")); + case "response.completed" -> { + if (event.path("response").isObject()) { + response = (tools.jackson.databind.node.ObjectNode) event.path("response"); + } + } + case "response.failed", "error" -> throw new IllegalStateException(this.getEventErrorMessage(event)); + default -> { + // Ignore unknown event types + LOGGER.debug("Unhandled event type in streamed response: {}", eventType); + } + } + } + } + + if (!response.path("output").isArray()) { + response.putArray("output"); + } + + final var completedText = responseText.isEmpty() ? this.extractAssistantMessageText(response.path("output")) : responseText.toString(); + final var completedReasoning = reasoningText.isEmpty() ? this.extractReasoningSummaryText(response.path("output")) : reasoningText.toString(); + final var completedToolCalls = this.extractToolCalls(response.path("output")); + final var finalToolCalls = completedToolCalls.isEmpty() ? this.toToolCalls(toolCalls) : completedToolCalls; + var outputItems = this.extractOutputItems(response.path("output")); + if (outputItems.isEmpty()) { + outputItems = this.createFallbackOutputItems(completedText, finalToolCalls); + } + final var finishReason = this.determineFinishReason(response, finalToolCalls); + final var statistics = this.createCompletionStatistics(modelProfile, start, response.path("usage"), prompt, completedText, completedReasoning); + return new Completion(completedText, completedReasoning, finishReason, statistics, finalToolCalls, outputItems); + } + + private void mergeFunctionCallItem(final List toolCalls, final JsonNode item, final int outputIndex) { + if (!"function_call".equals(this.text(item, "type"))) { + return; + } + final var toolCall = this.ensureFunctionCall(toolCalls, outputIndex); + final var callId = this.text(item, "call_id"); + if (!callId.isEmpty()) { + toolCall.callId = callId; + } + final var name = this.text(item, "name"); + if (!name.isEmpty()) { + toolCall.name = name; + } + final var arguments = this.text(item, "arguments"); + if (!arguments.isEmpty()) { + toolCall.arguments = arguments; + } + toolCall.type = "function"; + } + + private void mergeFunctionCallArguments(final List toolCalls, final int outputIndex, final String delta) { + if (delta == null || delta.isEmpty()) { + return; + } + final var toolCall = this.ensureFunctionCall(toolCalls, outputIndex); + toolCall.arguments = (toolCall.arguments == null ? "" : toolCall.arguments) + delta; + } + + private void setFunctionCallArguments(final List toolCalls, final int outputIndex, final String arguments) { + if (arguments == null || arguments.isEmpty()) { + return; + } + this.ensureFunctionCall(toolCalls, outputIndex).arguments = arguments; + } + + private StreamingToolCall ensureFunctionCall(final List toolCalls, final int outputIndex) { + final var index = outputIndex < 0 ? toolCalls.size() : outputIndex; + while (toolCalls.size() <= index) { + toolCalls.add(new StreamingToolCall()); + } + return toolCalls.get(index); + } + + private List extractToolCalls(final JsonNode output) { + if (!output.isArray()) { + return List.of(); + } + final var toolCalls = new ArrayList(); + for (final var item : output) { + if (!"function_call".equals(this.text(item, "type"))) { + continue; + } + final var callId = this.text(item, "call_id"); + toolCalls.add(new ToolCall( + callId.isBlank() ? this.text(item, "id") : callId, + "function", + this.text(item, "name"), + this.text(item, "arguments"))); + } + return List.copyOf(toolCalls); + } + + private List toToolCalls(final List streamingToolCalls) { + return streamingToolCalls.stream() + .filter(toolCall -> toolCall.callId != null && !toolCall.callId.isBlank()) + .map(toolCall -> new ToolCall(toolCall.callId, toolCall.type == null ? "function" : toolCall.type, toolCall.name, toolCall.arguments)) + .toList(); + } + + private List> extractOutputItems(final JsonNode output) { + if (!output.isArray()) { + return List.of(); + } + final var outputItems = new ArrayList>(); + for (final var item : output) { + if (!item.isObject()) { + continue; + } + outputItems.add(this.objectMapper.convertValue(item, this.objectMapper.getTypeFactory().constructMapType(LinkedHashMap.class, String.class, Object.class))); + } + return List.copyOf(outputItems); + } + + private String extractAssistantMessageText(final JsonNode output) { + if (!output.isArray()) { + return ""; + } + final var response = new StringBuilder(); + for (final var item : output) { + if (!"message".equals(this.text(item, "type")) || !"assistant".equals(this.text(item, "role"))) { + continue; + } + for (final var contentItem : item.path("content")) { + final var type = this.text(contentItem, "type"); + if ("output_text".equals(type) || "text".equals(type)) { + response.append(this.text(contentItem, "text")); + } + } + } + return response.toString(); + } + + private String extractReasoningSummaryText(final JsonNode output) { + if (!output.isArray()) { + return ""; + } + final var reasoning = new StringBuilder(); + for (final var item : output) { + if (!"reasoning".equals(this.text(item, "type"))) { + continue; + } + var hasSummary = false; + for (final var summaryItem : item.path("summary")) { + final var text = this.text(summaryItem, "text"); + if (!text.isEmpty()) { + reasoning.append(text); + hasSummary = true; + } + } + if (!hasSummary) { + for (final var contentItem : item.path("content")) { + final var type = this.text(contentItem, "type"); + if ("reasoning_text".equals(type) || "text".equals(type)) { + reasoning.append(this.text(contentItem, "text")); + } + } + } + } + return reasoning.toString(); + } + + private String extractReasoningDelta(final String eventType, final JsonNode event) { + if (eventType == null || !eventType.contains("reasoning")) { + return ""; + } + final var delta = this.text(event, "delta"); + if (!delta.isEmpty()) { + return delta; + } + return this.text(event, "text"); + } + + private CompletionFinishReason determineFinishReason(final JsonNode response, final List toolCalls) { + if (!toolCalls.isEmpty()) { + return CompletionFinishReason.TOOL_CALLS; + } + final var status = this.text(response, "status"); + return switch (status) { + case "completed" -> CompletionFinishReason.DONE; + case "failed" -> CompletionFinishReason.ERROR; + case "incomplete" -> { + final var incompleteReason = this.text(response.path("incomplete_details"), "reason"); + yield switch (incompleteReason) { + case "max_output_tokens", "max_tokens", "length" -> CompletionFinishReason.TOKEN_LIMIT_REACHED; + case "content_filter", "content_filtered" -> CompletionFinishReason.CONTENT_FILTERED; + case "" -> CompletionFinishReason.DONE; + default -> CompletionFinishReason.ERROR; + }; + } + default -> CompletionFinishReason.DONE; + }; + } + + private CompletionStatistics createCompletionStatistics(final ModelProfile modelProfile, final LocalDateTime start, final JsonNode usage, final String prompt, final String response, final String reasoning) { + var inTokens = 0; + var cachedInTokens = 0; + var outTokens = 0; + var reasoningOutTokens = 0; + + if (usage != null && !usage.isMissingNode()) { + final var inputTokens = this.integerValue(usage.path("input_tokens"), 0); + cachedInTokens = this.integerValue(usage.path("input_tokens_details").path("cached_tokens"), 0); + final var outputTokens = this.integerValue(usage.path("output_tokens"), 0); + reasoningOutTokens = this.integerValue(usage.path("output_tokens_details").path("reasoning_tokens"), 0); + inTokens = Math.max(0, inputTokens - cachedInTokens); + outTokens = Math.max(0, outputTokens - reasoningOutTokens); + } + if (inTokens <= 0 && prompt != null && !prompt.isBlank()) { + inTokens = this.countTokens(prompt, modelProfile); + } + if (reasoningOutTokens <= 0 && reasoning != null && !reasoning.isBlank()) { + reasoningOutTokens = this.countTokens(reasoning, modelProfile); + outTokens = Math.max(0, outTokens - reasoningOutTokens); + } + if (outTokens <= 0 && response != null && !response.isBlank()) { + outTokens = this.countTokens(response, modelProfile); + } + + final var cost = this.getUsageCost(usage) + .max(this.calculateTokenCost(modelProfile, inTokens, cachedInTokens, outTokens, reasoningOutTokens)); + return new CompletionStatistics( + 1, + Duration.between(start, LocalDateTime.now()), + inTokens, + cachedInTokens, + outTokens, + reasoningOutTokens, + cost, + false); + } + + private BigDecimal getUsageCost(final JsonNode usage) { + if (usage == null || usage.isMissingNode()) { + return BigDecimal.ZERO; + } + var cost = this.decimalValue(usage.path("cost")); + if (cost.compareTo(BigDecimal.ZERO) > 0) { + return cost; + } + cost = this.decimalValue(usage.path("cost_details").path("upstream_inference_cost")); + if (cost.compareTo(BigDecimal.ZERO) > 0) { + return cost; + } + cost = cost.add(this.decimalValue(usage.path("cost_details").path("upstream_inference_prompt_cost"))); + cost = cost.add(this.decimalValue(usage.path("cost_details").path("upstream_inference_completions_cost"))); + cost = cost.add(this.decimalValue(usage.path("cost_details").path("upstream_inference_input_cost"))); + cost = cost.add(this.decimalValue(usage.path("cost_details").path("upstream_inference_output_cost"))); + return cost; + } + + private BigDecimal calculateTokenCost(final ModelProfile modelProfile, final int inTokens, final int cachedInTokens, final int outTokens, final int reasoningOutTokens) { + var cost = BigDecimal.ZERO; + cost = cost.add(new BigDecimal(inTokens).multiply(modelProfile.costPerInToken())); + cost = cost.add(new BigDecimal(cachedInTokens).multiply(modelProfile.costPerCachedInToken())); + cost = cost.add(new BigDecimal(outTokens).multiply(modelProfile.costPerOutToken())); + cost = cost.add(new BigDecimal(reasoningOutTokens).multiply(modelProfile.costPerReasoningOutToken())); + return cost; + } + + private List loadModelNames(final ModelProfile modelProfile) { + try { + var requestSpec = this.restClientBuilder.baseUrl(modelProfile.url()).build() + .method(HttpMethod.GET) + .uri("/v1/models"); + if (modelProfile.apiKey() != null && !modelProfile.apiKey().isBlank()) { + requestSpec = requestSpec.header("Authorization", "Bearer " + modelProfile.apiKey()); + } + final var response = requestSpec.retrieve().body(String.class); + if (response == null || response.isBlank()) { + return List.of(); + } + final var root = this.objectMapper.readTree(response); + final var models = new ArrayList(); + for (final var item : root.path("data")) { + final var modelName = this.text(item, "id"); + if (!modelName.isBlank()) { + models.add(modelName); + } + } + return List.copyOf(models); + } + catch (final RestClientResponseException e) { + LOGGER.warn("Failed to load model names: {} {}", e.getStatusCode(), e.getStatusText()); + return List.of(); + } + catch (final RestClientException e) { + LOGGER.warn("Failed to load model names: {}", e.getMessage()); + return List.of(); + } + catch (final RuntimeException e) { + LOGGER.warn("Failed to load model names", e); + return List.of(); + } + } + + private String getPrimaryModelName(final ModelProfile modelProfile) { + if (modelProfile.modelName() != null && !modelProfile.modelName().isBlank()) { + return modelProfile.modelName(); + } + return this.getModelNames(modelProfile, false).stream().findFirst().orElse(""); + } + + private String getErrorMessage(final String errorBody) { + if (errorBody == null || errorBody.isBlank()) { + return ""; + } + try { + final var errorNode = this.objectMapper.readTree(errorBody).path("error").path("message"); + final var error = this.nodeText(errorNode); + return error.isBlank() ? errorBody : error; + } + catch (final JacksonException e) { + LOGGER.debug("Failed to parse error response body", e); + return errorBody; + } + } + + private String getEventErrorMessage(final JsonNode event) { + final var message = this.text(event.path("error"), "message"); + if (!message.isEmpty()) { + return message; + } + final var directMessage = this.text(event, "message"); + if (!directMessage.isEmpty()) { + return directMessage; + } + return event.toString(); + } + + private OpenAiResponsesApiDialect responsesDialect() { + return this.openAiResponsesApiDialect == null ? DEFAULT_DIALECT : this.openAiResponsesApiDialect; + } + + private Map createReasoningRequest(final OpenAiResponsesApiDialect.ResponsesReasoning reasoning) { + if (reasoning == null) { + return null; + } + final var request = new LinkedHashMap(); + if (reasoning.effort() != null) { + request.put("effort", reasoning.effort()); + } + if (reasoning.generateSummary() != null) { + request.put("generate_summary", reasoning.generateSummary()); + } + if (reasoning.summary() != null) { + request.put("summary", reasoning.summary()); + } + return request.isEmpty() ? null : request; + } + + private List createIncludeList(final ModelProfile modelProfile, final boolean store) { + final var include = new LinkedHashSet<>(this.responsesDialect().include(modelProfile)); + if (!store) { + include.add(OpenAiResponsesApiDialect.INCLUDE_REASONING_ENCRYPTED_CONTENT); + } + return List.copyOf(include); + } + + private Map createStreamOptionsRequest(final OpenAiResponsesApiDialect.ResponsesStreamOptions streamOptions) { + if (streamOptions == null) { + return null; + } + final var request = new LinkedHashMap(); + if (streamOptions.includeObfuscation() != null) { + request.put("include_obfuscation", streamOptions.includeObfuscation()); + } + return request.isEmpty() ? null : request; + } + + private Map createTextRequest(final OpenAiResponsesApiDialect.ResponsesText text) { + if (text == null) { + return null; + } + final var request = new LinkedHashMap(); + if (text.verbosity() != null) { + request.put("verbosity", text.verbosity()); + } + return request.isEmpty() ? null : request; + } + + private List> createFallbackOutputItems(final String assistantText, final List toolCalls) { + final var outputItems = new ArrayList>(); + if (assistantText != null && !assistantText.isBlank()) { + final var messageItem = new LinkedHashMap(); + messageItem.put("type", "message"); + messageItem.put("role", "assistant"); + messageItem.put("phase", this.resolveAssistantPhase(null, toolCalls)); + messageItem.put("status", "completed"); + messageItem.put("content", List.of(Map.of("type", "output_text", "text", assistantText))); + outputItems.add(messageItem); + } + for (final var toolCall : toolCalls) { + final var functionCallItem = new LinkedHashMap(); + functionCallItem.put("type", "function_call"); + functionCallItem.put("call_id", toolCall.id()); + functionCallItem.put("name", toolCall.name()); + functionCallItem.put("arguments", toolCall.arguments()); + functionCallItem.put("status", "completed"); + outputItems.add(functionCallItem); + } + return List.copyOf(outputItems); + } + + private void appendInputText(final StringBuilder inputTextBuilder, final String text) { + if (text == null || text.isBlank()) { + return; + } + if (!inputTextBuilder.isEmpty()) { + inputTextBuilder.append('\n'); + } + inputTextBuilder.append(text); + } + + private void appendInputItemText(final StringBuilder inputTextBuilder, final Map inputItem) { + if (inputItem == null || inputItem.isEmpty()) { + return; + } + try { + this.appendInputText(inputTextBuilder, this.objectMapper.writeValueAsString(inputItem)); + } + catch (final JacksonException e) { + this.appendInputText(inputTextBuilder, inputItem.toString()); + } + } + + private String resolveAssistantPhase(final String phase, final List toolCalls) { + if (phase != null && !phase.isBlank()) { + return phase; + } + return toolCalls != null && !toolCalls.isEmpty() ? ASSISTANT_PHASE_COMMENTARY : ASSISTANT_PHASE_FINAL_ANSWER; + } + + private Map normalizeAssistantOutputItemPhase(final Map outputItem, final boolean containsFunctionCalls) { + if (outputItem == null || outputItem.isEmpty()) { + return Map.of(); + } + if (!"message".equals(outputItem.get("type")) || !"assistant".equals(outputItem.get("role")) || outputItem.get("phase") instanceof final String phase && !phase.isBlank()) { + return outputItem; + } + final var normalizedOutputItem = new LinkedHashMap<>(outputItem); + normalizedOutputItem.put("phase", containsFunctionCalls ? ASSISTANT_PHASE_COMMENTARY : ASSISTANT_PHASE_FINAL_ANSWER); + return Map.copyOf(normalizedOutputItem); + } + + private int countTokens(final String text, final ModelProfile modelProfile) { + return text == null || text.isBlank() ? 0 : this.tokenizerResolver.count(text, modelProfile); + } + + private String text(final JsonNode node, final String field) { + if (node == null || node.isMissingNode() || node.isNull()) { + return ""; + } + return this.nodeText(node.path(field)); + } + + private int integerValue(final JsonNode node, final int defaultValue) { + if (node == null || node.isMissingNode() || node.isNull()) { + return defaultValue; + } + return node.canConvertToInt() ? node.intValue() : defaultValue; + } + + private BigDecimal decimalValue(final JsonNode node) { + if (node == null || node.isMissingNode() || node.isNull()) { + return BigDecimal.ZERO; + } + try { + return new BigDecimal(this.nodeText(node)); + } + catch (final NumberFormatException e) { + return BigDecimal.ZERO; + } + } + + private String nodeText(final JsonNode node) { + if (node == null || node.isMissingNode() || node.isNull()) { + return ""; + } + switch (node.getNodeType()) { + case STRING: + try { + return this.objectMapper.treeToValue(node, String.class); + } + catch (final JacksonException e) { + return ""; + } + case NUMBER, BOOLEAN: + return node.toString(); + default: + return ""; + } + } + + private static final class StreamingToolCall { + + private String callId; + + private String type; + + private String name; + + private String arguments; + } + +} diff --git a/src/main/java/de/extio/lmlib/profile/ModelProfile.java b/src/main/java/de/extio/lmlib/profile/ModelProfile.java index 27cee39..cce4a97 100755 --- a/src/main/java/de/extio/lmlib/profile/ModelProfile.java +++ b/src/main/java/de/extio/lmlib/profile/ModelProfile.java @@ -42,6 +42,6 @@ public record ModelProfile( } public static enum ModelProvider { - OAI_TEXT_COMPLETION, OAI_CHAT_COMPLETION, OPENROUTER, OLLAMA; + OAI_TEXT_COMPLETION, OAI_CHAT_COMPLETION, OAI_RESPONSES, OPENROUTER, OLLAMA; } } diff --git a/src/test/java/de/extio/lmlib/client/oai/CompletionClientTest.java b/src/test/java/de/extio/lmlib/client/oai/completion/CompletionClientTest.java similarity index 99% rename from src/test/java/de/extio/lmlib/client/oai/CompletionClientTest.java rename to src/test/java/de/extio/lmlib/client/oai/completion/CompletionClientTest.java index e26c546..5867951 100755 --- a/src/test/java/de/extio/lmlib/client/oai/CompletionClientTest.java +++ b/src/test/java/de/extio/lmlib/client/oai/completion/CompletionClientTest.java @@ -1,4 +1,4 @@ -package de.extio.lmlib.client.oai; +package de.extio.lmlib.client.oai.completion; import static org.junit.jupiter.api.Assertions.assertEquals; import static org.junit.jupiter.api.Assertions.assertFalse; diff --git a/src/test/java/de/extio/lmlib/client/oai/responses/ResponsesApiClientTest.java b/src/test/java/de/extio/lmlib/client/oai/responses/ResponsesApiClientTest.java new file mode 100644 index 0000000..31b400a --- /dev/null +++ b/src/test/java/de/extio/lmlib/client/oai/responses/ResponsesApiClientTest.java @@ -0,0 +1,264 @@ +package de.extio.lmlib.client.oai.responses; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.util.List; +import java.util.Map; + +import org.junit.jupiter.api.Disabled; +import org.junit.jupiter.api.Test; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.boot.SpringBootConfiguration; +import org.springframework.boot.autoconfigure.EnableAutoConfiguration; +import org.springframework.boot.test.context.SpringBootTest; +import org.springframework.boot.test.context.SpringBootTest.WebEnvironment; +import org.springframework.context.annotation.Import; +import org.springframework.test.context.TestPropertySource; + +import de.extio.lmlib.client.ClientService; +import de.extio.lmlib.client.CompletionFinishReason; +import de.extio.lmlib.client.Conversation; +import de.extio.lmlib.client.ToolCall; +import de.extio.lmlib.client.ToolCallData; +import de.extio.lmlib.client.ToolDefinition; +import de.extio.lmlib.client.ToolParameters; +import de.extio.lmlib.grader.Grader2; +import de.extio.lmlib.profile.ModelProfile; +import de.extio.lmlib.profile.ModelProfileService; + +@Disabled("This test requires a Responses API compatible endpoint") +@SpringBootTest(webEnvironment = WebEnvironment.NONE) +@SpringBootConfiguration +@EnableAutoConfiguration +@Import(ResponsesApiDialectTestConfiguration.class) +@TestPropertySource("classpath:/application-test.properties") +public class ResponsesApiClientTest { + + private static final Logger LOGGER = LoggerFactory.getLogger(ResponsesApiClientTest.class); + + @Autowired + private ResponsesApiClient responsesApiClient; + + @Autowired + private ModelProfileService modelProfileService; + + @Autowired + private ClientService clientService; + + @Autowired + private ResponsesApiDialectTestConfiguration.MutableOpenAiResponsesApiDialect openAiResponsesApiDialect; + + @Test + void listModels() throws Exception { + final var modelProfile = this.modelProfile(); + final var modelNames = this.responsesApiClient.getModelNames(modelProfile, true); + LOGGER.info("Models: {}", modelNames); + assertFalse(modelNames.isEmpty()); + assertFalse(modelNames.getFirst().isBlank()); + assertEquals(ModelProfile.ModelProvider.OAI_RESPONSES, this.clientService.getClient(modelProfile).getModelProvider()); + } + + @Test + void conversation() throws Exception { + this.openAiResponsesApiDialect.setReasoning(OpenAiResponsesApiDialect.ReasoningEffort.NONE); + runCompletion(); + } + + @Test + void conversationWithReasoning() throws Exception { + this.openAiResponsesApiDialect.setReasoning(OpenAiResponsesApiDialect.ReasoningEffort.MEDIUM); + runCompletion(); + } + + private void runCompletion() { + final var modelProfile = this.modelProfile(); + final var completion = this.responsesApiClient.conversation( + modelProfile, + Conversation.create("You are a helpful assistant", "Say the color is green"), + null, + false); + + LOGGER.info(completion.reasoning()); + LOGGER.info(completion.response()); + LOGGER.info(completion.statistics().toString()); + + assertTrue(Grader2.assessScoreBinary("Does the text mention the color green?", completion.response(), modelProfile, this.clientService)); + } + + @Test + void streamConversation() throws Exception { + this.openAiResponsesApiDialect.setReasoning(OpenAiResponsesApiDialect.ReasoningEffort.NONE); + runStreamedCompletion(); + } + + @Test + void streamConversationWithReasoning() throws Exception { + this.openAiResponsesApiDialect.setReasoning(OpenAiResponsesApiDialect.ReasoningEffort.MEDIUM); + runStreamedCompletion(); + } + + private void runStreamedCompletion() { + final var modelProfile = this.modelProfile(); + final var conversation = Conversation.create("You are a helpful assistant", "I would like to play a game of chess."); + conversation.addTurn(new Conversation.Turn(Conversation.TurnType.ASSISTANT, "Which opening do you choose?")); + conversation.addTurn(new Conversation.Turn(Conversation.TurnType.USER, "You choose, but explain the opening to me.")); + final var responseBuilder = new StringBuilder(); + final var reasoningBuilder = new StringBuilder(); + final var completion = this.responsesApiClient.streamConversation(modelProfile, conversation, chunk -> { + if (chunk.reasoningContent() != null) { + System.out.print(chunk.reasoningContent()); + reasoningBuilder.append(chunk.reasoningContent()); + } + if (chunk.content() != null) { + System.out.print(chunk.content()); + responseBuilder.append(chunk.content()); + } + }, null, false); + + LOGGER.info(completion.statistics().toString()); + + assertEquals(completion.response(), responseBuilder.toString()); + if (completion.reasoning() != null && !completion.reasoning().isBlank()) { + assertEquals(completion.reasoning(), reasoningBuilder.toString()); + } + assertTrue(Grader2.assessScoreBinary("Does the text explain a chess opening?", completion.response(), modelProfile, this.clientService)); + } + + @Test + void toolCallingConversation() throws Exception { + this.openAiResponsesApiDialect.setReasoning(OpenAiResponsesApiDialect.ReasoningEffort.MEDIUM); + final var modelProfile = this.modelProfile(); + final var weatherTool = new ToolDefinition( + "get_weather", + "Gets the current weather for a given location.", + ToolParameters.create(Map.of("location", "City and country, for example Berlin, Germany")), + true); + final var conversation = Conversation.create( + "You are a helpful assistant. Use the provided tools when the user asks for location information. After receiving tool results, answer the question from the tool output.", + "What is the weather in Berlin, Germany? Use the tool."); + final var completion = this.responsesApiClient.conversation( + modelProfile, + conversation, + ToolCallData.required(List.of(weatherTool)).withParallelToolCalls(true), + false); + + LOGGER.info(completion.reasoning()); + LOGGER.info(completion.response()); + LOGGER.info("Tool calls: {}", completion.toolCalls()); + LOGGER.info(completion.statistics().toString()); + + assertEquals(CompletionFinishReason.TOOL_CALLS, completion.finishReason()); + assertFalse(completion.outputItems().isEmpty()); + assertFalse(completion.toolCalls().isEmpty()); + final var weatherToolCall = this.findToolCall(completion.toolCalls(), "get_weather"); + assertTrue(weatherToolCall.arguments().contains("Berlin")); + + conversation.appendToolCallRound(completion, List.of( + new Conversation.ToolResult(ToolParameters.create() + .add("forecast", "Berlin is rainy and 21C") + .json(), weatherToolCall.id()))); + + final var finalCompletion = this.responsesApiClient.conversation(modelProfile, conversation, ToolCallData.auto(List.of(weatherTool)), false); + LOGGER.info(finalCompletion.reasoning()); + LOGGER.info(finalCompletion.response()); + LOGGER.info(finalCompletion.statistics().toString()); + assertEquals(CompletionFinishReason.DONE, finalCompletion.finishReason()); + assertTrue(Grader2.assessScoreBinary("Does the text say that Berlin is rainy and 21C?", finalCompletion.response(), modelProfile, this.clientService)); + } + + @Test + void streamToolCallingConversationBothRounds() throws Exception { + this.openAiResponsesApiDialect.setReasoning(OpenAiResponsesApiDialect.ReasoningEffort.MEDIUM); + final var modelProfile = this.modelProfile(); + final var weatherTool = new ToolDefinition( + "get_weather", + "Gets the current weather for a given location.", + ToolParameters.create(Map.of("location", "City and country, for example Berlin, Germany")), + true); + final var conversation = Conversation.create( + "You are a helpful assistant. Use the provided tools when the user asks for location information. After receiving tool results, answer the question from the tool output.", + "What is the weather in Berlin, Germany? Use the tool."); + final var firstRoundResponseBuilder = new StringBuilder(); + final var firstRoundReasoningBuilder = new StringBuilder(); + final var completion = this.responsesApiClient.streamConversation( + modelProfile, + conversation, + chunk -> { + if (chunk.reasoningContent() != null) { + System.out.print(chunk.reasoningContent()); + firstRoundReasoningBuilder.append(chunk.reasoningContent()); + } + if (chunk.content() != null) { + System.out.print(chunk.content()); + firstRoundResponseBuilder.append(chunk.content()); + } + }, + ToolCallData.required(List.of(weatherTool)).withParallelToolCalls(true), + false); + + LOGGER.info(completion.reasoning()); + LOGGER.info(completion.response()); + LOGGER.info("Tool calls: {}", completion.toolCalls()); + LOGGER.info(completion.statistics().toString()); + + assertEquals(CompletionFinishReason.TOOL_CALLS, completion.finishReason()); + assertEquals(completion.response(), firstRoundResponseBuilder.toString()); + if (completion.reasoning() != null && !completion.reasoning().isBlank()) { + assertEquals(completion.reasoning(), firstRoundReasoningBuilder.toString()); + } + assertFalse(completion.outputItems().isEmpty()); + assertFalse(completion.toolCalls().isEmpty()); + final var weatherToolCall = this.findToolCall(completion.toolCalls(), "get_weather"); + assertTrue(weatherToolCall.arguments().contains("Berlin")); + + conversation.appendToolCallRound(completion, List.of( + new Conversation.ToolResult(ToolParameters.create() + .add("forecast", "Berlin is rainy and 21C") + .json(), weatherToolCall.id()))); + + final var secondRoundResponseBuilder = new StringBuilder(); + final var secondRoundReasoningBuilder = new StringBuilder(); + final var finalCompletion = this.responsesApiClient.streamConversation( + modelProfile, + conversation, + chunk -> { + if (chunk.reasoningContent() != null) { + System.out.print(chunk.reasoningContent()); + secondRoundReasoningBuilder.append(chunk.reasoningContent()); + } + if (chunk.content() != null) { + System.out.print(chunk.content()); + secondRoundResponseBuilder.append(chunk.content()); + } + }, + ToolCallData.auto(List.of(weatherTool)), + false); + + LOGGER.info(finalCompletion.reasoning()); + LOGGER.info(finalCompletion.response()); + LOGGER.info(finalCompletion.statistics().toString()); + + assertEquals(CompletionFinishReason.DONE, finalCompletion.finishReason()); + assertEquals(finalCompletion.response(), secondRoundResponseBuilder.toString()); + if (finalCompletion.reasoning() != null && !finalCompletion.reasoning().isBlank()) { + assertEquals(finalCompletion.reasoning(), secondRoundReasoningBuilder.toString()); + } + assertTrue(Grader2.assessScoreBinary("Does the text say that Berlin is rainy and 21C?", finalCompletion.response(), modelProfile, this.clientService)); + } + + private ModelProfile modelProfile() { + return this.modelProfileService.getModelProfile("profile.model.responses"); + } + + private ToolCall findToolCall(final List toolCalls, final String toolName) { + return toolCalls.stream() + .filter(toolCall -> toolName.equals(toolCall.name())) + .findFirst() + .orElseThrow(() -> new IllegalStateException("Missing tool call: " + toolName)); + } + +} diff --git a/src/test/java/de/extio/lmlib/client/oai/responses/ResponsesApiDialectTestConfiguration.java b/src/test/java/de/extio/lmlib/client/oai/responses/ResponsesApiDialectTestConfiguration.java new file mode 100644 index 0000000..88388d0 --- /dev/null +++ b/src/test/java/de/extio/lmlib/client/oai/responses/ResponsesApiDialectTestConfiguration.java @@ -0,0 +1,29 @@ +package de.extio.lmlib.client.oai.responses; + +import org.springframework.boot.test.context.TestConfiguration; +import org.springframework.context.annotation.Bean; + +import de.extio.lmlib.profile.ModelProfile; + +@TestConfiguration +public class ResponsesApiDialectTestConfiguration { + + @Bean + MutableOpenAiResponsesApiDialect openAiResponsesApiDialect() { + return new MutableOpenAiResponsesApiDialect(); + } + + public static class MutableOpenAiResponsesApiDialect implements OpenAiResponsesApiDialect { + + private volatile ReasoningEffort reasoning = ReasoningEffort.MEDIUM; + + @Override + public ResponsesReasoning reasoning(final ModelProfile modelProfile) { + return this.reasoning == null ? null : new ResponsesReasoning(this.reasoning, null, null); + } + + public void setReasoning(final ReasoningEffort reasoning) { + this.reasoning = reasoning; + } + } +} \ No newline at end of file diff --git a/src/test/resources/application-test.properties b/src/test/resources/application-test.properties index f14a582..df707d8 100755 --- a/src/test/resources/application-test.properties +++ b/src/test/resources/application-test.properties @@ -6,6 +6,7 @@ profile.model.cold=local-chatcompletion profile.model.textcompletion=local-completion profile.model.chatcompletion=local-chatcompletion +profile.model.responses=local-responses profile.model.ollama=local-ollama-chatcompletion profile.model.openrouter=local-openrouter-chatcompletion #profile.model.azure=llama3.3-70b-azure diff --git a/src/test/resources/local-responses.properties b/src/test/resources/local-responses.properties new file mode 100644 index 0000000..b4191ba --- /dev/null +++ b/src/test/resources/local-responses.properties @@ -0,0 +1,16 @@ +category=RESPONSES +prompts= +tokenizer=llamaServer +tokenEncoding=cl100k_base +maxTokens=2500 +maxContextLength=16000 +temperature=0.4 +topP=1.0 +modelProvider=OAI_RESPONSES +modelName= +url=http://localhost:5001 +apiKey= +cost1MInTokens=0 +cost1MCachedInTokens=0 +cost1MOutTokens=0 +cost1MReasoningOutTokens=0