diff --git a/README.md b/README.md index 889a78c..fedd3a0 100644 --- a/README.md +++ b/README.md @@ -20,6 +20,7 @@ The current built-in client set supports both chat completions and legacy text c ## Key Capabilities - OpenAI-compatible chat and text completions +- Chat-completion tool calling with tool definitions, returned tool calls, and tool-result follow-up rounds - Streaming and non-streaming APIs - Separate reasoning capture for models that expose reasoning output - Model profile loading from properties or direct programmatic profile creation @@ -170,7 +171,7 @@ Each value refers to a profile file on the classpath without the `.properties` s ```java final var client = clientService.getClient(ModelCategory.MEDIUM); -final var completion = client.conversation(ModelCategory.LARGE, conversation, false); +final var completion = client.conversation(ModelCategory.LARGE, conversation, null, false); ``` ### 2. Custom Categories @@ -181,7 +182,7 @@ Create categories dynamically when predefined ones are too restrictive: final var customCategory = new ModelCategory("profile-name", "CUSTOM"); final var client = clientService.getClient(customCategory); -final var completion = client.conversation(customCategory, conversation, false); +final var completion = client.conversation(customCategory, conversation, null, false); ``` This is useful for runtime routing, model experiments, or per-tenant profile selection. @@ -212,7 +213,7 @@ final var modelProfile = new ModelProfile( ); final var client = clientService.getClient(modelProfile); -final var completion = client.conversation(modelProfile, conversation, false); +final var completion = client.conversation(modelProfile, conversation, null, false); ``` Agents can also use a specific profile directly: @@ -225,6 +226,7 @@ final var agent = new Agent( modelProfile, "You are a helpful assistant.", "{{userInput}}", + null, new TextAgentResponseHandler("response"), null, null, null, AgentNext::end @@ -317,6 +319,8 @@ modelProvider=OAI_TEXT_COMPLETION Chat completions are the preferred default. They are simpler to use because the library only needs to send the conversation. Text completions are still useful when you need explicit prompt formatting or must work with instruct models that depend on raw prompt structure. +Tool calling is currently supported only for chat completion models. `TextCompletionClient` does not support tools, and `supportsToolCalling()` can be used on `Client`, `ClientService`, and `BaseAgent` to detect availability before enabling tool-aware flows. + These built-in providers are OpenAI-compatible today, but that is an implementation detail of the current client set, not a hard architectural limit of the library. Additional provider integrations can be added in the library itself, and applications can plug in their own client implementations where needed. For text completions, select a prompt strategy with the `prompts` property. lm-lib ships built-in strategies for common model families including Llama, GPT-OSS, Gemma, Mistral, Phi, Qwen, Vicuna, ChatML, Alpaca, and a no-formatting strategy. @@ -343,9 +347,61 @@ Typical integration flow: ```java final var client = clientService.getClient(ModelCategory.MEDIUM); -final var completion = client.conversation(ModelCategory.MEDIUM, conversation, false); +final var completion = client.conversation(ModelCategory.MEDIUM, conversation, null, false); ``` +### Tool Calling + +Tool calling lets chat completion models return one or more structured tool invocations instead of a final natural-language answer. The library exposes this in both the direct client API and the agent runtime. + +The main tool-calling types are: + +- `ToolDefinition`: describes a callable tool and its input schema +- `ToolParameters`: helper for building nested parameter maps and JSON outputs +- `ToolCallData`: request-side tool configuration such as available tools and tool choice mode +- `ToolCall`: a tool call returned by the model + +Client-side flow: + +```java +final var weatherTool = new ToolDefinition( + "get_weather", + "Gets the current weather for a given location.", + ToolParameters.create(Map.of("location", "City and country, for example Berlin, Germany")), + true +); + +final var toolCallData = ToolCallData.required(List.of(weatherTool)); +final var completion = client.conversation(ModelCategory.MEDIUM, conversation, toolCallData, false); + +if (completion.finishReason() == CompletionFinishReason.TOOL_CALLS) { + final var toolCall = completion.toolCalls().getFirst(); + + conversation.addTurn(new Conversation.Turn(Conversation.TurnType.ASSISTANT, "", completion.toolCalls(), null)); + conversation.addTurn(new Conversation.Turn( + Conversation.TurnType.TOOL, + ToolParameters.create().add("forecast", "Berlin is rainy and 21C").json(), + null, + toolCall.id() + )); + + final var finalCompletion = client.conversation( + ModelCategory.MEDIUM, + conversation, + ToolCallData.auto(List.of(weatherTool)), + false + ); +} +``` + +`ToolCallData` helpers: + +- `ToolCallData.auto(...)`: model may decide whether to call a tool +- `ToolCallData.required(...)`: model must choose one of the provided tools +- `ToolCallData.force(...)`: model must call a specific named tool + +The returned `Completion` now exposes `toolCalls()` in addition to `response()`, `reasoning()`, `finishReason()`, and `statistics()`. + ## Completions and Streaming Both streaming and non-streaming execution paths are supported. @@ -353,7 +409,9 @@ Both streaming and non-streaming execution paths are supported. - Non-streaming calls return a final `Completion` object when the request finishes. - Streaming calls accept a `Consumer` and still return the final `Completion` after the stream completes. - Each `Chunk` exposes both `content()` and `reasoningContent()`. -- Final `Completion` objects expose `response()`, `reasoning()`, `finishReason()`, and `statistics()`. +- Final `Completion` objects expose `response()`, `reasoning()`, `finishReason()`, `statistics()`, and `toolCalls()`. + +If a chat completion decides to call a tool, `finishReason()` is `CompletionFinishReason.TOOL_CALLS` and `toolCalls()` contains the requested function calls. Streaming chat completions also aggregate tool calls into the final `Completion`. This lets you stream visible output while still capturing final structured accounting data. @@ -364,12 +422,12 @@ If a `CachedClientRepository` bean is available, lm-lib will wrap clients with a Direct client calls support a `skipCache` flag: ```java -final var cached = client.conversation(ModelCategory.MEDIUM, conversation, false); -final var fresh = client.conversation(ModelCategory.MEDIUM, conversation, true); +final var cached = client.conversation(ModelCategory.MEDIUM, conversation, null, false); +final var fresh = client.conversation(ModelCategory.MEDIUM, conversation, null, true); final var streamed = client.streamConversation(ModelCategory.MEDIUM, conversation, chunk -> { System.out.print(chunk.content()); -}, true); +}, null, true); ``` `skipCache=true` bypasses reading a cached response for that request but still stores the fresh result afterwards. @@ -588,6 +646,7 @@ final var blogPostAgent = new Agent( null, "You are a concise technical blog writer.", "Write a blog post about {{topic}} using these bullets:\n\n{{points}}", + null, new TextAgentResponseHandler("blogPost"), null, null, @@ -606,12 +665,50 @@ final var blogPostAgent = new Agent( | `modelProfile(context)` | Supplies a specific model profile and overrides `modelCategory()`. | `null` | | `systemPrompt()` | Optional system instruction. | `null` | | `textTemplate()` | Optional user prompt template. | `null` | +| `toolDefinitions(context)` | Optional tool definitions for tool-capable chat models. | `null` | +| `toolCallData(context)` | Optional tool request configuration. Defaults to `ToolCallData.auto(toolDefinitions)`. | `null` | +| `supportsToolCalling(context, clientService)` | Reports whether the resolved client can handle tool calls. | derived from model/provider | | `responseHandler(context)` | Parses the model response. | `new TextAgentResponseHandler("response")` | | `preProcess(context)` | Runs before completion. | no-op | | `postProcess(context)` | Runs after completion. | no-op | | `merge(contexts)` | Merges split contexts after branching. | `null` | | `chooseNextAgent(context)` | Selects the next agent or ends the flow. Required. | none | +### Agent Tool Calling + +Agents can declare tool definitions directly. For simple in-agent tool execution loops, wrap a normal response handler with `ToolCallingAgentResponseHandler`. + +```java +final var weatherAgent = new Agent( + "WeatherAgent", + AgentType.START_CONVERSATION, + ModelCategory.MEDIUM, + null, + "Use the provided tool when the user asks for weather information.", + "What is the weather in Berlin, Germany? Use the tool.", + List.of(weatherTool), + new ToolCallingAgentResponseHandler( + new TextAgentResponseHandler("answer"), + (completion, context, toolCallResults) -> { + if (completion.toolCalls().isEmpty()) { + return false; + } + toolCallResults.add( + completion.toolCalls().getFirst(), + ToolParameters.create().add("forecast", "Berlin is rainy and 21C") + ); + return true; + } + ), + null, + null, + null, + AgentNext::end +); +``` + +If you do not want the current agent to resolve tool calls itself, the conversation handoff now preserves assistant tool-call turns so the next `CONVERSATION` or `PROCESSING_ONLY` agent can inspect them in `preProcess()` and decide how to continue. + ### Agent Types | Type | Behavior | Typical Use | diff --git a/src/main/java/de/extio/lmlib/agent/Agent.java b/src/main/java/de/extio/lmlib/agent/Agent.java index 4d16bde..901e4b1 100755 --- a/src/main/java/de/extio/lmlib/agent/Agent.java +++ b/src/main/java/de/extio/lmlib/agent/Agent.java @@ -5,6 +5,7 @@ import java.util.function.Consumer; import java.util.function.Function; import de.extio.lmlib.agent.responsehandler.AgentResponseHandler; +import de.extio.lmlib.client.ToolDefinition; import de.extio.lmlib.profile.ModelCategory; import de.extio.lmlib.profile.ModelProfile; @@ -14,6 +15,7 @@ public record Agent(String name, ModelProfile modelProfile, String systemPrompt, String textTemplate, + List toolDefinitions, AgentResponseHandler responseHandler, Consumer preProcessor, Consumer postProcessor, @@ -53,6 +55,14 @@ public record Agent(String name, } return BaseAgent.super.modelProfile(context); } + + @Override + public List toolDefinitions(final AgentContext context) { + if (toolDefinitions != null) { + return toolDefinitions; + } + return BaseAgent.super.toolDefinitions(context); + } @Override public AgentResponseHandler responseHandler(final AgentContext context) { diff --git a/src/main/java/de/extio/lmlib/agent/AgentContext.java b/src/main/java/de/extio/lmlib/agent/AgentContext.java index ab198af..e91a5de 100755 --- a/src/main/java/de/extio/lmlib/agent/AgentContext.java +++ b/src/main/java/de/extio/lmlib/agent/AgentContext.java @@ -13,6 +13,7 @@ import java.util.stream.Collectors; import de.extio.lmlib.client.Completion; import de.extio.lmlib.client.Conversation; +import de.extio.lmlib.client.ToolCallData; public final class AgentContext { @@ -47,6 +48,8 @@ public final class AgentContext { private volatile AgentNext nextAgent; private volatile Completion lastCompletion; + + private volatile ToolCallData toolCallData; private Consumer agentContextUpdateConsumer; @@ -79,6 +82,7 @@ public final class AgentContext { this.nextAgent = other.nextAgent; this.requestStatistic = other.requestStatistic; this.lastCompletion = other.lastCompletion; + this.toolCallData = other.toolCallData; this.agentContextUpdateConsumer = other.agentContextUpdateConsumer; this.errorType = other.errorType; this.errorException = other.errorException; @@ -196,6 +200,14 @@ public final class AgentContext { void setLastCompletion(final Completion lastCompletion) { this.lastCompletion = lastCompletion; } + + ToolCallData getToolCallData() { + return this.toolCallData; + } + + void setToolCallData(final ToolCallData toolCallData) { + this.toolCallData = toolCallData; + } public boolean isError() { return this.errorType != null; diff --git a/src/main/java/de/extio/lmlib/agent/BaseAgent.java b/src/main/java/de/extio/lmlib/agent/BaseAgent.java index f3e57e7..8b6c28a 100644 --- a/src/main/java/de/extio/lmlib/agent/BaseAgent.java +++ b/src/main/java/de/extio/lmlib/agent/BaseAgent.java @@ -4,6 +4,7 @@ import java.util.ArrayList; import java.util.HashMap; import java.util.List; import java.util.Map.Entry; +import java.util.Objects; import java.util.concurrent.CompletableFuture; import java.util.concurrent.ExecutorService; import java.util.function.Consumer; @@ -18,14 +19,19 @@ import tools.jackson.databind.ObjectMapper; import de.extio.lmlib.agent.responsehandler.AgentResponseHandler; import de.extio.lmlib.agent.responsehandler.StreamedAgentResponseHandler; import de.extio.lmlib.agent.responsehandler.TextAgentResponseHandler; +import de.extio.lmlib.agent.responsehandler.ToolCallingAgentResponseHandler; import de.extio.lmlib.client.Chunk; import de.extio.lmlib.client.Client; +import de.extio.lmlib.client.ClientService; import de.extio.lmlib.client.Completion; import de.extio.lmlib.client.Conversation; +import de.extio.lmlib.client.ToolCallData; +import de.extio.lmlib.client.ToolDefinition; import de.extio.lmlib.client.Conversation.Turn; import de.extio.lmlib.client.Conversation.TurnType; import de.extio.lmlib.profile.ModelCategory; import de.extio.lmlib.profile.ModelProfile; +import de.extio.lmlib.profile.ModelProfile.ModelProvider; public interface BaseAgent { @@ -56,6 +62,38 @@ public interface BaseAgent { default String textTemplate() { return null; } + + default List toolDefinitions(final AgentContext context) { + return null; + } + + default ToolCallData toolCallData(final AgentContext context) { + final var toolDefinitions = this.toolDefinitions(context); + if (toolDefinitions == null || toolDefinitions.isEmpty()) { + final var rememberedToolCallData = context.getToolCallData(); + if (rememberedToolCallData == null || !rememberedToolCallData.hasTools()) { + return null; + } + return ToolCallData.auto(rememberedToolCallData.tools()); + } + return ToolCallData.auto(toolDefinitions); + } + + default boolean supportsToolCalling(final AgentContext context) { + final var modelProfile = this.modelProfile(context); + return modelProfile != null && modelProfile.modelProvider() == ModelProvider.OAI_CHAT_COMPLETION; + } + + default boolean supportsToolCalling(final AgentContext context, final ClientService clientService) { + if (clientService == null) { + return this.supportsToolCalling(context); + } + final var modelProfile = this.modelProfile(context); + if (modelProfile != null) { + return clientService.supportsToolCalling(modelProfile); + } + return clientService.supportsToolCalling(this.modelCategory(context)); + } default AgentResponseHandler responseHandler(final AgentContext context) { return new TextAgentResponseHandler("response"); @@ -115,13 +153,18 @@ public interface BaseAgent { split.context().getGraph().add(this.name()); if (!skipCompletionInSplit) { + final var completionClient = Objects.requireNonNull(client); final boolean skipCache = split.context().isSkipCache() || split.context().isAlwaysSkipCache(); split.context().setSkipCache(false); final var conversation = this.setupConversation(split); + final var toolCallData = completionClient.supportsToolCalling() ? this.toolCallData(split.context()) : null; + this.rememberToolCallData(split.context(), toolCallData); boolean parseable = false; Exception parseException = null; - for (int i = 0; i < 2; i++) { + int parseAttempts = 0; + int toolCallRounds = 0; + while (parseAttempts < 2) { LOGGER.debug("Conversation: {}", conversation); final var modelProfile = this.modelProfile(split.context()); @@ -141,33 +184,46 @@ public interface BaseAgent { }; if (modelProfile != null) { requestStart = System.currentTimeMillis(); - completion = client.streamConversation(modelProfile, conversation, chunkConsumer, skipCache); + completion = completionClient.streamConversation(modelProfile, conversation, chunkConsumer, toolCallData, skipCache); } else { requestStart = System.currentTimeMillis(); - completion = client.streamConversation(this.modelCategory(split.context()), conversation, chunkConsumer, skipCache); + completion = completionClient.streamConversation(this.modelCategory(split.context()), conversation, chunkConsumer, toolCallData, skipCache); } } else { if (modelProfile != null) { requestStart = System.currentTimeMillis(); - completion = client.conversation(modelProfile, conversation, skipCache); + completion = completionClient.conversation(modelProfile, conversation, toolCallData, skipCache); } else { requestStart = System.currentTimeMillis(); - completion = client.conversation(this.modelCategory(split.context()), conversation, skipCache); + completion = completionClient.conversation(this.modelCategory(split.context()), conversation, toolCallData, skipCache); } } final var durationMs = System.currentTimeMillis() - requestStart; split.context().getGraph().add("(" + durationMs + " ms)"); split.context().setLastCompletion(completion); split.context().getRequestStatistic().add(completion); + if (!completion.toolCalls().isEmpty() && responseHandler instanceof final ToolCallingAgentResponseHandler toolCallingAgentResponseHandler) { + if (toolCallRounds >= 4) { + parseException = new IllegalStateException("Too many tool call rounds"); + break; + } + final var toolCallResults = new ToolCallingAgentResponseHandler.ToolCallResults(); + if (toolCallingAgentResponseHandler.handleToolCalls(completion, split.context(), toolCallResults)) { + toolCallRounds++; + this.appendToolCallResults(conversation, completion, toolCallResults.results()); + continue; + } + } try { if (!responseHandler.handle(completion, split.context())) { split.context().getGraph().add("⚠"); LOGGER.warn("{} Cannot parse response: {}", this.name(), completion.response()); split.context().setSkipCache(true); + parseAttempts++; continue; } @@ -179,6 +235,7 @@ public interface BaseAgent { split.context().getGraph().add("⚠"); LOGGER.warn("{} Cannot parse response: {}", this.name(), completion.response(), e); split.context().setSkipCache(true); + parseAttempts++; continue; } } @@ -248,6 +305,13 @@ public interface BaseAgent { } return List.of(first); } + + private void appendToolCallResults(final Conversation conversation, final Completion completion, final List toolCallResults) { + conversation.addTurn(new Conversation.Turn(TurnType.ASSISTANT, completion.response() == null ? "" : completion.response(), completion.toolCalls(), null)); + for (final var toolCallResult : toolCallResults) { + conversation.addTurn(new Conversation.Turn(TurnType.TOOL, toolCallResult.output(), null, toolCallResult.toolCallId())); + } + } private List applyTemplate(final AgentContext context) { String text = this.textTemplate() != null ? this.textTemplate() : ""; @@ -358,6 +422,35 @@ public interface BaseAgent { } return conversation; } + + private void appendLastCompletionToConversation(final AgentContext context, final boolean skipCompletion) { + if (skipCompletion || context.getConversation() == null || context.getLastCompletion() == null) { + return; + } + final var currentAgent = context.getAgents().get(context.getCurrentAgentName()); + if (currentAgent == null || currentAgent.agentType(context) == AgentType.PROCESSING_ONLY) { + return; + } + if (context.getConversation().getConversation().getLast().type() == TurnType.ASSISTANT) { + return; + } + final var completion = context.getLastCompletion(); + if ((completion.response() == null || completion.response().isEmpty()) && completion.toolCalls().isEmpty()) { + return; + } + context.getConversation().addTurn(new Conversation.Turn(TurnType.ASSISTANT, completion.response() == null ? "" : completion.response(), completion.toolCalls(), null)); + } + + private void rememberToolCallData(final AgentContext context, final ToolCallData toolCallData) { + if (toolCallData == null || !toolCallData.hasTools()) { + return; + } + context.setToolCallData(toolCallData); + } + + private void clearRememberedToolCallData(final AgentContext context) { + context.setToolCallData(null); + } private void nextAgent(final AgentContext context, final boolean skipCompletion) { context.setNextAgent(this.chooseNextAgent(context)); @@ -388,11 +481,13 @@ public interface BaseAgent { case final BaseAgent a when a.agentType(context) == AgentType.COMPLETION -> { context.getGraph().add("→"); context.setConversation(null); + this.clearRememberedToolCallData(context); } case final BaseAgent a when a.agentType(context) == AgentType.START_CONVERSATION || a.agentType(context) == AgentType.START_CONVERSATION_WITH_SYSTEM_PROMPT -> { context.getGraph().add("🗨"); context.setConversation(null); + this.clearRememberedToolCallData(context); } case final BaseAgent a when a.agentType(context) == AgentType.CONVERSATION || a.agentType(context) == AgentType.CONVERSATION_WITH_SYSTEM_PROMPT -> { @@ -402,15 +497,12 @@ public interface BaseAgent { } else { context.getGraph().add("↴"); - final var currentAgent = context.getAgents().get(context.getCurrentAgentName()); - if (! skipCompletion && currentAgent.agentType(context) != AgentType.PROCESSING_ONLY && context.getConversation().getConversation().getLast().type() != TurnType.ASSISTANT) { - context.getConversation().addTurn(new Conversation.Turn(TurnType.ASSISTANT, context.getLastCompletion().response())); - } + this.appendLastCompletionToConversation(context, skipCompletion); } } case final BaseAgent a when a.agentType(context) == AgentType.PROCESSING_ONLY -> { - + this.appendLastCompletionToConversation(context, skipCompletion); } default -> { diff --git a/src/main/java/de/extio/lmlib/agent/responsehandler/ToolCallingAgentResponseHandler.java b/src/main/java/de/extio/lmlib/agent/responsehandler/ToolCallingAgentResponseHandler.java new file mode 100644 index 0000000..285f0c3 --- /dev/null +++ b/src/main/java/de/extio/lmlib/agent/responsehandler/ToolCallingAgentResponseHandler.java @@ -0,0 +1,65 @@ +package de.extio.lmlib.agent.responsehandler; + +import java.util.ArrayList; +import java.util.List; + +import de.extio.lmlib.agent.AgentContext; +import de.extio.lmlib.client.Completion; +import de.extio.lmlib.client.ToolCall; +import de.extio.lmlib.client.ToolParameters; + +public class ToolCallingAgentResponseHandler implements AgentResponseHandler { + + @FunctionalInterface + public interface ToolCallHandler { + + boolean handle(Completion completion, AgentContext context, ToolCallResults toolCallResults); + + } + + public record ToolCallResult(String toolCallId, String output) { + + } + + public static final class ToolCallResults { + + private final List toolCallResults = new ArrayList<>(); + + public ToolCallResults add(final ToolCall toolCall, final String output) { + return this.add(toolCall.id(), output); + } + + public ToolCallResults add(final ToolCall toolCall, final ToolParameters output) { + return this.add(toolCall.id(), output.json()); + } + + public ToolCallResults add(final String toolCallId, final String output) { + this.toolCallResults.add(new ToolCallResult(toolCallId, output)); + return this; + } + + public List results() { + return List.copyOf(this.toolCallResults); + } + + } + + private final AgentResponseHandler delegate; + + private final ToolCallHandler toolCallHandler; + + public ToolCallingAgentResponseHandler(final AgentResponseHandler delegate, final ToolCallHandler toolCallHandler) { + this.delegate = delegate; + this.toolCallHandler = toolCallHandler; + } + + @Override + public boolean handle(final Completion completion, final AgentContext context) { + return this.delegate == null || this.delegate.handle(completion, context); + } + + public boolean handleToolCalls(final Completion completion, final AgentContext context, final ToolCallResults toolCallResults) { + return this.toolCallHandler.handle(completion, context, toolCallResults); + } + +} \ No newline at end of file diff --git a/src/main/java/de/extio/lmlib/client/Client.java b/src/main/java/de/extio/lmlib/client/Client.java index 9014584..7984376 100755 --- a/src/main/java/de/extio/lmlib/client/Client.java +++ b/src/main/java/de/extio/lmlib/client/Client.java @@ -8,13 +8,15 @@ import de.extio.lmlib.profile.ModelProfile.ModelProvider; public interface Client { - Completion conversation(ModelCategory modelCategory, Conversation conversation, boolean skipCache); - - Completion conversation(ModelProfile modelProfile, Conversation conversation, boolean skipCache); + Completion conversation(ModelCategory modelCategory, Conversation conversation, ToolCallData toolCallData, boolean skipCache); - Completion streamConversation(ModelCategory modelCategory, Conversation conversation, Consumer chunkConsumer, boolean skipCache); + Completion conversation(ModelProfile modelProfile, Conversation conversation, ToolCallData toolCallData, boolean skipCache); + + Completion streamConversation(ModelCategory modelCategory, Conversation conversation, Consumer chunkConsumer, ToolCallData toolCallData, boolean skipCache); - Completion streamConversation(ModelProfile modelProfile, Conversation conversation, Consumer chunkConsumer, boolean skipCache); + Completion streamConversation(ModelProfile modelProfile, Conversation conversation, Consumer chunkConsumer, ToolCallData toolCallData, boolean skipCache); ModelProvider getModelProvider(); + + boolean supportsToolCalling(); } diff --git a/src/main/java/de/extio/lmlib/client/ClientService.java b/src/main/java/de/extio/lmlib/client/ClientService.java index 2b3704d..f547277 100755 --- a/src/main/java/de/extio/lmlib/client/ClientService.java +++ b/src/main/java/de/extio/lmlib/client/ClientService.java @@ -31,6 +31,10 @@ public class ClientService { final var modelProfile = this.modelProfileService.getModelProfile(category.getModelProfile(), category); return this.getClient(modelProfile); } + + public boolean supportsToolCalling(final ModelCategory category) { + return this.getClient(category).supportsToolCalling(); + } @Cacheable("clients") public Client getClient(final ModelProfile modelProfile) { @@ -41,6 +45,10 @@ public class ClientService { .map(client -> this.completionInterceptors == null || this.completionInterceptors.isEmpty() ? client : new InterceptingClient(client, this.modelProfileService, this.completionInterceptors)) .orElseThrow(() -> new IllegalStateException("No client found for model profile " + modelProfile.modelName())); } + + public boolean supportsToolCalling(final ModelProfile modelProfile) { + return this.getClient(modelProfile).supportsToolCalling(); + } private static class InterceptingClient implements Client { @@ -57,24 +65,24 @@ public class ClientService { } @Override - public Completion conversation(final ModelCategory modelCategory, final Conversation conversation_, final boolean skipCache) { + public Completion conversation(final ModelCategory modelCategory, final Conversation conversation_, final ToolCallData toolCallData, final boolean skipCache) { final var modelProfile = this.modelProfileService.getModelProfile(modelCategory.getModelProfile(), modelCategory); - return this.conversation(modelProfile, conversation_, skipCache); + return this.conversation(modelProfile, conversation_, toolCallData, skipCache); } @Override - public Completion streamConversation(final ModelCategory modelCategory, final Conversation conversation_, final Consumer chunkConsumer, final boolean skipCache) { + public Completion streamConversation(final ModelCategory modelCategory, final Conversation conversation_, final Consumer chunkConsumer, final ToolCallData toolCallData, final boolean skipCache) { final var modelProfile = this.modelProfileService.getModelProfile(modelCategory.getModelProfile(), modelCategory); - return this.streamConversation(modelProfile, conversation_, chunkConsumer, skipCache); + return this.streamConversation(modelProfile, conversation_, chunkConsumer, toolCallData, skipCache); } @Override - public Completion conversation(final ModelProfile modelProfile, final Conversation conversation_, final boolean skipCache) { + public Completion conversation(final ModelProfile modelProfile, final Conversation conversation_, final ToolCallData toolCallData, final boolean skipCache) { Conversation conversation = conversation_; for (final var interceptor : this.completionInterceptors) { conversation = interceptor.before(modelProfile, conversation); } - var completion = this.client.conversation(modelProfile, conversation, skipCache); + var completion = this.client.conversation(modelProfile, conversation, toolCallData, skipCache); for (final var interceptor : this.completionInterceptors) { completion = interceptor.after(modelProfile, conversation, completion); } @@ -82,12 +90,12 @@ public class ClientService { } @Override - public Completion streamConversation(final ModelProfile modelProfile, final Conversation conversation_, final Consumer chunkConsumer, final boolean skipCache) { + public Completion streamConversation(final ModelProfile modelProfile, final Conversation conversation_, final Consumer chunkConsumer, final ToolCallData toolCallData, final boolean skipCache) { Conversation conversation = conversation_; for (final var interceptor : this.completionInterceptors) { conversation = interceptor.before(modelProfile, conversation); } - var completion = this.client.streamConversation(modelProfile, conversation, chunkConsumer, skipCache); + var completion = this.client.streamConversation(modelProfile, conversation, chunkConsumer, toolCallData, skipCache); for (final var interceptor : this.completionInterceptors) { completion = interceptor.after(modelProfile, conversation, completion); } @@ -98,6 +106,11 @@ public class ClientService { public ModelProvider getModelProvider() { return this.client.getModelProvider(); } + + @Override + public boolean supportsToolCalling() { + return this.client.supportsToolCalling(); + } } diff --git a/src/main/java/de/extio/lmlib/client/Completion.java b/src/main/java/de/extio/lmlib/client/Completion.java index c66c30c..7048fd9 100755 --- a/src/main/java/de/extio/lmlib/client/Completion.java +++ b/src/main/java/de/extio/lmlib/client/Completion.java @@ -1,5 +1,15 @@ package de.extio.lmlib.client; -public final record Completion(String response, String reasoning, CompletionFinishReason finishReason, CompletionStatistics statistics) { +import java.util.List; + +public final record Completion(String response, String reasoning, CompletionFinishReason finishReason, CompletionStatistics statistics, List toolCalls) { + + public Completion { + toolCalls = toolCalls == null ? List.of() : List.copyOf(toolCalls); + } + + public Completion(final String response, final String reasoning, final CompletionFinishReason finishReason, final CompletionStatistics statistics) { + this(response, reasoning, finishReason, statistics, List.of()); + } } diff --git a/src/main/java/de/extio/lmlib/client/CompletionFinishReason.java b/src/main/java/de/extio/lmlib/client/CompletionFinishReason.java index 136b58c..3ba5ab7 100644 --- a/src/main/java/de/extio/lmlib/client/CompletionFinishReason.java +++ b/src/main/java/de/extio/lmlib/client/CompletionFinishReason.java @@ -15,6 +15,11 @@ public enum CompletionFinishReason { /** * Completion was stopped because the content was filtered. */ - CONTENT_FILTERED + CONTENT_FILTERED, + + /** + * Completion stopped because the model emitted one or more tool calls. + */ + TOOL_CALLS } \ No newline at end of file diff --git a/src/main/java/de/extio/lmlib/client/Conversation.java b/src/main/java/de/extio/lmlib/client/Conversation.java index a3957ad..d81e0f1 100755 --- a/src/main/java/de/extio/lmlib/client/Conversation.java +++ b/src/main/java/de/extio/lmlib/client/Conversation.java @@ -80,10 +80,18 @@ public final class Conversation { } public enum TurnType { - SYSTEM, USER, ASSISTANT + SYSTEM, USER, ASSISTANT, TOOL } - public record Turn(TurnType type, String text) { + public record Turn(TurnType type, String text, List toolCalls, String toolCallId) { + + public Turn { + toolCalls = toolCalls == null ? List.of() : List.copyOf(toolCalls); + } + + public Turn(final TurnType type, final String text) { + this(type, text, List.of(), null); + } } } diff --git a/src/main/java/de/extio/lmlib/client/ToolCall.java b/src/main/java/de/extio/lmlib/client/ToolCall.java new file mode 100644 index 0000000..d2e3f62 --- /dev/null +++ b/src/main/java/de/extio/lmlib/client/ToolCall.java @@ -0,0 +1,5 @@ +package de.extio.lmlib.client; + +public record ToolCall(String id, String type, String name, String arguments) { + +} \ No newline at end of file diff --git a/src/main/java/de/extio/lmlib/client/ToolCallData.java b/src/main/java/de/extio/lmlib/client/ToolCallData.java new file mode 100644 index 0000000..908bbd5 --- /dev/null +++ b/src/main/java/de/extio/lmlib/client/ToolCallData.java @@ -0,0 +1,34 @@ +package de.extio.lmlib.client; + +import java.util.Collection; +import java.util.List; + +public record ToolCallData(List tools, String toolChoice, String forcedToolName, Boolean parallelToolCalls) { + + public static final String TOOL_CHOICE_AUTO = "auto"; + + public static final String TOOL_CHOICE_REQUIRED = "required"; + + public static final String TOOL_CHOICE_NONE = "none"; + + public ToolCallData { + tools = tools == null ? List.of() : List.copyOf(tools); + } + + public static ToolCallData auto(final Collection tools) { + return new ToolCallData(List.copyOf(tools), TOOL_CHOICE_AUTO, null, null); + } + + public static ToolCallData required(final Collection tools) { + return new ToolCallData(List.copyOf(tools), TOOL_CHOICE_REQUIRED, null, null); + } + + public static ToolCallData force(final Collection tools, final String forcedToolName) { + return new ToolCallData(List.copyOf(tools), null, forcedToolName, null); + } + + public boolean hasTools() { + return !this.tools.isEmpty(); + } + +} \ No newline at end of file diff --git a/src/main/java/de/extio/lmlib/client/ToolDefinition.java b/src/main/java/de/extio/lmlib/client/ToolDefinition.java new file mode 100644 index 0000000..2d547dc --- /dev/null +++ b/src/main/java/de/extio/lmlib/client/ToolDefinition.java @@ -0,0 +1,26 @@ +package de.extio.lmlib.client; + +import java.util.Map; + +public record ToolDefinition(String name, String description, Map parameters, Boolean strict) { + + public ToolDefinition { + if (name == null || name.isBlank()) { + throw new IllegalArgumentException("Tool name cannot be null or blank"); + } + parameters = parameters == null ? Map.of() : Map.copyOf(parameters); + } + + public ToolDefinition(final String name, final String description, final Map parameters) { + this(name, description, parameters, null); + } + + public ToolDefinition(final String name, final String description, final ToolParameters parameters, final Boolean strict) { + this(name, description, parameters != null ? parameters.map() : null, strict); + } + + public ToolDefinition(final String name, final String description, final ToolParameters parameters) { + this(name, description, parameters != null ? parameters.map() : null, null); + } + +} \ No newline at end of file diff --git a/src/main/java/de/extio/lmlib/client/ToolParameters.java b/src/main/java/de/extio/lmlib/client/ToolParameters.java new file mode 100644 index 0000000..c00fc32 --- /dev/null +++ b/src/main/java/de/extio/lmlib/client/ToolParameters.java @@ -0,0 +1,83 @@ +package de.extio.lmlib.client; + +import java.util.ArrayList; +import java.util.Collection; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; + +import tools.jackson.databind.ObjectMapper; + +public final class ToolParameters { + + private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper(); + + private final LinkedHashMap values = new LinkedHashMap<>(); + + private ToolParameters() { + } + + public static ToolParameters create() { + return new ToolParameters(); + } + + public static ToolParameters create(final Map properties) { + final var schemaProperties = ToolParameters.create(); + for (final var entry : properties.entrySet()) { + schemaProperties.add(entry.getKey(), ToolParameters.create() + .add("type", "string") + .add("description", entry.getValue())); + } + + return new ToolParameters() + .add("type", "object") + .add("properties", schemaProperties) + .add("required", List.copyOf(properties.keySet())) + .add("additionalProperties", false); + } + + public ToolParameters add(final String key, final Object value) { + this.values.put(key, this.normalize(value)); + return this; + } + + public Map map() { + return Map.copyOf(this.values); + } + + public String json() { + try { + return OBJECT_MAPPER.writeValueAsString(this.values); + } + catch (final Exception e) { + throw new IllegalStateException("Cannot serialize tool parameters", e); + } + } + + private Object normalize(final Object value) { + return switch (value) { + case null -> null; + case final ToolParameters toolParameters -> toolParameters.map(); + case final Map map -> this.normalizeMap(map); + case final Collection collection -> this.normalizeCollection(collection); + default -> value; + }; + } + + private Map normalizeMap(final Map map) { + final var normalized = new LinkedHashMap(); + for (final var entry : map.entrySet()) { + normalized.put(String.valueOf(entry.getKey()), this.normalize(entry.getValue())); + } + return normalized; + } + + private List normalizeCollection(final Collection collection) { + final var normalized = new ArrayList(collection.size()); + for (final var item : collection) { + normalized.add(this.normalize(item)); + } + return List.copyOf(normalized); + } + +} \ No newline at end of file diff --git a/src/main/java/de/extio/lmlib/client/cached/CachedClient.java b/src/main/java/de/extio/lmlib/client/cached/CachedClient.java index 8e505fe..088c66f 100644 --- a/src/main/java/de/extio/lmlib/client/cached/CachedClient.java +++ b/src/main/java/de/extio/lmlib/client/cached/CachedClient.java @@ -9,18 +9,22 @@ import java.util.HexFormat; import java.util.function.Consumer; import java.util.function.Supplier; +import tools.jackson.databind.ObjectMapper; + import de.extio.lmlib.client.Chunk; import de.extio.lmlib.client.Client; import de.extio.lmlib.client.Completion; -import de.extio.lmlib.client.CompletionFinishReason; import de.extio.lmlib.client.CompletionStatistics; import de.extio.lmlib.client.Conversation; +import de.extio.lmlib.client.ToolCallData; import de.extio.lmlib.profile.ModelCategory; import de.extio.lmlib.profile.ModelProfile; import de.extio.lmlib.profile.ModelProfile.ModelProvider; import de.extio.lmlib.profile.ModelProfileService; public class CachedClient implements Client { + + private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper(); private final CachedClientRepository cacheRepository; @@ -35,18 +39,18 @@ public class CachedClient implements Client { } @Override - public Completion conversation(final ModelCategory modelCategory, final Conversation conversation, final boolean skipCache) { - return this.getCachedResponse(modelCategory, null, skipCache, () -> this.client.conversation(modelCategory, conversation, skipCache), conversation.getConversation().toString()); + public Completion conversation(final ModelCategory modelCategory, final Conversation conversation, final ToolCallData toolCallData, final boolean skipCache) { + return this.getCachedResponse(modelCategory, null, skipCache, () -> this.client.conversation(modelCategory, conversation, toolCallData, skipCache), conversation.getConversation().toString(), this.serializeToolCallData(toolCallData)); } @Override - public Completion conversation(final ModelProfile modelProfile, final Conversation conversation, final boolean skipCache) { - return this.getCachedResponse(null, modelProfile, skipCache, () -> this.client.conversation(modelProfile, conversation, skipCache), conversation.getConversation().toString()); + public Completion conversation(final ModelProfile modelProfile, final Conversation conversation, final ToolCallData toolCallData, final boolean skipCache) { + return this.getCachedResponse(null, modelProfile, skipCache, () -> this.client.conversation(modelProfile, conversation, toolCallData, skipCache), conversation.getConversation().toString(), this.serializeToolCallData(toolCallData)); } @Override - public Completion streamConversation(final ModelCategory modelCategory, final Conversation conversation, final Consumer chunkConsumer, final boolean skipCache) { - final var response = this.getCachedResponse(modelCategory, null, skipCache, () -> this.client.streamConversation(modelCategory, conversation, chunkConsumer, skipCache), conversation.getConversation().toString()); + public Completion streamConversation(final ModelCategory modelCategory, final Conversation conversation, final Consumer chunkConsumer, final ToolCallData toolCallData, final boolean skipCache) { + final var response = this.getCachedResponse(modelCategory, null, skipCache, () -> this.client.streamConversation(modelCategory, conversation, chunkConsumer, toolCallData, skipCache), conversation.getConversation().toString(), this.serializeToolCallData(toolCallData)); if (response.statistics().cached()) { chunkConsumer.accept(new Chunk(response.response(), response.reasoning())); } @@ -54,8 +58,8 @@ public class CachedClient implements Client { } @Override - public Completion streamConversation(final ModelProfile modelProfile, final Conversation conversation, final Consumer chunkConsumer, final boolean skipCache) { - final var response = this.getCachedResponse(null, modelProfile, skipCache, () -> this.client.streamConversation(modelProfile, conversation, chunkConsumer, skipCache), conversation.getConversation().toString()); + public Completion streamConversation(final ModelProfile modelProfile, final Conversation conversation, final Consumer chunkConsumer, final ToolCallData toolCallData, final boolean skipCache) { + final var response = this.getCachedResponse(null, modelProfile, skipCache, () -> this.client.streamConversation(modelProfile, conversation, chunkConsumer, toolCallData, skipCache), conversation.getConversation().toString(), this.serializeToolCallData(toolCallData)); if (response.statistics().cached()) { chunkConsumer.accept(new Chunk(response.response(), response.reasoning())); } @@ -66,21 +70,38 @@ public class CachedClient implements Client { public ModelProvider getModelProvider() { return this.client.getModelProvider(); } + + @Override + public boolean supportsToolCalling() { + return this.client.supportsToolCalling(); + } private Completion getCachedResponse(final ModelCategory modelCategory, final ModelProfile modelProfile, final boolean skipCache, final Supplier supplier, final String... keys) { final var key = this.createCacheKey(modelCategory, modelProfile, keys); final var cachedCompletion = skipCache ? null : this.cacheRepository.get(key); if (cachedCompletion != null) { - return new Completion(cachedCompletion.response(), cachedCompletion.reasoning(), CompletionFinishReason.DONE, new CompletionStatistics(0, Duration.ZERO, cachedCompletion.inTokens(), cachedCompletion.cachedInTokens(), cachedCompletion.outTokens(), cachedCompletion.reasoningOutTokens(), BigDecimal.ZERO, true)); + return new Completion(cachedCompletion.response(), cachedCompletion.reasoning(), cachedCompletion.finishReason(), new CompletionStatistics(0, Duration.ZERO, cachedCompletion.inTokens(), cachedCompletion.cachedInTokens(), cachedCompletion.outTokens(), cachedCompletion.reasoningOutTokens(), BigDecimal.ZERO, true), cachedCompletion.toolCalls()); } final var completion = supplier.get(); if (completion != null) { - this.cacheRepository.put(key, new CachedCompletion(completion.response(), completion.reasoning(), completion.finishReason(), completion.statistics().inTokens(), completion.statistics().cachedInTokens(), completion.statistics().outTokens(), completion.statistics().reasoningOutTokens(), OffsetDateTime.now())); + this.cacheRepository.put(key, new CachedCompletion(completion.response(), completion.reasoning(), completion.finishReason(), completion.toolCalls(), completion.statistics().inTokens(), completion.statistics().cachedInTokens(), completion.statistics().outTokens(), completion.statistics().reasoningOutTokens(), OffsetDateTime.now())); } return completion; } + + private String serializeToolCallData(final ToolCallData toolCallData) { + if (toolCallData == null) { + return null; + } + try { + return OBJECT_MAPPER.writeValueAsString(toolCallData); + } + catch (final Exception e) { + return toolCallData.toString(); + } + } private String createCacheKey(ModelCategory modelCategory, ModelProfile modelProfile, final String... keys) { if (modelProfile == null) { diff --git a/src/main/java/de/extio/lmlib/client/cached/CachedCompletion.java b/src/main/java/de/extio/lmlib/client/cached/CachedCompletion.java index 240e44c..be5466b 100644 --- a/src/main/java/de/extio/lmlib/client/cached/CachedCompletion.java +++ b/src/main/java/de/extio/lmlib/client/cached/CachedCompletion.java @@ -1,9 +1,11 @@ package de.extio.lmlib.client.cached; import java.time.OffsetDateTime; +import java.util.List; import de.extio.lmlib.client.CompletionFinishReason; +import de.extio.lmlib.client.ToolCall; -public record CachedCompletion(String response, String reasoning, CompletionFinishReason finishReason, long inTokens, long cachedInTokens, long outTokens, long reasoningOutTokens, OffsetDateTime created) { +public record CachedCompletion(String response, String reasoning, CompletionFinishReason finishReason, List toolCalls, long inTokens, long cachedInTokens, long outTokens, long reasoningOutTokens, OffsetDateTime created) { } diff --git a/src/main/java/de/extio/lmlib/client/oai/completion/AbstractCompletionClient.java b/src/main/java/de/extio/lmlib/client/oai/completion/AbstractCompletionClient.java index 05737fa..c2b8c07 100755 --- a/src/main/java/de/extio/lmlib/client/oai/completion/AbstractCompletionClient.java +++ b/src/main/java/de/extio/lmlib/client/oai/completion/AbstractCompletionClient.java @@ -26,6 +26,7 @@ import de.extio.lmlib.client.Client; import de.extio.lmlib.client.Completion; import de.extio.lmlib.client.CompletionFinishReason; import de.extio.lmlib.client.Conversation; +import de.extio.lmlib.client.ToolCallData; import de.extio.lmlib.client.oai.ModelNameSupplier; import de.extio.lmlib.profile.ModelCategory; import de.extio.lmlib.profile.ModelProfile; @@ -82,37 +83,38 @@ public abstract class AbstractCompletionClient implements Client, DisposableBean } @Override - public Completion conversation(final ModelCategory modelCategory, final Conversation conversation, final boolean skipCache) { - return this.streamConversation(modelCategory, conversation, null, skipCache); + public Completion conversation(final ModelCategory modelCategory, final Conversation conversation, final ToolCallData toolCallData, final boolean skipCache) { + return this.streamConversation(modelCategory, conversation, null, toolCallData, skipCache); } @Override - public Completion conversation(final ModelProfile modelProfile, final Conversation conversation, final boolean skipCache) { - return this.streamConversation(modelProfile, conversation, null, skipCache); + public Completion conversation(final ModelProfile modelProfile, final Conversation conversation, final ToolCallData toolCallData, final boolean skipCache) { + return this.streamConversation(modelProfile, conversation, null, toolCallData, skipCache); } @Override - public Completion streamConversation(final ModelCategory modelCategory_, final Conversation conversation, final Consumer chunkConsumer, final boolean skipCache) { + public Completion streamConversation(final ModelCategory modelCategory_, final Conversation conversation, final Consumer chunkConsumer, final ToolCallData toolCallData, final boolean skipCache) { final ModelCategory modelCategory = (modelCategory_ == null) ? ModelCategory.MEDIUM : modelCategory_; final var modelProfile = this.modelProfileService.getModelProfile(modelCategory.getModelProfile(), modelCategory); - return this.streamConversation(modelProfile, conversation, chunkConsumer, skipCache); + return this.streamConversation(modelProfile, conversation, chunkConsumer, toolCallData, skipCache); } @Override - public Completion streamConversation(final ModelProfile modelProfile, final Conversation conversation, final Consumer chunkConsumer, final boolean skipCache) { + public Completion streamConversation(final ModelProfile modelProfile, final Conversation conversation, final Consumer chunkConsumer, final ToolCallData toolCallData, final boolean skipCache) { if (modelProfile == null || (modelProfile.modelProvider() != ModelProvider.OAI_TEXT_COMPLETION && modelProfile.modelProvider() != ModelProvider.OAI_CHAT_COMPLETION)) { throw new IllegalArgumentException("Invalid ModelProfile"); } - return this.requestCompletion(conversation, modelProfile, chunkConsumer); + return this.requestCompletion(conversation, modelProfile, chunkConsumer, toolCallData); } - protected abstract Completion requestCompletion(final Conversation conversation, final ModelProfile modelProfile, final Consumer chunkConsumer); + protected abstract Completion requestCompletion(final Conversation conversation, final ModelProfile modelProfile, final Consumer chunkConsumer, final ToolCallData toolCallData); protected CompletionFinishReason mapFinishReason(final String finishReason) { return switch (finishReason) { case FinishReasons.FINISH_REASON_CONTENT_FILTER -> CompletionFinishReason.CONTENT_FILTERED; case FinishReasons.FINISH_REASON_LENGTH -> CompletionFinishReason.TOKEN_LIMIT_REACHED; + case FinishReasons.FINISH_REASON_TOOL_CALLS -> CompletionFinishReason.TOOL_CALLS; default -> CompletionFinishReason.DONE; }; } diff --git a/src/main/java/de/extio/lmlib/client/oai/completion/FinishReasons.java b/src/main/java/de/extio/lmlib/client/oai/completion/FinishReasons.java index 3118afd..df39a16 100644 --- a/src/main/java/de/extio/lmlib/client/oai/completion/FinishReasons.java +++ b/src/main/java/de/extio/lmlib/client/oai/completion/FinishReasons.java @@ -7,5 +7,7 @@ public final class FinishReasons { public static final String FINISH_REASON_LENGTH = "length"; public static final String FINISH_REASON_CONTENT_FILTER = "content_filter"; + + public static final String FINISH_REASON_TOOL_CALLS = "tool_calls"; } diff --git a/src/main/java/de/extio/lmlib/client/oai/completion/chat/ChatCompletionClient.java b/src/main/java/de/extio/lmlib/client/oai/completion/chat/ChatCompletionClient.java index 901c390..38b916b 100755 --- a/src/main/java/de/extio/lmlib/client/oai/completion/chat/ChatCompletionClient.java +++ b/src/main/java/de/extio/lmlib/client/oai/completion/chat/ChatCompletionClient.java @@ -4,23 +4,24 @@ import java.io.BufferedReader; import java.io.InputStreamReader; import java.nio.charset.StandardCharsets; import java.time.LocalDateTime; -import java.util.ArrayDeque; import java.util.ArrayList; -import java.util.Deque; import java.util.List; +import java.util.Objects; import java.util.concurrent.atomic.AtomicInteger; import java.util.function.Consumer; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.springframework.http.HttpMethod; -import org.springframework.web.client.RestClient; import tools.jackson.core.JacksonException; import de.extio.lmlib.client.Chunk; import de.extio.lmlib.client.Completion; import de.extio.lmlib.client.Conversation; +import de.extio.lmlib.client.ToolCall; +import de.extio.lmlib.client.ToolCallData; +import de.extio.lmlib.client.ToolDefinition; import de.extio.lmlib.client.Conversation.Turn; import de.extio.lmlib.client.oai.completion.AbstractCompletionClient; import de.extio.lmlib.profile.ModelProfile; @@ -34,13 +35,18 @@ public class ChatCompletionClient extends AbstractCompletionClient { super(); } + @Override + public boolean supportsToolCalling() { + return true; + } + @Override public ModelProvider getModelProvider() { return ModelProvider.OAI_CHAT_COMPLETION; } @Override - protected Completion requestCompletion(final Conversation conversation, final ModelProfile modelProfile, final Consumer chunkConsumer) { + protected Completion requestCompletion(final Conversation conversation, final ModelProfile modelProfile, final Consumer chunkConsumer, final ToolCallData toolCallData) { final var chat = createChats(conversation, modelProfile); final var request = new ChatCompletionRequest(); @@ -76,6 +82,11 @@ public class ChatCompletionClient extends AbstractCompletionClient { throw new IllegalArgumentException("Invalid reasoning summary details in the model profile: " + modelProfile.reasoningSummaryDetails(), e); } } + if (toolCallData != null && toolCallData.hasTools()) { + request.setTools(this.createTools(toolCallData.tools())); + request.setToolChoice(this.createToolChoice(toolCallData)); + request.setParallelToolCalls(toolCallData.parallelToolCalls()); + } String requestBody; try { @@ -112,6 +123,7 @@ public class ChatCompletionClient extends AbstractCompletionClient { chatCompletionResponse.setChoices(List.of(new ChatChoice())); final StringBuilder contentStringBuilder = new StringBuilder(); final StringBuilder reasoningStringBuilder = new StringBuilder(); + final List toolCalls = new ArrayList<>(); try (BufferedReader reader = new BufferedReader(new InputStreamReader(clientResponse.getBody(), StandardCharsets.UTF_8))) { String line; @@ -162,6 +174,9 @@ public class ChatCompletionClient extends AbstractCompletionClient { if (content != null || reasoning != null) { chunkConsumer.accept(new Chunk(content, reasoning)); } + if (streamResponse.getChoices().getFirst().getMessage().getToolCalls() != null && !streamResponse.getChoices().getFirst().getMessage().getToolCalls().isEmpty()) { + this.mergeToolCalls(toolCalls, streamResponse.getChoices().getFirst().getMessage().getToolCalls()); + } } } } @@ -171,6 +186,7 @@ public class ChatCompletionClient extends AbstractCompletionClient { } } chatCompletionResponse.getChoices().getFirst().setMessage(new ChatMessage("assistant", contentStringBuilder.toString(), reasoningStringBuilder.toString())); + chatCompletionResponse.getChoices().getFirst().getMessage().setToolCalls(List.copyOf(toolCalls)); return chatCompletionResponse; } }); @@ -181,65 +197,118 @@ public class ChatCompletionClient extends AbstractCompletionClient { final var choice = response.getChoices().getFirst(); final var content = response.getChoices().getFirst().getMessage().getContent(); final var reasoning = response.getChoices().getFirst().getMessage().getReasoningContent(); + final var toolCalls = this.toToolCalls(response.getChoices().getFirst().getMessage().getToolCalls()); + if ((content == null || content.isBlank()) && toolCalls.isEmpty()) { + throw new IllegalStateException("No parseable response content from server"); + } final var finishReason = this.mapFinishReason(choice.getFinishReason()); final var statistics = createCompletionStatistics(modelProfile, start, response.getUsage(), response.getTimings(), null, content, reasoning); - return new Completion(content, reasoning, finishReason, statistics); + return new Completion(content, reasoning, finishReason, statistics, toolCalls); + } + + private List createTools(final List toolDefinitions) { + return toolDefinitions.stream().map(toolDefinition -> { + final var function = new ChatCompletionRequest.ChatCompletionFunction(); + function.setName(toolDefinition.name()); + function.setDescription(toolDefinition.description()); + function.setParameters(toolDefinition.parameters()); + function.setStrict(toolDefinition.strict()); + + final var tool = new ChatCompletionRequest.ChatCompletionTool(); + tool.setType("function"); + tool.setFunction(function); + return tool; + }).toList(); + } + + private Object createToolChoice(final ToolCallData toolCallData) { + if (toolCallData == null) { + return null; + } + if (toolCallData.forcedToolName() != null && !toolCallData.forcedToolName().isBlank()) { + final var function = new ChatCompletionRequest.ChatCompletionNamedToolChoiceFunction(); + function.setName(toolCallData.forcedToolName()); + + final var namedToolChoice = new ChatCompletionRequest.ChatCompletionNamedToolChoice(); + namedToolChoice.setType("function"); + namedToolChoice.setFunction(function); + return namedToolChoice; + } + return toolCallData.toolChoice(); + } + + private void mergeToolCalls(final List currentToolCalls, final List deltaToolCalls) { + for (final var deltaToolCall : deltaToolCalls) { + final int index = deltaToolCall.getIndex() != null ? deltaToolCall.getIndex() : currentToolCalls.size(); + while (currentToolCalls.size() <= index) { + currentToolCalls.add(new ChatToolCall()); + } + final var currentToolCall = currentToolCalls.get(index); + if (deltaToolCall.getId() != null && !deltaToolCall.getId().isBlank()) { + currentToolCall.setId(deltaToolCall.getId()); + } + if (deltaToolCall.getType() != null && !deltaToolCall.getType().isBlank()) { + currentToolCall.setType(deltaToolCall.getType()); + } + if (deltaToolCall.getFunction() != null) { + if (currentToolCall.getFunction() == null) { + currentToolCall.setFunction(new ChatToolFunction()); + } + if (deltaToolCall.getFunction().getName() != null && !deltaToolCall.getFunction().getName().isBlank()) { + currentToolCall.getFunction().setName(deltaToolCall.getFunction().getName()); + } + if (deltaToolCall.getFunction().getArguments() != null && !deltaToolCall.getFunction().getArguments().isEmpty()) { + final var existingArguments = currentToolCall.getFunction().getArguments(); + currentToolCall.getFunction().setArguments((existingArguments == null ? "" : existingArguments) + deltaToolCall.getFunction().getArguments()); + } + } + } + } + + private List toToolCalls(final List chatToolCalls) { + if (chatToolCalls == null || chatToolCalls.isEmpty()) { + return List.of(); + } + return chatToolCalls.stream() + .filter(Objects::nonNull) + .map(toolCall -> new ToolCall(toolCall.getId(), toolCall.getType(), toolCall.getFunction() != null ? toolCall.getFunction().getName() : null, toolCall.getFunction() != null ? toolCall.getFunction().getArguments() : null)) + .toList(); } private List createChats(final Conversation conversation, final ModelProfile modelProfile) { final List chat = new ArrayList<>(); - - final Deque turns = new ArrayDeque<>(conversation.getConversation()); - Turn systemTurn = null; - Turn userTurn = null; - Turn assistantTurn = null; - boolean first = true; - final AtomicInteger tokens = new AtomicInteger(); - - while (!turns.isEmpty()) { - final Turn turn = turns.pop(); + + for (final Turn turn : conversation.getConversation()) { switch (turn.type()) { case SYSTEM: - systemTurn = turn; + chat.add(this.createChatMessage("system", turn)); break; case USER: - userTurn = turn; + if (this.addChatRequestMessage("user", turn, tokens, chat, modelProfile)) { + return chat; + } break; case ASSISTANT: - assistantTurn = turn; + if (this.addChatRequestMessage("assistant", turn, tokens, chat, modelProfile)) { + return chat; + } + break; + case TOOL: + if (this.addChatRequestMessage("tool", turn, tokens, chat, modelProfile)) { + return chat; + } break; default: throw new IllegalArgumentException("Unknown turn type: " + turn.type()); } - - if (first && userTurn != null) { - if (systemTurn != null) { - chat.add(new ChatMessage("system", systemTurn.text(), null)); - } - if (this.addChatRequestMessage("user", userTurn.text(), tokens, chat, modelProfile)) { - break; - } - userTurn = null; - first = false; - } - else if (!first && assistantTurn != null && userTurn != null) { - if (this.addChatRequestMessage("assistant", assistantTurn.text(), tokens, chat, modelProfile)) { - break; - } - if (this.addChatRequestMessage("user", userTurn.text(), tokens, chat, modelProfile)) { - break; - } - userTurn = null; - assistantTurn = null; - } } - + return chat; } - private boolean addChatRequestMessage(final String role, final String text, final AtomicInteger tokens, final List chat, final ModelProfile modelProfile) { - var prompt = text; + private boolean addChatRequestMessage(final String role, final Turn turn, final AtomicInteger tokens, final List chat, final ModelProfile modelProfile) { + var prompt = turn.text() != null ? turn.text() : ""; var trimmed = false; final var textTokens = this.tokenizer.count(prompt, modelProfile); @@ -252,9 +321,30 @@ public class ChatCompletionClient extends AbstractCompletionClient { trimmed = true; } - chat.add(new ChatMessage(role, prompt, null)); + final var message = this.createChatMessage(role, turn); + message.setContent(prompt); + chat.add(message); return trimmed; } + + private ChatMessage createChatMessage(final String role, final Turn turn) { + final var message = new ChatMessage(role, turn.text() != null ? turn.text() : "", null); + if (!turn.toolCalls().isEmpty()) { + message.setToolCalls(turn.toolCalls().stream().map(toolCall -> { + final var function = new ChatToolFunction(); + function.setName(toolCall.name()); + function.setArguments(toolCall.arguments()); + + final var chatToolCall = new ChatToolCall(); + chatToolCall.setId(toolCall.id()); + chatToolCall.setType(toolCall.type()); + chatToolCall.setFunction(function); + return chatToolCall; + }).toList()); + } + message.setToolCallId(turn.toolCallId()); + return message; + } } diff --git a/src/main/java/de/extio/lmlib/client/oai/completion/chat/ChatCompletionRequest.java b/src/main/java/de/extio/lmlib/client/oai/completion/chat/ChatCompletionRequest.java index 613ef61..aa0437e 100755 --- a/src/main/java/de/extio/lmlib/client/oai/completion/chat/ChatCompletionRequest.java +++ b/src/main/java/de/extio/lmlib/client/oai/completion/chat/ChatCompletionRequest.java @@ -1,6 +1,7 @@ package de.extio.lmlib.client.oai.completion.chat; import java.util.List; +import java.util.Map; import com.fasterxml.jackson.annotation.*; @@ -13,6 +14,12 @@ final class ChatCompletionRequest extends AbstractCompletionRequest { private Integer maxCompletionTokens; private ChatCompletionsRequestReasoning reasoning; + + private List tools; + + private Object toolChoice; + + private Boolean parallelToolCalls; @JsonProperty("messages") public List getMessages() { @@ -41,6 +48,33 @@ final class ChatCompletionRequest extends AbstractCompletionRequest { this.reasoning = reasoning; } + @JsonProperty("tools") + public List getTools() { + return this.tools; + } + + public void setTools(final List tools) { + this.tools = tools; + } + + @JsonProperty("tool_choice") + public Object getToolChoice() { + return this.toolChoice; + } + + public void setToolChoice(final Object toolChoice) { + this.toolChoice = toolChoice; + } + + @JsonProperty("parallel_tool_calls") + public Boolean getParallelToolCalls() { + return this.parallelToolCalls; + } + + public void setParallelToolCalls(final Boolean parallelToolCalls) { + this.parallelToolCalls = parallelToolCalls; + } + final static class ChatCompletionsRequestReasoning { private ChatCompletionsRequestReasoningEffort effort; @@ -75,4 +109,119 @@ final class ChatCompletionRequest extends AbstractCompletionRequest { auto, concise, detailed } + static final class ChatCompletionTool { + + private String type; + + private ChatCompletionFunction function; + + @JsonProperty("type") + public String getType() { + return this.type; + } + + public void setType(final String type) { + this.type = type; + } + + @JsonProperty("function") + public ChatCompletionFunction getFunction() { + return this.function; + } + + public void setFunction(final ChatCompletionFunction function) { + this.function = function; + } + + } + + static final class ChatCompletionFunction { + + private String name; + + private String description; + + private Map parameters; + + private Boolean strict; + + @JsonProperty("name") + public String getName() { + return this.name; + } + + public void setName(final String name) { + this.name = name; + } + + @JsonProperty("description") + public String getDescription() { + return this.description; + } + + public void setDescription(final String description) { + this.description = description; + } + + @JsonProperty("parameters") + public Map getParameters() { + return this.parameters; + } + + public void setParameters(final Map parameters) { + this.parameters = parameters; + } + + @JsonProperty("strict") + public Boolean getStrict() { + return this.strict; + } + + public void setStrict(final Boolean strict) { + this.strict = strict; + } + + } + + static final class ChatCompletionNamedToolChoice { + + private String type; + + private ChatCompletionNamedToolChoiceFunction function; + + @JsonProperty("type") + public String getType() { + return this.type; + } + + public void setType(final String type) { + this.type = type; + } + + @JsonProperty("function") + public ChatCompletionNamedToolChoiceFunction getFunction() { + return this.function; + } + + public void setFunction(final ChatCompletionNamedToolChoiceFunction function) { + this.function = function; + } + + } + + static final class ChatCompletionNamedToolChoiceFunction { + + private String name; + + @JsonProperty("name") + public String getName() { + return this.name; + } + + public void setName(final String name) { + this.name = name; + } + + } + } diff --git a/src/main/java/de/extio/lmlib/client/oai/completion/chat/ChatMessage.java b/src/main/java/de/extio/lmlib/client/oai/completion/chat/ChatMessage.java index 81b3b4f..5c8c457 100644 --- a/src/main/java/de/extio/lmlib/client/oai/completion/chat/ChatMessage.java +++ b/src/main/java/de/extio/lmlib/client/oai/completion/chat/ChatMessage.java @@ -1,5 +1,7 @@ package de.extio.lmlib.client.oai.completion.chat; +import java.util.List; + import com.fasterxml.jackson.annotation.*; final class ChatMessage { @@ -9,6 +11,10 @@ final class ChatMessage { private String content; private String reasoningContent; + + private List toolCalls; + + private String toolCallId; public ChatMessage() { } @@ -45,4 +51,22 @@ final class ChatMessage { public void setReasoningContent(final String reasoningContent) { this.reasoningContent = reasoningContent; } + + @JsonProperty("tool_calls") + public List getToolCalls() { + return this.toolCalls; + } + + public void setToolCalls(final List toolCalls) { + this.toolCalls = toolCalls; + } + + @JsonProperty("tool_call_id") + public String getToolCallId() { + return this.toolCallId; + } + + public void setToolCallId(final String toolCallId) { + this.toolCallId = toolCallId; + } } diff --git a/src/main/java/de/extio/lmlib/client/oai/completion/chat/ChatToolCall.java b/src/main/java/de/extio/lmlib/client/oai/completion/chat/ChatToolCall.java new file mode 100644 index 0000000..5e039de --- /dev/null +++ b/src/main/java/de/extio/lmlib/client/oai/completion/chat/ChatToolCall.java @@ -0,0 +1,51 @@ +package de.extio.lmlib.client.oai.completion.chat; + +import com.fasterxml.jackson.annotation.JsonProperty; + +final class ChatToolCall { + + private Integer index; + + private String id; + + private String type; + + private ChatToolFunction function; + + @JsonProperty("index") + public Integer getIndex() { + return this.index; + } + + public void setIndex(final Integer index) { + this.index = index; + } + + @JsonProperty("id") + public String getId() { + return this.id; + } + + public void setId(final String id) { + this.id = id; + } + + @JsonProperty("type") + public String getType() { + return this.type; + } + + public void setType(final String type) { + this.type = type; + } + + @JsonProperty("function") + public ChatToolFunction getFunction() { + return this.function; + } + + public void setFunction(final ChatToolFunction function) { + this.function = function; + } + +} \ No newline at end of file diff --git a/src/main/java/de/extio/lmlib/client/oai/completion/chat/ChatToolFunction.java b/src/main/java/de/extio/lmlib/client/oai/completion/chat/ChatToolFunction.java new file mode 100644 index 0000000..41c686e --- /dev/null +++ b/src/main/java/de/extio/lmlib/client/oai/completion/chat/ChatToolFunction.java @@ -0,0 +1,29 @@ +package de.extio.lmlib.client.oai.completion.chat; + +import com.fasterxml.jackson.annotation.JsonProperty; + +final class ChatToolFunction { + + private String name; + + private String arguments; + + @JsonProperty("name") + public String getName() { + return this.name; + } + + public void setName(final String name) { + this.name = name; + } + + @JsonProperty("arguments") + public String getArguments() { + return this.arguments; + } + + public void setArguments(final String arguments) { + this.arguments = arguments; + } + +} \ No newline at end of file diff --git a/src/main/java/de/extio/lmlib/client/oai/completion/text/TextCompletionClient.java b/src/main/java/de/extio/lmlib/client/oai/completion/text/TextCompletionClient.java index abd0552..b8b7d7c 100755 --- a/src/main/java/de/extio/lmlib/client/oai/completion/text/TextCompletionClient.java +++ b/src/main/java/de/extio/lmlib/client/oai/completion/text/TextCompletionClient.java @@ -13,7 +13,6 @@ import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.http.HttpMethod; -import org.springframework.web.client.RestClient; import tools.jackson.core.JacksonException; @@ -21,6 +20,7 @@ import de.extio.lmlib.client.Chunk; import de.extio.lmlib.client.Completion; import de.extio.lmlib.client.CompletionFinishReason; import de.extio.lmlib.client.Conversation; +import de.extio.lmlib.client.ToolCallData; import de.extio.lmlib.client.Conversation.Turn; import de.extio.lmlib.client.oai.completion.AbstractCompletionClient; import de.extio.lmlib.profile.ModelProfile; @@ -41,7 +41,12 @@ public class TextCompletionClient extends AbstractCompletionClient { } @Override - protected Completion requestCompletion(final Conversation conversation, final ModelProfile modelProfile, final Consumer chunkConsumer) { + public boolean supportsToolCalling() { + return false; + } + + @Override + protected Completion requestCompletion(final Conversation conversation, final ModelProfile modelProfile, final Consumer chunkConsumer, final ToolCallData toolCallData) { final var promptStrategy = this.promptStrategyFactory.getStrategy(modelProfile.prompt()); if (promptStrategy == null) { throw new IllegalArgumentException("Prompt strategy not found: " + modelProfile.prompt()); diff --git a/src/main/java/de/extio/lmlib/grader/Grader.java b/src/main/java/de/extio/lmlib/grader/Grader.java index 93d5811..f281970 100644 --- a/src/main/java/de/extio/lmlib/grader/Grader.java +++ b/src/main/java/de/extio/lmlib/grader/Grader.java @@ -36,6 +36,7 @@ public class Grader { Only provide the response as a single word and no preamble and no explanation.""", "Here is the text:\n" + text + "\n\n" + "Here is the user question: " + question), + null, true); LOGGER.info(completion.response()); score += Boolean.parseBoolean(completion.response().strip()) ? 1 : -1; diff --git a/src/main/java/de/extio/lmlib/grader/Grader2.java b/src/main/java/de/extio/lmlib/grader/Grader2.java index 80b5284..ec2fb2a 100644 --- a/src/main/java/de/extio/lmlib/grader/Grader2.java +++ b/src/main/java/de/extio/lmlib/grader/Grader2.java @@ -50,6 +50,7 @@ public class Grader2 { - 'false': The text does not provide a truthful answer. Only provide the response as a single word. Do not include any preamble or explanation. """.formatted(question, text)), + null, true); LOGGER.info(completion.response()); score += Boolean.parseBoolean(completion.response().strip()) ? 1 : -1; diff --git a/src/main/java/de/extio/lmlib/grader/agent/AnswerGrader.java b/src/main/java/de/extio/lmlib/grader/agent/AnswerGrader.java index 8126334..ff993ff 100755 --- a/src/main/java/de/extio/lmlib/grader/agent/AnswerGrader.java +++ b/src/main/java/de/extio/lmlib/grader/agent/AnswerGrader.java @@ -30,6 +30,7 @@ public class AnswerGrader { The question is: {{questions}} The answer is: {{answer}} """, + null, new JsonAgentResponseHandler(), context -> { context.setSkipCache(true); diff --git a/src/main/java/de/extio/lmlib/grader/agent/AnswerGrader2.java b/src/main/java/de/extio/lmlib/grader/agent/AnswerGrader2.java index 4c87b78..4e16196 100755 --- a/src/main/java/de/extio/lmlib/grader/agent/AnswerGrader2.java +++ b/src/main/java/de/extio/lmlib/grader/agent/AnswerGrader2.java @@ -42,6 +42,7 @@ public class AnswerGrader2 { ### Output Format {{qaOutputFormat}} """, + null, new JsonAgentResponseHandler(), context -> { final Boolean outputDescription = context.getValue("qaOutputDescription", Boolean.class); diff --git a/src/main/java/de/extio/lmlib/grader/agent/AnswerGraderBinary.java b/src/main/java/de/extio/lmlib/grader/agent/AnswerGraderBinary.java index 3c3bc9c..1206292 100644 --- a/src/main/java/de/extio/lmlib/grader/agent/AnswerGraderBinary.java +++ b/src/main/java/de/extio/lmlib/grader/agent/AnswerGraderBinary.java @@ -33,6 +33,7 @@ public class AnswerGraderBinary { Output Format: {{qaOutputFormat}} """, + null, new JsonAgentResponseHandler(), context -> { final Boolean outputDescription = context.getValue("qaOutputDescription", Boolean.class); diff --git a/src/test/java/de/extio/lmlib/agent/AgentErrorHandlingTest.java b/src/test/java/de/extio/lmlib/agent/AgentErrorHandlingTest.java index 831920a..8193ed0 100644 --- a/src/test/java/de/extio/lmlib/agent/AgentErrorHandlingTest.java +++ b/src/test/java/de/extio/lmlib/agent/AgentErrorHandlingTest.java @@ -6,21 +6,32 @@ import static org.junit.jupiter.api.Assertions.assertNull; import static org.junit.jupiter.api.Assertions.assertTrue; import static org.mockito.ArgumentMatchers.any; import static org.mockito.ArgumentMatchers.anyBoolean; +import static org.mockito.Mockito.doAnswer; import static org.mockito.Mockito.mock; import static org.mockito.Mockito.when; import java.util.Map; import java.math.BigDecimal; import java.time.Duration; +import java.util.List; import java.util.concurrent.ExecutorService; import java.util.concurrent.Executors; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.atomic.AtomicReference; import org.junit.jupiter.api.Test; import de.extio.lmlib.agent.responsehandler.AgentResponseHandler; +import de.extio.lmlib.agent.responsehandler.TextAgentResponseHandler; import de.extio.lmlib.client.Client; import de.extio.lmlib.client.Completion; +import de.extio.lmlib.client.CompletionFinishReason; import de.extio.lmlib.client.CompletionStatistics; +import de.extio.lmlib.client.Conversation; +import de.extio.lmlib.client.ToolCall; +import de.extio.lmlib.client.ToolCallData; +import de.extio.lmlib.client.ToolDefinition; +import de.extio.lmlib.client.ToolParameters; import de.extio.lmlib.profile.ModelCategory; class AgentErrorHandlingTest { @@ -28,7 +39,7 @@ class AgentErrorHandlingTest { @Test void parsingErrorsKeepTheirTypeWithoutAnExceptionWhenTheHandlerReturnsFalse() { final Client client = mock(Client.class); - when(client.conversation(any(ModelCategory.class), any(), anyBoolean())).thenReturn(new Completion("not json", null, null, new CompletionStatistics(1, Duration.ofMillis(1), 1, 0, 1, 0, BigDecimal.ZERO, false))); + when(client.conversation(any(ModelCategory.class), any(), any(ToolCallData.class), anyBoolean())).thenReturn(new Completion("not json", null, null, new CompletionStatistics(1, Duration.ofMillis(1), 1, 0, 1, 0, BigDecimal.ZERO, false))); final AgentResponseHandler responseHandler = (completion, context) -> false; final var agent = new Agent("ParseAgent", @@ -37,6 +48,7 @@ class AgentErrorHandlingTest { null, null, "prompt", + null, responseHandler, null, null, @@ -63,6 +75,7 @@ class AgentErrorHandlingTest { null, null, null, + null, context -> { throw exception; }, @@ -80,4 +93,109 @@ class AgentErrorHandlingTest { assertEquals("boom", result.getErrorException().getMessage()); } } + + @Test + void toolCallDataIsRetainedForFollowUpConversationRequests() { + final var client = mock(Client.class); + when(client.supportsToolCalling()).thenReturn(true); + final var requestCounter = new AtomicInteger(); + final var firstRequestToolCallData = new AtomicReference(); + final var secondRequestToolCallData = new AtomicReference(); + final var weatherTool = new ToolDefinition( + "get_weather", + "Gets the current weather for a given location.", + ToolParameters.create(Map.of("location", "")), + true); + doAnswer(invocation -> { + final var toolCallData = invocation.getArgument(2, ToolCallData.class); + final var currentRequest = requestCounter.incrementAndGet(); + if (currentRequest == 1) { + firstRequestToolCallData.set(toolCallData); + return new Completion( + "", + null, + CompletionFinishReason.TOOL_CALLS, + new CompletionStatistics(1, Duration.ofMillis(1), 1, 0, 1, 0, BigDecimal.ZERO, false), + List.of(new ToolCall("call_1", "function", "get_weather", "{\"location\":\"Berlin, Germany\"}"))); + } + secondRequestToolCallData.set(toolCallData); + return new Completion("Berlin is rainy and 21C", null, CompletionFinishReason.DONE, new CompletionStatistics(1, Duration.ofMillis(1), 1, 0, 1, 0, BigDecimal.ZERO, false)); + }).when(client).conversation(any(ModelCategory.class), any(), any(ToolCallData.class), anyBoolean()); + + final BaseAgent toolCaller = new BaseAgent() { + + @Override + public String name() { + return "ToolCaller"; + } + + @Override + public AgentType agentType(final AgentContext context) { + return AgentType.START_CONVERSATION; + } + + @Override + public ModelCategory modelCategory(final AgentContext context) { + return ModelCategory.MEDIUM; + } + + @Override + public String systemPrompt() { + return "You must call the weather tool."; + } + + @Override + public String textTemplate() { + return "What is the weather in Berlin, Germany?"; + } + + @Override + public List toolDefinitions(final AgentContext context) { + return List.of(weatherTool); + } + + @Override + public ToolCallData toolCallData(final AgentContext context) { + return ToolCallData.required(List.of(weatherTool)); + } + + @Override + public AgentResponseHandler responseHandler(final AgentContext context) { + return (completion, agentContext) -> true; + } + + @Override + public AgentNext chooseNextAgent(final AgentContext context) { + return new AgentNext("ToolResultResponder", null); + } + }; + final var toolResultResponder = new Agent("ToolResultResponder", + AgentType.CONVERSATION, + ModelCategory.MEDIUM, + null, + null, + "Answer using the tool result already present in the conversation.", + null, + new TextAgentResponseHandler("answer"), + context -> { + final var assistantTurn = context.getConversation().getConversation().getLast(); + context.getConversation().addTurn(new Conversation.Turn(Conversation.TurnType.TOOL, ToolParameters.create().add("forecast", "Berlin is rainy and 21C").json(), null, assistantTurn.toolCalls().getFirst().id())); + }, + null, + null, + AgentNext::end); + final var context = new AgentContext(Map.of( + toolCaller.name(), toolCaller, + toolResultResponder.name(), toolResultResponder)); + + try (ExecutorService executorService = Executors.newVirtualThreadPerTaskExecutor()) { + final var firstResult = toolCaller.execute(client, executorService, context).getFirst(); + final var secondResult = toolResultResponder.execute(client, executorService, firstResult).getFirst(); + + assertEquals(2, requestCounter.get()); + assertEquals(ToolCallData.required(List.of(weatherTool)), firstRequestToolCallData.get()); + assertEquals(ToolCallData.auto(List.of(weatherTool)), secondRequestToolCallData.get()); + assertEquals("Berlin is rainy and 21C", secondResult.getStringValue("answer")); + } + } } \ No newline at end of file diff --git a/src/test/java/de/extio/lmlib/agent/AgentTest.java b/src/test/java/de/extio/lmlib/agent/AgentTest.java index a30507a..e50a8c0 100644 --- a/src/test/java/de/extio/lmlib/agent/AgentTest.java +++ b/src/test/java/de/extio/lmlib/agent/AgentTest.java @@ -24,9 +24,15 @@ import org.springframework.context.annotation.PropertySource; import de.extio.lmlib.agent.responsehandler.JsonAgentResponseHandler; import de.extio.lmlib.agent.responsehandler.TextAgentResponseHandler; +import de.extio.lmlib.agent.responsehandler.ToolCallingAgentResponseHandler; import de.extio.lmlib.client.ClientService; -import de.extio.lmlib.grader.Grader; +import de.extio.lmlib.client.Conversation; +import de.extio.lmlib.client.ToolCallData; +import de.extio.lmlib.client.ToolDefinition; +import de.extio.lmlib.client.ToolParameters; +import de.extio.lmlib.grader.Grader2; import de.extio.lmlib.profile.ModelCategory; +import de.extio.lmlib.profile.ModelProfileService; /** * These tests have been verified with Llama 3.1 8B, Gemma2 27B, Gemma3 27B and GPT-OSS models @@ -45,6 +51,9 @@ public class AgentTest { @Autowired private ClientService clientService; + + @Autowired + private ModelProfileService modelProfileService; @Test void agenticFlow() throws IOException { @@ -56,6 +65,7 @@ public class AgentTest { null, "Generate a summary of the following Java source code.", "The source code is:\n{{code}}", + null, new TextAgentResponseHandler("summary"), null, null, @@ -69,6 +79,7 @@ public class AgentTest { null, "", "Generate a list of 5 distinct functional features. In scope is business logic, not code mechanics. Provide no preamble and no explanation. Return the response in JSON format with the following field: { \"features\": [\"Feature 1\", \"Feature 2\", ... ] }", + null, new JsonAgentResponseHandler(), null, context -> context.getStringValues("features").forEach(feature -> LOGGER.info(feature)), @@ -85,6 +96,7 @@ public class AgentTest { The source code is: {{code}} The feature is: [[features]]""", + null, new TextAgentResponseHandler("featureDescription"), null, null, @@ -102,6 +114,7 @@ public class AgentTest { The code analysis: {{summary}} {{featureDescription}}""", + null, new TextAgentResponseHandler("blogPost"), context -> { context.setStringValues("topics", List.of(String.join(", ", context.getStringValues("features").stream().toArray(size -> new String[size])))); @@ -123,7 +136,7 @@ public class AgentTest { assertEquals(1, resultContext.getContext().get("blogPost").size()); assertEquals(8, resultContext.getRequestStatistic().getRequests().get()); for (final var feature : resultContext.getContext().get("features")) { - assertTrue(Grader.assessScoreBinary("Does the following blog post mention the feature " + feature, resultContext.getStringValue("blogPost"), this.clientService)); + assertTrue(Grader2.assessScoreBinary("Does the following blog post mention the feature " + feature, resultContext.getStringValue("blogPost"), ModelCategory.MEDIUM, this.modelProfileService, this.clientService)); } } @@ -138,6 +151,7 @@ public class AgentTest { null, "Generate a summary of the following Java source code.", "The source code is:\n{{code}}", + null, new TextAgentResponseHandler("summary", null, null, @@ -176,6 +190,146 @@ public class AgentTest { LOGGER.info("### final result"); LOGGER.info(resultContexts.getFirst().toString()); } + + @Test + void toolCallingAgenticFlow() { + final var agents = Map.of( + "ToolCaller", + new Agent("ToolCaller", + AgentType.START_CONVERSATION, + ModelCategory.MEDIUM, + null, + "Use the provided tool when the user asks for weather information. After receiving tool results, respond using the forecast from the tool output.", + "What is the weather in Berlin, Germany? Use the tool.", + List.of(new ToolDefinition( + "get_weather", + "Gets the current weather for a given location.", + ToolParameters.create(Map.of("location", "City and country, for example Berlin, Germany")), + true)), + new ToolCallingAgentResponseHandler( + new TextAgentResponseHandler("answer"), + (completion, context, toolCallResults) -> { + if (completion.toolCalls().isEmpty()) { + return false; + } + context.setStringValue("toolName", completion.toolCalls().getFirst().name()); + context.setStringValue("toolArguments", completion.toolCalls().getFirst().arguments()); + toolCallResults.add(completion.toolCalls().getFirst(), ToolParameters.create().add("forecast", "Berlin is rainy and 21C")); + return true; + }), + null, + null, + null, + AgentNext::end)); + + final var context = new AgentContext(agents); + assertTrue(agents.get("ToolCaller").supportsToolCalling(context, this.clientService)); + + final var resultContexts = this.agentExecutor.walk(agents.get("ToolCaller"), context); + final var resultContext = resultContexts.getFirst(); + + assertEquals(1, resultContexts.size()); + assertEquals("get_weather", resultContext.getStringValue("toolName")); + assertTrue(resultContext.getStringValue("toolArguments").contains("Berlin")); + assertTrue(Grader2.assessScoreBinary("Does the text say that Berlin is rainy and 21C?", resultContext.getStringValue("answer"), ModelCategory.MEDIUM, this.modelProfileService, this.clientService)); + } + + @Test + void toolCallsCanBeHandledByNextAgentConversation() { + final var weatherTool = new ToolDefinition( + "get_weather", + "Gets the current weather for a given location.", + ToolParameters.create(Map.of("location", "City and country, for example Berlin, Germany")), + true); + final BaseAgent toolCaller = new BaseAgent() { + + @Override + public String name() { + return "ToolCaller"; + } + + @Override + public AgentType agentType(final AgentContext context) { + return AgentType.START_CONVERSATION; + } + + @Override + public ModelCategory modelCategory(final AgentContext context) { + return ModelCategory.MEDIUM; + } + + @Override + public String systemPrompt() { + return "You must call the provided tool for weather questions. Do not answer before the tool has been called."; + } + + @Override + public String textTemplate() { + return "What is the weather in Berlin, Germany? Use the tool."; + } + + @Override + public List toolDefinitions(final AgentContext context) { + return List.of(weatherTool); + } + + @Override + public ToolCallData toolCallData(final AgentContext context) { + return ToolCallData.required(List.of(weatherTool)); + } + + @Override + public boolean supportsToolCalling(final AgentContext context) { + return true; + } + + @Override + public AgentNext chooseNextAgent(final AgentContext context) { + return new AgentNext("ToolResultResponder", null); + } + + @Override + public de.extio.lmlib.agent.responsehandler.AgentResponseHandler responseHandler(final AgentContext context) { + return (completion, agentContext) -> true; + } + }; + final BaseAgent toolResultResponder = new Agent("ToolResultResponder", + AgentType.CONVERSATION, + ModelCategory.MEDIUM, + null, + "Use the existing tool result in the conversation to answer the original weather question.", + "Answer the original question using the tool result already present in the conversation. Do not call any tool.", + null, + new TextAgentResponseHandler("answer"), + context -> { + final var conversation = context.getConversation(); + final var assistantTurn = conversation.getConversation().getLast(); + context.setStringValue("handoffTurnType", assistantTurn.type().name()); + context.setStringValue("handoffToolName", assistantTurn.toolCalls().getFirst().name()); + context.setStringValue("handoffToolArguments", assistantTurn.toolCalls().getFirst().arguments()); + context.setStringValue("handoffToolCallId", assistantTurn.toolCalls().getFirst().id()); + conversation.addTurn(new Conversation.Turn(Conversation.TurnType.TOOL, ToolParameters.create().add("forecast", "Berlin is rainy and 21C").json(), null, assistantTurn.toolCalls().getFirst().id())); + }, + null, + null, + AgentNext::end); + final var agents = Map.of( + "ToolCaller", toolCaller, + "ToolResultResponder", toolResultResponder); + + final var context = new AgentContext(agents); + final var resultContexts = this.agentExecutor.walk(toolCaller, context); + final var resultContext = resultContexts.getFirst(); + + assertEquals(1, resultContexts.size()); + assertEquals("ASSISTANT", resultContext.getStringValue("handoffTurnType")); + assertEquals("get_weather", resultContext.getStringValue("handoffToolName")); + assertTrue(resultContext.getStringValue("handoffToolArguments").contains("Berlin")); + assertTrue(resultContext.getConversation().getConversation().stream().anyMatch(turn -> turn.type() == Conversation.TurnType.TOOL + && resultContext.getStringValue("handoffToolCallId").equals(turn.toolCallId()) + && turn.text().contains("Berlin is rainy and 21C"))); + assertTrue(Grader2.assessScoreBinary("Does the text say that Berlin is rainy and 21C?", resultContext.getStringValue("answer"), ModelCategory.MEDIUM, this.modelProfileService, this.clientService)); + } @Test void doubleBranch() { @@ -187,6 +341,7 @@ public class AgentTest { null, "", "Generate 5 random numbers from 1 to 10. Provide no preamble and no explanation. Return the response in JSON format with the following field: { \"numbers\": [\"Number 1\", \"Number 2\", ... ] }", + null, new JsonAgentResponseHandler(), null, null, @@ -200,6 +355,7 @@ public class AgentTest { null, "", "Generate 5 random numbers from 1 to 10. The previous number was [[numbers]]. Provide no preamble and no explanation. Return the response in JSON format with the following field: { \"random\": [\"Number 1\", \"Number 2\", ... ] }", + null, new JsonAgentResponseHandler(), null, null, @@ -213,6 +369,7 @@ public class AgentTest { null, "", "Add these 2 numbers: {{numbers}} + [[random]] ; Provide no preamble and no explanation. Return the response in JSON format with the following field: { \"result\": \"result\" }", + null, new JsonAgentResponseHandler(), null, null, @@ -226,6 +383,7 @@ public class AgentTest { null, "You are a helpful assistant.", "If you add {{numbers}} marbles to {{random}} marbles, do you get {{result}} marbles as the result? Answer with 'yes' or 'no'", + null, new TextAgentResponseHandler("grade"), null, null, diff --git a/src/test/java/de/extio/lmlib/client/cached/CachedClientTest.java b/src/test/java/de/extio/lmlib/client/cached/CachedClientTest.java index 36e510b..794c38e 100755 --- a/src/test/java/de/extio/lmlib/client/cached/CachedClientTest.java +++ b/src/test/java/de/extio/lmlib/client/cached/CachedClientTest.java @@ -50,6 +50,7 @@ public class CachedClientTest { var completion = this.clientService.getClient(ModelCategory.SMALL).conversation( ModelCategory.SMALL, Conversation.create("You are a funny person", "Tell me a cat joke"), + null, false); LOGGER.info(completion.toString()); assertFalse(completion.statistics().cached()); @@ -60,6 +61,7 @@ public class CachedClientTest { completion = this.clientService.getClient(ModelCategory.SMALL).conversation( ModelCategory.SMALL, Conversation.create("You are a funny person", "Tell me a cat joke"), + null, false); LOGGER.info(completion.toString()); assertTrue(completion.statistics().cached()); @@ -69,6 +71,7 @@ public class CachedClientTest { completion = this.clientService.getClient(ModelCategory.SMALL).conversation( ModelCategory.SMALL, Conversation.create("You are a funny person", "Tell me a dog joke"), + null, false); LOGGER.info(completion.toString()); assertFalse(completion.statistics().cached()); @@ -80,14 +83,14 @@ public class CachedClientTest { void cacheChat() throws Exception { final var conversation = Conversation.create("You are an assistant who stutters", "Hi there, I would like to know the climate in Munich."); - var completion = this.clientService.getClient(ModelCategory.MEDIUM).conversation(ModelCategory.MEDIUM, conversation, false); + var completion = this.clientService.getClient(ModelCategory.MEDIUM).conversation(ModelCategory.MEDIUM, conversation, null, false); LOGGER.info(completion.toString()); assertFalse(completion.statistics().cached()); assertEquals(1, completion.statistics().requests()); final var firstResponse = completion.response(); assertFalse(firstResponse.isBlank()); - completion = this.clientService.getClient(ModelCategory.MEDIUM).conversation(ModelCategory.MEDIUM, conversation, false); + completion = this.clientService.getClient(ModelCategory.MEDIUM).conversation(ModelCategory.MEDIUM, conversation, null, false); LOGGER.info(completion.toString()); assertTrue(completion.statistics().cached()); assertEquals(0, completion.statistics().requests()); @@ -96,7 +99,7 @@ public class CachedClientTest { conversation.addTurn(new Turn(TurnType.ASSISTANT, "T-t-the climate in B-b-berlin, um, it's, uh, quite, c-c-cool, actually. B-b-berlin has a, um, temperate climate, with, uh, four distinct s-s-seasons. S-s-spring is, um, pretty mild, with, uh, temperatures ranging from, uh, 10 to 20 degrees Celsius. S-s-summer is, um, warm, with, uh, highs around 25 degrees Celsius. F-f-fall is, um, cool, with, uh, temperatures ranging from 5 to 15 degrees Celsius. And, um, w-w-winter is, uh, quite, c-c-cold, with, uh, temperatures ranging from -2 to 5 degrees Celsius.")); conversation.addTurn(new Turn(TurnType.USER, "Great! How is the weather in summer?")); - completion = this.clientService.getClient(ModelCategory.MEDIUM).conversation(ModelCategory.MEDIUM, conversation, false); + completion = this.clientService.getClient(ModelCategory.MEDIUM).conversation(ModelCategory.MEDIUM, conversation, null, false); LOGGER.info(completion.toString()); assertFalse(completion.statistics().cached()); assertEquals(1, completion.statistics().requests()); @@ -107,19 +110,19 @@ public class CachedClientTest { void bypassCacheStillStoresFreshResult() throws Exception { final var conversation = Conversation.create("You are a witty assistant", "Give me a short joke about caches."); - final var initialCompletion = this.clientService.getClient(ModelCategory.MEDIUM).conversation(ModelCategory.MEDIUM, conversation, false); + final var initialCompletion = this.clientService.getClient(ModelCategory.MEDIUM).conversation(ModelCategory.MEDIUM, conversation, null, false); LOGGER.info(initialCompletion.toString()); assertFalse(initialCompletion.statistics().cached()); assertEquals(1, initialCompletion.statistics().requests()); assertFalse(initialCompletion.response().isBlank()); - final var bypassedCompletion = this.clientService.getClient(ModelCategory.MEDIUM).conversation(ModelCategory.MEDIUM, conversation, true); + final var bypassedCompletion = this.clientService.getClient(ModelCategory.MEDIUM).conversation(ModelCategory.MEDIUM, conversation, null, true); LOGGER.info(bypassedCompletion.toString()); assertFalse(bypassedCompletion.statistics().cached()); assertEquals(1, bypassedCompletion.statistics().requests()); assertFalse(bypassedCompletion.response().isBlank()); - final var cachedCompletion = this.clientService.getClient(ModelCategory.MEDIUM).conversation(ModelCategory.MEDIUM, conversation, false); + final var cachedCompletion = this.clientService.getClient(ModelCategory.MEDIUM).conversation(ModelCategory.MEDIUM, conversation, null, false); LOGGER.info(cachedCompletion.toString()); assertTrue(cachedCompletion.statistics().cached()); assertEquals(0, cachedCompletion.statistics().requests()); @@ -140,7 +143,7 @@ public class CachedClientTest { sb.append(chunk.content()); System.out.print(chunk.content()); } - }, false); + }, null, false); assertFalse(completion.statistics().cached()); assertEquals(1, completion.statistics().requests()); assertTrue(!sb.isEmpty()); @@ -158,7 +161,7 @@ public class CachedClientTest { sb2.append(chunk.content()); System.out.print(chunk.content()); } - }, false); + }, null, false); assertTrue(completion.statistics().cached()); assertEquals(sb.toString(), sb2.toString()); assertEquals(sb2.toString(), completion.response()); diff --git a/src/test/java/de/extio/lmlib/client/oai/CompletionClientTest.java b/src/test/java/de/extio/lmlib/client/oai/CompletionClientTest.java index 8717039..24cc03a 100755 --- a/src/test/java/de/extio/lmlib/client/oai/CompletionClientTest.java +++ b/src/test/java/de/extio/lmlib/client/oai/CompletionClientTest.java @@ -5,6 +5,8 @@ import static org.junit.jupiter.api.Assertions.assertFalse; import static org.junit.jupiter.api.Assertions.assertTrue; import java.time.LocalDateTime; +import java.util.List; +import java.util.Map; import java.util.concurrent.CompletableFuture; import java.util.concurrent.ExecutorService; import java.util.concurrent.Executors; @@ -21,9 +23,13 @@ import org.springframework.boot.test.context.SpringBootTest.WebEnvironment; import org.springframework.context.annotation.PropertySource; import de.extio.lmlib.client.ClientService; +import de.extio.lmlib.client.CompletionFinishReason; import de.extio.lmlib.client.Conversation; import de.extio.lmlib.client.Conversation.Turn; import de.extio.lmlib.client.Conversation.TurnType; +import de.extio.lmlib.client.ToolCallData; +import de.extio.lmlib.client.ToolDefinition; +import de.extio.lmlib.client.ToolParameters; import de.extio.lmlib.client.oai.completion.chat.ChatCompletionClient; import de.extio.lmlib.client.oai.completion.text.TextCompletionClient; import de.extio.lmlib.grader.Grader2; @@ -57,6 +63,7 @@ public class CompletionClientTest { final var completion = this.textCompletionClient.conversation( modelProfile, Conversation.create("You are a helpful assistant", "Say the color is green"), + null, false); LOGGER.info(completion.reasoning()); @@ -71,7 +78,7 @@ public class CompletionClientTest { final var conversation = Conversation.create("You are a helpful assistant", "I would like to paint my canvas in a solid color."); conversation.addTurn(new Turn(TurnType.ASSISTANT, "Which color do you want to see?")); conversation.addTurn(new Turn(TurnType.USER, "You choose!")); - final var completion = this.textCompletionClient.conversation(modelProfile, conversation, false); + final var completion = this.textCompletionClient.conversation(modelProfile, conversation, null, false); LOGGER.info(completion.reasoning()); LOGGER.info(completion.response()); @@ -94,7 +101,7 @@ public class CompletionClientTest { System.out.print(chunk.content()); sb.append(chunk.content()); } - }, false); + }, null, false); if (completion.reasoning() != null) { // Prompt template does not separate reasoning and response from the stream at the moment @@ -114,7 +121,7 @@ public class CompletionClientTest { final var conversation = Conversation.create("You are a helpful assistant", "I would like to paint my canvas in a solid color."); conversation.addTurn(new Turn(TurnType.ASSISTANT, "Which color do you want to see?")); conversation.addTurn(new Turn(TurnType.USER, "You choose!")); - final var completion = this.chatCompletionclient.conversation(modelProfile, conversation, false); + final var completion = this.chatCompletionclient.conversation(modelProfile, conversation, null, false); LOGGER.info(completion.reasoning()); LOGGER.info(completion.response()); @@ -137,12 +144,84 @@ public class CompletionClientTest { System.out.print(chunk.content()); sb.append(chunk.content()); } - }, false); + }, null, false); assertEquals(completion.response(), sb.toString()); assertTrue(Grader2.assessScoreBinary("Does the text explain a chess opening?", completion.response(), modelProfile, this.clientService)); } + @Test + void toolCallingChatCompletion() throws Exception { + final var modelProfile = this.modelProfileService.getModelProfile("profile.model.chatcompletion"); + final var toolDefinition = new ToolDefinition( + "get_weather", + "Gets the current weather for a given location.", + ToolParameters.create(Map.of("location", "City and country, for example Berlin, Germany")), + true); + final var conversation = Conversation.create( + "You are a helpful assistant. Use the provided tool when the user asks for weather information. After receiving tool results, answer using the forecast from the tool output.", + "What is the weather in Berlin, Germany? Use the tool."); + final var toolCallData = ToolCallData.required(List.of(toolDefinition)); + final var completion = this.chatCompletionclient.conversation( + modelProfile, + conversation, + toolCallData, + false); + + assertTrue(this.chatCompletionclient.supportsToolCalling()); + assertFalse(this.textCompletionClient.supportsToolCalling()); + assertTrue(this.clientService.supportsToolCalling(modelProfile)); + assertTrue(completion.finishReason() == CompletionFinishReason.TOOL_CALLS); + assertFalse(completion.toolCalls().isEmpty()); + assertEquals("get_weather", completion.toolCalls().getFirst().name()); + assertTrue(completion.toolCalls().getFirst().arguments().contains("Berlin")); + + conversation.addTurn(new Turn(TurnType.ASSISTANT, "", completion.toolCalls(), null)); + conversation.addTurn(new Turn(TurnType.TOOL, ToolParameters.create().add("forecast", "Berlin is rainy and 21C").json(), null, completion.toolCalls().getFirst().id())); + + final var finalCompletion = this.chatCompletionclient.conversation(modelProfile, conversation, ToolCallData.auto(List.of(toolDefinition)), false); + assertTrue(finalCompletion.finishReason() == CompletionFinishReason.DONE); + assertTrue(Grader2.assessScoreBinary("Does the text say that Berlin is rainy and 21C?", finalCompletion.response(), modelProfile, this.clientService)); + } + + @Test + void streamToolCallingChatCompletion() throws Exception { + final var modelProfile = this.modelProfileService.getModelProfile("profile.model.chatcompletion"); + final var getWeather = new ToolDefinition( + "get_weather", + "Gets the current weather for a given location.", + ToolParameters.create(Map.of("location", "City and country, for example Berlin, Germany")), + true); + final var getCpu = new ToolDefinition( + "get_cpu", + "Gets the current CPU utilization.", + ToolParameters.create(Map.of("cpu", "CPU identifier, for example CPU0. Use CPU for overall utilization.")), + true); + final var completion = this.chatCompletionclient.streamConversation( + modelProfile, + Conversation.create( + "You are a helpful assistant. Use the provided tool when applicable.", + "How high is the current CPU utilization?"), + chunk -> { + if (chunk.reasoningContent() != null) { + System.out.print(chunk.reasoningContent()); + } + if (chunk.content() != null) { + System.out.print(chunk.content()); + } + }, + ToolCallData.required(List.of(getWeather, getCpu)), + false); + + assertTrue(this.chatCompletionclient.supportsToolCalling()); + assertFalse(this.textCompletionClient.supportsToolCalling()); + assertTrue(this.clientService.supportsToolCalling(modelProfile)); + assertTrue(completion.finishReason() == CompletionFinishReason.TOOL_CALLS); + assertFalse(completion.toolCalls().isEmpty()); + assertEquals("get_cpu", completion.toolCalls().getFirst().name()); + assertTrue(completion.toolCalls().getFirst().arguments().contains("CPU")); + } + @Disabled @Test void hugePrompt() throws Exception { @@ -156,6 +235,7 @@ public class CompletionClientTest { final var completion = this.textCompletionClient.conversation( ModelCategory.MEDIUM, Conversation.create("You are counting machine", "How many numbers do you count?\n" + sb.toString()), + null, false); LOGGER.info(completion.response()); } @@ -204,6 +284,7 @@ public class CompletionClientTest { final var completion = this.textCompletionClient.conversation( ModelCategory.MEDIUM, Conversation.create("You are a helpful assistant", "How do I calculate the annual profit margin?"), + null, false); LOGGER.info(completion.response()); } @@ -215,6 +296,7 @@ public class CompletionClientTest { final var completion = this.textCompletionClient.conversation( ModelCategory.MEDIUM, Conversation.create("You are a helpful assistant", "How do I calculate the operating profit ratio?"), + null, false); LOGGER.info(completion.response()); } @@ -226,6 +308,7 @@ public class CompletionClientTest { final var completion = this.textCompletionClient.conversation( ModelCategory.MEDIUM, Conversation.create("You are a helpful assistant", "How do I calculate the return of investment?"), + null, false); LOGGER.info(completion.response()); } @@ -237,6 +320,7 @@ public class CompletionClientTest { final var completion = this.textCompletionClient.conversation( ModelCategory.MEDIUM, Conversation.create("You are a helpful assistant", "How do I calculate the return on net worth?"), + null, false); LOGGER.info(completion.response()); }