feat: add Anthropic Claude Messages API client
Adds AnthropicClient with tool use, SSE streaming, and extended thinking behind an optional AnthropicDialect bean, plus ANTHROPIC model provider, request-shaping unit tests, and integration tests against an Anthropic-compatible server.
This commit is contained in:
extio-de
16
README.md
16
README.md
@@ -13,13 +13,13 @@ lm-lib is published under the Apache License 2.0.
|
||||
|
||||
lm-lib is designed for applications that need more than single prompt-response calls. Its center of gravity is the agent framework: structured multi-step flows such as classification, planning, fan-out analysis, synthesis, grading, feedback loops, and other graph-based execution patterns.
|
||||
|
||||
It also supports direct completion use cases through a client abstraction layer. The built-in clients currently cover OpenAI-compatible chat completion endpoints, the newer OpenAI-compatible Responses API, legacy text completion endpoints, OpenRouter's OpenAI-compatible routing API, and Ollama's native chat API, and the abstraction remains open for additional provider implementations over time. Applications can also provide their own client implementations by implementing the relevant interfaces.
|
||||
It also supports direct completion use cases through a client abstraction layer. The built-in clients currently cover OpenAI-compatible chat completion endpoints, the newer OpenAI-compatible Responses API, legacy text completion endpoints, OpenRouter's OpenAI-compatible routing API, Ollama's native chat API, and Anthropic's Claude Messages API, and the abstraction remains open for additional provider implementations over time. Applications can also provide their own client implementations by implementing the relevant interfaces.
|
||||
|
||||
The current built-in client set supports OpenAI-compatible chat completions, OpenAI-compatible Responses, OpenRouter chat completions with cached metadata discovery, legacy text completions, and native Ollama chat completions, with model-specific prompt strategies for text completion models.
|
||||
The current built-in client set supports OpenAI-compatible chat completions, OpenAI-compatible Responses, OpenRouter chat completions with cached metadata discovery, legacy text completions, native Ollama chat completions, and Anthropic Claude messages, with model-specific prompt strategies for text completion models.
|
||||
|
||||
## Key Capabilities
|
||||
|
||||
- OpenAI-compatible chat completions, OpenAI-compatible Responses, legacy text completions, OpenRouter chat completions, plus native Ollama chat completions
|
||||
- OpenAI-compatible chat completions, OpenAI-compatible Responses, legacy text completions, OpenRouter chat completions, native Ollama chat completions, plus Anthropic Claude messages
|
||||
- Chat-completion tool calling with tool definitions, returned tool calls, and tool-result follow-up rounds
|
||||
- Streaming and non-streaming APIs
|
||||
- Separate reasoning capture for models that expose reasoning output
|
||||
@@ -243,7 +243,7 @@ Each profile file is a `.properties` resource on the classpath.
|
||||
| `maxContextLength` | Integer | Yes | Maximum total context length. | `16000`, `32768`, `128000` |
|
||||
| `temperature` | Double | Yes | Sampling temperature. Must be greater than zero. | `0.2`, `0.4`, `0.7` |
|
||||
| `topP` | Double | Yes | Nucleus sampling parameter. Must be greater than zero. | `1.0`, `0.9` |
|
||||
| `modelProvider` | Enum | Yes | Completion client type. | `OAI_CHAT_COMPLETION`, `OAI_RESPONSES`, `OAI_TEXT_COMPLETION`, `OPENROUTER`, `OLLAMA` |
|
||||
| `modelProvider` | Enum | Yes | Completion client type. | `OAI_CHAT_COMPLETION`, `OAI_RESPONSES`, `OAI_TEXT_COMPLETION`, `OPENROUTER`, `OLLAMA`, `ANTHROPIC` |
|
||||
| `modelName` | String | No | Model name or deployment. Can be empty for endpoints where the model should be auto-detected. | `gpt-4`, `Llama-3.3-70B-Instruct`, `` |
|
||||
| `url` | String | Yes | Base API URL. | `https://api.openai.com/v1`, `http://localhost:5001` |
|
||||
| `apiKey` | String | No | Bearer token or Spring placeholder. Can be empty for local servers. | `sk-...`, `${apikey}`, `` |
|
||||
@@ -510,6 +510,14 @@ If you need to shape Ollama thinking behavior, you can optionally register an `O
|
||||
|
||||
lm-lib caches `/api/show` model details per resolved model. Thinking and tool definitions are sent only when the model advertises the corresponding capability, and `supportsToolCalling(modelProfile)` probes the same cached capability data.
|
||||
|
||||
### Anthropic Provider
|
||||
|
||||
`ANTHROPIC` targets the Anthropic Claude Messages API (`POST /v1/messages`), including Anthropic-compatible servers that expose the same endpoint shape.
|
||||
|
||||
The client sends system turns as the top-level `system` parameter, uses `x-api-key` and `anthropic-version` headers, and requires a positive `maxTokens` in the profile because the Messages API mandates `max_tokens`. Tool calling uses native `tool_use`/`tool_result` content blocks, and streamed responses are consumed through the standard server-sent event protocol (`message_start`, `content_block_delta`, `message_delta`, `message_stop`). Extended thinking is supported: thinking blocks are captured as reasoning and replayed on follow-up tool rounds. There is no model discovery endpoint, so `getModelNames(...)` returns an empty list and `modelName` must be configured.
|
||||
|
||||
To enable extended thinking, register an `AnthropicDialect` bean and return a budget (at least 1024 tokens) from `thinkingBudgetTokens(...)`; the budget is clamped below the profile's `maxTokens`. Omitting the bean or returning `null` keeps thinking disabled.
|
||||
|
||||
### Tool Calling Support
|
||||
|
||||
Tool calling is exposed through the same lm-lib API across providers that support it. The capability is model-aware rather than purely provider-wide, so callers should check `supportsToolCalling(modelProfile)` or `supportsToolCalling(modelCategory)` on `Client` or `ClientService` before enabling tool-aware flows.
|
||||
|
||||
@@ -6,6 +6,7 @@ import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean
|
||||
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
|
||||
import org.springframework.context.annotation.Bean;
|
||||
|
||||
import de.extio.lmlib.client.anthropic.AnthropicClient;
|
||||
import de.extio.lmlib.client.openrouter.OpenRouterClient;
|
||||
import de.extio.lmlib.client.openrouter.OpenRouterModelDiscovery;
|
||||
import de.extio.lmlib.client.ollama.OllamaClient;
|
||||
@@ -61,5 +62,11 @@ public class ClientAutoConfiguration {
|
||||
OllamaClient ollamaClient() {
|
||||
return new OllamaClient();
|
||||
}
|
||||
|
||||
|
||||
@Bean
|
||||
@ConditionalOnMissingBean
|
||||
AnthropicClient anthropicClient() {
|
||||
return new AnthropicClient();
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
@@ -0,0 +1,687 @@
|
||||
package de.extio.lmlib.client.anthropic;
|
||||
|
||||
import java.io.BufferedReader;
|
||||
import java.io.IOException;
|
||||
import java.io.InputStream;
|
||||
import java.io.InputStreamReader;
|
||||
import java.math.BigDecimal;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.time.Duration;
|
||||
import java.time.LocalDateTime;
|
||||
import java.util.ArrayList;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.function.Consumer;
|
||||
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.beans.factory.annotation.Qualifier;
|
||||
import org.springframework.http.HttpMethod;
|
||||
import org.springframework.http.HttpStatusCode;
|
||||
import org.springframework.web.client.RestClient;
|
||||
|
||||
import com.fasterxml.jackson.annotation.JsonInclude;
|
||||
|
||||
import de.extio.lmlib.client.Chunk;
|
||||
import de.extio.lmlib.client.Client;
|
||||
import de.extio.lmlib.client.Completion;
|
||||
import de.extio.lmlib.client.CompletionFinishReason;
|
||||
import de.extio.lmlib.client.CompletionStatistics;
|
||||
import de.extio.lmlib.client.Conversation;
|
||||
import de.extio.lmlib.client.Conversation.Turn;
|
||||
import de.extio.lmlib.client.ProxyAuthorizationSupport;
|
||||
import de.extio.lmlib.client.ToolCall;
|
||||
import de.extio.lmlib.client.ToolCallData;
|
||||
import de.extio.lmlib.client.ToolDefinition;
|
||||
import de.extio.lmlib.client.anthropic.AnthropicStreamEvent.Delta;
|
||||
import de.extio.lmlib.profile.ModelCategory;
|
||||
import de.extio.lmlib.profile.ModelProfile;
|
||||
import de.extio.lmlib.profile.ModelProfile.ModelProvider;
|
||||
import de.extio.lmlib.profile.ModelProfileService;
|
||||
import de.extio.lmlib.token.TokenizerResolver;
|
||||
import tools.jackson.core.JacksonException;
|
||||
import tools.jackson.core.type.TypeReference;
|
||||
import tools.jackson.databind.DeserializationFeature;
|
||||
import tools.jackson.databind.ObjectMapper;
|
||||
import tools.jackson.databind.json.JsonMapper;
|
||||
|
||||
/**
|
||||
* {@link Client} implementation for the Anthropic Claude Messages API.
|
||||
* <p>
|
||||
* Implements the {@code POST /v1/messages} endpoint, including tool use with
|
||||
* {@code tool_use}/{@code tool_result} content blocks, server-sent event
|
||||
* streaming, and extended thinking (dialect-controlled). System turns are sent
|
||||
* as the top-level {@code system} parameter since the Messages API has no
|
||||
* system message role.
|
||||
*/
|
||||
public class AnthropicClient implements Client {
|
||||
|
||||
private static final Logger LOGGER = LoggerFactory.getLogger(AnthropicClient.class);
|
||||
|
||||
private static final String MESSAGES_PATH = "/v1/messages";
|
||||
|
||||
private static final AnthropicDialect DEFAULT_ANTHROPIC_DIALECT = new AnthropicDialect() {
|
||||
};
|
||||
|
||||
private static final TypeReference<Map<String, Object>> TOOL_INPUT_TYPE = new TypeReference<Map<String, Object>>() {
|
||||
};
|
||||
|
||||
private final ObjectMapper objectMapper = JsonMapper.builder()
|
||||
.changeDefaultPropertyInclusion(incl -> incl.withValueInclusion(JsonInclude.Include.NON_NULL))
|
||||
.changeDefaultPropertyInclusion(incl -> incl.withContentInclusion(JsonInclude.Include.NON_NULL))
|
||||
.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false)
|
||||
.build();
|
||||
|
||||
@Autowired
|
||||
@Qualifier("lmLibRestClientBuilder")
|
||||
protected RestClient.Builder restClientBuilder;
|
||||
|
||||
@Autowired
|
||||
protected TokenizerResolver tokenizerResolver;
|
||||
|
||||
@Autowired
|
||||
protected ModelProfileService modelProfileService;
|
||||
|
||||
@Autowired(required = false)
|
||||
protected AnthropicDialect anthropicDialect;
|
||||
|
||||
@Autowired
|
||||
protected ProxyAuthorizationSupport proxyAuthorizationSupport;
|
||||
|
||||
@Override
|
||||
public Completion conversation(final ModelCategory modelCategory, final Conversation conversation, final ToolCallData toolCallData, final boolean skipCache) {
|
||||
return this.streamConversation(modelCategory, conversation, null, toolCallData, skipCache);
|
||||
}
|
||||
|
||||
@Override
|
||||
public Completion conversation(final ModelProfile modelProfile, final Conversation conversation, final ToolCallData toolCallData, final boolean skipCache) {
|
||||
return this.streamConversation(modelProfile, conversation, null, toolCallData, skipCache);
|
||||
}
|
||||
|
||||
@Override
|
||||
public Completion streamConversation(final ModelCategory modelCategory_, final Conversation conversation, final Consumer<Chunk> chunkConsumer, final ToolCallData toolCallData, final boolean skipCache) {
|
||||
final var modelCategory = modelCategory_ == null ? ModelCategory.MEDIUM : modelCategory_;
|
||||
final var modelProfile = this.modelProfileService.getModelProfile(modelCategory.getModelProfile(), modelCategory);
|
||||
return this.streamConversation(modelProfile, conversation, chunkConsumer, toolCallData, skipCache);
|
||||
}
|
||||
|
||||
@Override
|
||||
public Completion streamConversation(final ModelProfile modelProfile, final Conversation conversation, final Consumer<Chunk> chunkConsumer, final ToolCallData toolCallData, final boolean skipCache) {
|
||||
if (modelProfile == null || modelProfile.modelProvider() != ModelProvider.ANTHROPIC) {
|
||||
throw new IllegalArgumentException("Invalid ModelProfile");
|
||||
}
|
||||
|
||||
final var request = this.createRequest(conversation, modelProfile, chunkConsumer != null, toolCallData);
|
||||
final String requestBody;
|
||||
try {
|
||||
requestBody = this.objectMapper.writeValueAsString(request);
|
||||
}
|
||||
catch (final JacksonException e) {
|
||||
throw new IllegalStateException("Cannot serialize request body", e);
|
||||
}
|
||||
|
||||
LOGGER.debug("Requesting Anthropic message at {}", modelProfile.url());
|
||||
final var start = LocalDateTime.now();
|
||||
final var base = stripMessagesSuffix(modelProfile.url());
|
||||
final var path = messagesPath(modelProfile.url());
|
||||
|
||||
var requestSpec = this.restClientBuilder.baseUrl(base).build()
|
||||
.method(HttpMethod.POST)
|
||||
.uri(path)
|
||||
.header("Content-Type", "application/json")
|
||||
.header("anthropic-version", this.getAnthropicDialect().anthropicVersion(modelProfile));
|
||||
if (modelProfile.apiKey() != null && !modelProfile.apiKey().isBlank()) {
|
||||
requestSpec = requestSpec.header("x-api-key", modelProfile.apiKey());
|
||||
}
|
||||
requestSpec = this.proxyAuthorizationSupport.apply(requestSpec);
|
||||
|
||||
final var response = requestSpec
|
||||
.body(requestBody)
|
||||
.exchange((clientRequest, clientResponse) -> {
|
||||
if (clientResponse.getStatusCode().isError()) {
|
||||
final var errorBody = new String(clientResponse.getBody().readAllBytes(), StandardCharsets.UTF_8);
|
||||
final var errorMessage = this.getErrorMessage(errorBody);
|
||||
LOGGER.warn("Error response from server: {} {}", clientResponse.getStatusCode(), errorMessage);
|
||||
throw new IllegalStateException("Error response from server: " + clientResponse.getStatusCode() + " " + errorMessage);
|
||||
}
|
||||
if (chunkConsumer == null) {
|
||||
return this.objectMapper.readValue(clientResponse.getBody(), AnthropicMessageResponse.class);
|
||||
}
|
||||
return this.readStream(clientResponse.getBody(), chunkConsumer);
|
||||
});
|
||||
|
||||
return this.mapResponse(response, modelProfile, start, conversation);
|
||||
}
|
||||
|
||||
@Override
|
||||
public List<String> getModelNames(final ModelProfile modelProfile, final boolean forceReload) {
|
||||
// The Anthropic API does not offer a model discovery endpoint.
|
||||
return List.of();
|
||||
}
|
||||
|
||||
@Override
|
||||
public ModelProvider getModelProvider() {
|
||||
return ModelProvider.ANTHROPIC;
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean supportsToolCalling(final ModelCategory modelCategory) {
|
||||
final var modelProfile = this.modelProfileService.getModelProfile(modelCategory.getModelProfile(), modelCategory);
|
||||
return this.supportsToolCalling(modelProfile);
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean supportsToolCalling(final ModelProfile modelProfile) {
|
||||
return modelProfile != null && modelProfile.modelProvider() == ModelProvider.ANTHROPIC;
|
||||
}
|
||||
|
||||
AnthropicMessageRequest createRequest(final Conversation conversation, final ModelProfile modelProfile, final boolean streaming, final ToolCallData toolCallData) {
|
||||
final var modelName = this.getPrimaryModelName(modelProfile);
|
||||
if (modelName == null || modelName.isBlank()) {
|
||||
throw new IllegalStateException("No model name resolved for Anthropic profile " + modelProfile.category());
|
||||
}
|
||||
if (modelProfile.maxTokens() <= 0) {
|
||||
throw new IllegalStateException("maxTokens must be greater than zero for Anthropic profile " + modelProfile.category());
|
||||
}
|
||||
final var systemBuilder = new StringBuilder();
|
||||
final var messages = this.createMessages(conversation, systemBuilder);
|
||||
final var request = new AnthropicMessageRequest(
|
||||
modelName,
|
||||
modelProfile.maxTokens(),
|
||||
systemBuilder.isEmpty() ? null : systemBuilder.toString(),
|
||||
messages,
|
||||
Double.valueOf(modelProfile.temperature()),
|
||||
Double.valueOf(modelProfile.topP()),
|
||||
null,
|
||||
Boolean.valueOf(streaming),
|
||||
toolCallData != null && toolCallData.hasTools() ? this.createTools(toolCallData.tools()) : null,
|
||||
toolCallData != null && toolCallData.hasTools() ? this.createToolChoice(toolCallData) : null,
|
||||
this.createThinkingConfig(modelProfile));
|
||||
return request;
|
||||
}
|
||||
|
||||
List<AnthropicMessage> createMessages(final Conversation conversation, final StringBuilder systemCollector) {
|
||||
final List<AnthropicMessage> messages = new ArrayList<>();
|
||||
final List<AnthropicContentBlock> pendingToolResults = new ArrayList<>();
|
||||
for (final Turn turn : conversation.getConversation()) {
|
||||
switch (turn.type()) {
|
||||
case SYSTEM: {
|
||||
final var text = turn.text() != null ? turn.text() : "";
|
||||
if (!text.isBlank()) {
|
||||
if (!systemCollector.isEmpty()) {
|
||||
systemCollector.append("\n\n");
|
||||
}
|
||||
systemCollector.append(text);
|
||||
}
|
||||
break;
|
||||
}
|
||||
case USER: {
|
||||
this.flushToolResults(messages, pendingToolResults);
|
||||
messages.add(AnthropicMessage.user(turn.text() != null ? turn.text() : ""));
|
||||
break;
|
||||
}
|
||||
case ASSISTANT: {
|
||||
this.flushToolResults(messages, pendingToolResults);
|
||||
messages.add(this.createAssistantMessage(turn));
|
||||
break;
|
||||
}
|
||||
case TOOL: {
|
||||
final var text = turn.text() != null && !turn.text().isBlank() ? turn.text() : " ";
|
||||
pendingToolResults.add(AnthropicContentBlock.toolResult(turn.toolCallId(), text));
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
this.flushToolResults(messages, pendingToolResults);
|
||||
return messages;
|
||||
}
|
||||
|
||||
private AnthropicMessage createAssistantMessage(final Turn turn) {
|
||||
final List<Object> blocks = new ArrayList<>();
|
||||
for (final var outputItem : turn.outputItems()) {
|
||||
final var type = outputItem.get("type") instanceof final String itemType ? itemType : null;
|
||||
if ("thinking".equals(type) || "redacted_thinking".equals(type)) {
|
||||
blocks.add(outputItem);
|
||||
}
|
||||
}
|
||||
final var text = turn.text() != null ? turn.text() : "";
|
||||
if (!text.isBlank() || (turn.toolCalls().isEmpty() && blocks.isEmpty())) {
|
||||
blocks.add(AnthropicContentBlock.text(text.isBlank() ? " " : text));
|
||||
}
|
||||
var toolCallIndex = 0;
|
||||
for (final var toolCall : turn.toolCalls()) {
|
||||
toolCallIndex++;
|
||||
final var id = toolCall.id() != null && !toolCall.id().isBlank() ? toolCall.id() : "toolu_lm_" + toolCallIndex;
|
||||
blocks.add(AnthropicContentBlock.toolUse(id, toolCall.name(), this.parseToolInput(toolCall.arguments())));
|
||||
}
|
||||
if (blocks.size() == 1 && blocks.getFirst() instanceof final AnthropicContentBlock block && "text".equals(block.type())) {
|
||||
return AnthropicMessage.assistant(block.text());
|
||||
}
|
||||
return AnthropicMessage.assistantBlocks(blocks);
|
||||
}
|
||||
|
||||
private void flushToolResults(final List<AnthropicMessage> messages, final List<AnthropicContentBlock> pendingToolResults) {
|
||||
if (pendingToolResults.isEmpty()) {
|
||||
return;
|
||||
}
|
||||
messages.add(AnthropicMessage.userBlocks(List.copyOf(pendingToolResults)));
|
||||
pendingToolResults.clear();
|
||||
}
|
||||
|
||||
private List<AnthropicTool> createTools(final List<ToolDefinition> toolDefinitions) {
|
||||
return toolDefinitions.stream()
|
||||
.map(toolDefinition -> new AnthropicTool(toolDefinition.name(), toolDefinition.description(), toolDefinition.parameters()))
|
||||
.toList();
|
||||
}
|
||||
|
||||
AnthropicToolChoice createToolChoice(final ToolCallData toolCallData) {
|
||||
final var parallel = toolCallData.parallelToolCalls();
|
||||
final Boolean disableParallelToolUse = parallel == null ? null : Boolean.valueOf(!parallel);
|
||||
if (toolCallData.forcedToolName() != null && !toolCallData.forcedToolName().isBlank()) {
|
||||
return new AnthropicToolChoice("tool", toolCallData.forcedToolName(), disableParallelToolUse);
|
||||
}
|
||||
final var toolChoice = toolCallData.toolChoice();
|
||||
if (toolChoice == null) {
|
||||
return disableParallelToolUse == null ? null : new AnthropicToolChoice("auto", null, disableParallelToolUse);
|
||||
}
|
||||
return switch (toolChoice) {
|
||||
case ToolCallData.TOOL_CHOICE_REQUIRED -> new AnthropicToolChoice("any", null, disableParallelToolUse);
|
||||
case ToolCallData.TOOL_CHOICE_NONE -> new AnthropicToolChoice("none", null, disableParallelToolUse);
|
||||
default -> new AnthropicToolChoice("auto", null, disableParallelToolUse);
|
||||
};
|
||||
}
|
||||
|
||||
AnthropicThinkingConfig createThinkingConfig(final ModelProfile modelProfile) {
|
||||
final var budgetTokens = this.getAnthropicDialect().thinkingBudgetTokens(modelProfile);
|
||||
if (budgetTokens == null) {
|
||||
return null;
|
||||
}
|
||||
final var maxTokens = modelProfile.maxTokens();
|
||||
final var clamped = Math.min(budgetTokens.intValue(), maxTokens - 1);
|
||||
if (clamped < AnthropicDialect.MIN_THINKING_BUDGET_TOKENS) {
|
||||
LOGGER.debug("Thinking budget {} does not fit maxTokens {} - thinking will be omitted.", budgetTokens, maxTokens);
|
||||
return null;
|
||||
}
|
||||
return AnthropicThinkingConfig.enabled(Integer.valueOf(clamped));
|
||||
}
|
||||
|
||||
AnthropicMessageResponse readStream(final InputStream inputStream, final Consumer<Chunk> chunkConsumer) throws IOException {
|
||||
final List<StreamBlock> blocks = new ArrayList<>();
|
||||
String id = null;
|
||||
String model = null;
|
||||
String stopReason = null;
|
||||
String stopSequence = null;
|
||||
AnthropicUsage usage = null;
|
||||
|
||||
try (BufferedReader reader = new BufferedReader(new InputStreamReader(inputStream, StandardCharsets.UTF_8))) {
|
||||
String line;
|
||||
while ((line = reader.readLine()) != null) {
|
||||
line = line.trim();
|
||||
if (line.isEmpty() || line.startsWith(":") || line.startsWith("event:")) {
|
||||
continue;
|
||||
}
|
||||
if (line.startsWith("data:")) {
|
||||
line = line.substring(5).trim();
|
||||
}
|
||||
if (line.isEmpty()) {
|
||||
continue;
|
||||
}
|
||||
|
||||
final AnthropicStreamEvent event;
|
||||
try {
|
||||
event = this.objectMapper.readValue(line, AnthropicStreamEvent.class);
|
||||
}
|
||||
catch (final JacksonException e) {
|
||||
LOGGER.warn("Streamed event is not parseable: {}", line, e);
|
||||
continue;
|
||||
}
|
||||
if (event.type() == null) {
|
||||
continue;
|
||||
}
|
||||
|
||||
switch (event.type()) {
|
||||
case AnthropicStreamEvent.EVENT_MESSAGE_START:
|
||||
if (event.message() != null) {
|
||||
id = event.message().id() != null ? event.message().id() : id;
|
||||
model = event.message().model() != null ? event.message().model() : model;
|
||||
usage = event.message().usage() != null ? event.message().usage().mergedWith(usage) : usage;
|
||||
}
|
||||
break;
|
||||
case AnthropicStreamEvent.EVENT_CONTENT_BLOCK_START:
|
||||
this.startBlock(blocks, event);
|
||||
break;
|
||||
case AnthropicStreamEvent.EVENT_CONTENT_BLOCK_DELTA:
|
||||
this.applyDelta(blocks, event, chunkConsumer);
|
||||
break;
|
||||
case AnthropicStreamEvent.EVENT_MESSAGE_DELTA:
|
||||
if (event.delta() != null) {
|
||||
stopReason = event.delta().stopReason() != null ? event.delta().stopReason() : stopReason;
|
||||
stopSequence = event.delta().stopSequence() != null ? event.delta().stopSequence() : stopSequence;
|
||||
}
|
||||
usage = event.usage() != null ? usage.mergedWith(event.usage()) : usage;
|
||||
break;
|
||||
case AnthropicStreamEvent.EVENT_ERROR:
|
||||
final var errorMessage = event.error() != null && event.error().message() != null ? event.error().message() : "unknown error";
|
||||
LOGGER.warn("Error event in streamed Anthropic response: {}", errorMessage);
|
||||
throw new IllegalStateException("Error in streamed Anthropic response: " + errorMessage);
|
||||
case AnthropicStreamEvent.EVENT_MESSAGE_STOP:
|
||||
return this.aggregate(id, model, stopReason, stopSequence, usage, blocks);
|
||||
default:
|
||||
break; // ping, content_block_stop, and unknown events
|
||||
}
|
||||
}
|
||||
}
|
||||
return this.aggregate(id, model, stopReason, stopSequence, usage, blocks);
|
||||
}
|
||||
|
||||
private void startBlock(final List<StreamBlock> blocks, final AnthropicStreamEvent event) {
|
||||
final var block = this.ensureBlock(blocks, event.index());
|
||||
final var contentBlock = event.contentBlock();
|
||||
if (contentBlock == null) {
|
||||
return;
|
||||
}
|
||||
block.type = contentBlock.type();
|
||||
if (contentBlock.id() != null && !contentBlock.id().isBlank()) {
|
||||
block.id = contentBlock.id();
|
||||
}
|
||||
if (contentBlock.name() != null && !contentBlock.name().isBlank()) {
|
||||
block.name = contentBlock.name();
|
||||
}
|
||||
if (contentBlock.text() != null) {
|
||||
block.text.append(contentBlock.text());
|
||||
}
|
||||
if (contentBlock.thinking() != null) {
|
||||
block.thinking.append(contentBlock.thinking());
|
||||
}
|
||||
if (contentBlock.data() != null) {
|
||||
block.data = contentBlock.data();
|
||||
}
|
||||
}
|
||||
|
||||
private void applyDelta(final List<StreamBlock> blocks, final AnthropicStreamEvent event, final Consumer<Chunk> chunkConsumer) {
|
||||
final var delta = event.delta();
|
||||
if (delta == null || delta.type() == null) {
|
||||
return;
|
||||
}
|
||||
final var block = this.ensureBlock(blocks, event.index());
|
||||
switch (delta.type()) {
|
||||
case Delta.DELTA_TYPE_TEXT:
|
||||
if (delta.text() != null && !delta.text().isEmpty()) {
|
||||
block.text.append(delta.text());
|
||||
chunkConsumer.accept(new Chunk(delta.text(), null));
|
||||
}
|
||||
break;
|
||||
case Delta.DELTA_TYPE_THINKING:
|
||||
if (delta.thinking() != null && !delta.thinking().isEmpty()) {
|
||||
block.thinking.append(delta.thinking());
|
||||
chunkConsumer.accept(new Chunk(null, delta.thinking()));
|
||||
}
|
||||
break;
|
||||
case Delta.DELTA_TYPE_INPUT_JSON:
|
||||
if (delta.partialJson() != null) {
|
||||
block.inputJson.append(delta.partialJson());
|
||||
}
|
||||
break;
|
||||
case Delta.DELTA_TYPE_SIGNATURE:
|
||||
if (delta.signature() != null) {
|
||||
block.signature = delta.signature();
|
||||
}
|
||||
break;
|
||||
default:
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
private StreamBlock ensureBlock(final List<StreamBlock> blocks, final Integer index) {
|
||||
final var target = index != null ? index.intValue() : blocks.size();
|
||||
while (blocks.size() <= target) {
|
||||
blocks.add(new StreamBlock());
|
||||
}
|
||||
return blocks.get(target);
|
||||
}
|
||||
|
||||
private AnthropicMessageResponse aggregate(final String id, final String model, final String stopReason, final String stopSequence, final AnthropicUsage usage,
|
||||
final List<StreamBlock> blocks) {
|
||||
final List<AnthropicContentBlock> content = new ArrayList<>();
|
||||
for (final var block : blocks) {
|
||||
final var type = block.type != null ? block.type : "text";
|
||||
switch (type) {
|
||||
case "text":
|
||||
if (!block.text.isEmpty()) {
|
||||
content.add(AnthropicContentBlock.text(block.text.toString()));
|
||||
}
|
||||
break;
|
||||
case "tool_use":
|
||||
content.add(AnthropicContentBlock.toolUse(block.id, block.name, this.parseToolInput(block.inputJson.toString())));
|
||||
break;
|
||||
case "thinking":
|
||||
if (!block.thinking.isEmpty() || block.signature != null) {
|
||||
content.add(AnthropicContentBlock.thinking(block.thinking.toString(), block.signature));
|
||||
}
|
||||
break;
|
||||
case "redacted_thinking":
|
||||
if (block.data != null) {
|
||||
content.add(AnthropicContentBlock.redactedThinking(block.data));
|
||||
}
|
||||
break;
|
||||
default:
|
||||
LOGGER.debug("Ignoring unsupported streamed content block type: {}", type);
|
||||
break;
|
||||
}
|
||||
}
|
||||
return new AnthropicMessageResponse(id, "message", "assistant", model, List.copyOf(content), stopReason, stopSequence, usage);
|
||||
}
|
||||
|
||||
Completion mapResponse(final AnthropicMessageResponse response, final ModelProfile modelProfile, final LocalDateTime start, final Conversation conversation) {
|
||||
final var contentBuilder = new StringBuilder();
|
||||
final var reasoningBuilder = new StringBuilder();
|
||||
final List<ToolCall> toolCalls = new ArrayList<>();
|
||||
final List<Map<String, Object>> outputItems = new ArrayList<>();
|
||||
|
||||
for (final var block : response.content() == null ? List.<AnthropicContentBlock>of() : response.content()) {
|
||||
if (block == null || block.type() == null) {
|
||||
continue;
|
||||
}
|
||||
switch (block.type()) {
|
||||
case "text":
|
||||
if (block.text() != null) {
|
||||
contentBuilder.append(block.text());
|
||||
}
|
||||
break;
|
||||
case "thinking":
|
||||
if (block.thinking() != null && !block.thinking().isEmpty()) {
|
||||
if (!reasoningBuilder.isEmpty()) {
|
||||
reasoningBuilder.append("\n");
|
||||
}
|
||||
reasoningBuilder.append(block.thinking());
|
||||
}
|
||||
final var thinkingItem = new LinkedHashMap<String, Object>();
|
||||
thinkingItem.put("type", "thinking");
|
||||
thinkingItem.put("thinking", block.thinking() != null ? block.thinking() : "");
|
||||
thinkingItem.put("signature", block.signature() != null ? block.signature() : "");
|
||||
outputItems.add(Map.copyOf(thinkingItem));
|
||||
break;
|
||||
case "redacted_thinking":
|
||||
if (block.data() != null) {
|
||||
final Map<String, Object> redactedItem = new LinkedHashMap<String, Object>();
|
||||
redactedItem.put("type", "redacted_thinking");
|
||||
redactedItem.put("data", block.data());
|
||||
outputItems.add(Map.copyOf(redactedItem));
|
||||
}
|
||||
break;
|
||||
case "tool_use":
|
||||
toolCalls.add(new ToolCall(block.id(), "tool_use", block.name(), this.serializeToolInput(block.input())));
|
||||
break;
|
||||
default:
|
||||
LOGGER.debug("Ignoring unsupported response content block type: {}", block.type());
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
final var content = contentBuilder.toString();
|
||||
final var reasoning = reasoningBuilder.toString();
|
||||
return new Completion(content, reasoning, this.mapFinishReason(response.stopReason(), toolCalls),
|
||||
this.createCompletionStatistics(modelProfile, start, response.usage(), conversation, content, reasoning),
|
||||
List.copyOf(toolCalls), List.copyOf(outputItems));
|
||||
}
|
||||
|
||||
CompletionFinishReason mapFinishReason(final String stopReason, final List<ToolCall> toolCalls) {
|
||||
if (!toolCalls.isEmpty()) {
|
||||
return CompletionFinishReason.TOOL_CALLS;
|
||||
}
|
||||
if (stopReason == null) {
|
||||
return CompletionFinishReason.DONE;
|
||||
}
|
||||
return switch (stopReason) {
|
||||
case AnthropicMessageResponse.STOP_REASON_MAX_TOKENS -> CompletionFinishReason.TOKEN_LIMIT_REACHED;
|
||||
case AnthropicMessageResponse.STOP_REASON_REFUSAL -> CompletionFinishReason.CONTENT_FILTERED;
|
||||
default -> CompletionFinishReason.DONE; // end_turn, stop_sequence, pause_turn
|
||||
};
|
||||
}
|
||||
|
||||
private CompletionStatistics createCompletionStatistics(final ModelProfile modelProfile, final LocalDateTime start, final AnthropicUsage usage,
|
||||
final Conversation conversation, final String content, final String reasoning) {
|
||||
int inTokens = 0;
|
||||
int cachedInTokens = 0;
|
||||
int outTokens = 0;
|
||||
int reasoningOutTokens = 0;
|
||||
|
||||
if (usage != null) {
|
||||
inTokens = usage.inputTokens() != null ? usage.inputTokens().intValue() : 0;
|
||||
cachedInTokens = usage.cacheReadInputTokens() != null ? usage.cacheReadInputTokens().intValue() : 0;
|
||||
inTokens += usage.cacheCreationInputTokens() != null ? usage.cacheCreationInputTokens().intValue() : 0;
|
||||
outTokens = usage.outputTokens() != null ? usage.outputTokens().intValue() : 0;
|
||||
}
|
||||
if (reasoningOutTokens <= 0 && !reasoning.isBlank()) {
|
||||
reasoningOutTokens = this.tokenizerResolver.count(reasoning, modelProfile);
|
||||
outTokens = Math.max(0, outTokens - reasoningOutTokens);
|
||||
}
|
||||
if (inTokens <= 0) {
|
||||
inTokens = this.tokenizerResolver.count(this.createPromptForCounting(conversation), modelProfile);
|
||||
}
|
||||
if (outTokens <= 0 && !content.isBlank()) {
|
||||
outTokens = this.tokenizerResolver.count(content, modelProfile);
|
||||
}
|
||||
|
||||
var cost = BigDecimal.ZERO;
|
||||
cost = cost.add(BigDecimal.valueOf(inTokens).multiply(modelProfile.costPerInToken()));
|
||||
cost = cost.add(BigDecimal.valueOf(cachedInTokens).multiply(modelProfile.costPerCachedInToken()));
|
||||
cost = cost.add(BigDecimal.valueOf(outTokens).multiply(modelProfile.costPerOutToken()));
|
||||
cost = cost.add(BigDecimal.valueOf(reasoningOutTokens).multiply(modelProfile.costPerReasoningOutToken()));
|
||||
final var statistics = new CompletionStatistics(1, Duration.between(start, LocalDateTime.now()), inTokens, cachedInTokens, outTokens, reasoningOutTokens, cost, false);
|
||||
LOGGER.debug("{}", statistics);
|
||||
return statistics;
|
||||
}
|
||||
|
||||
private String createPromptForCounting(final Conversation conversation) {
|
||||
final var builder = new StringBuilder();
|
||||
for (final var turn : conversation.getConversation()) {
|
||||
builder.append(turn.type().name());
|
||||
builder.append(':');
|
||||
builder.append(turn.text() != null ? turn.text() : "");
|
||||
builder.append('\n');
|
||||
for (final var toolCall : turn.toolCalls()) {
|
||||
builder.append(toolCall.name());
|
||||
builder.append(':');
|
||||
builder.append(toolCall.arguments());
|
||||
builder.append('\n');
|
||||
}
|
||||
}
|
||||
return builder.toString();
|
||||
}
|
||||
|
||||
private String getPrimaryModelName(final ModelProfile modelProfile) {
|
||||
if (modelProfile.modelName() != null && !modelProfile.modelName().isBlank()) {
|
||||
return modelProfile.modelName();
|
||||
}
|
||||
return this.getModelNames(modelProfile, false).stream().findFirst().orElse("");
|
||||
}
|
||||
|
||||
private String getErrorMessage(final String errorBody) {
|
||||
try {
|
||||
final var errorResponse = this.objectMapper.readValue(errorBody, AnthropicErrorResponse.class);
|
||||
if (errorResponse != null && errorResponse.error() != null && errorResponse.error().message() != null && !errorResponse.error().message().isBlank()) {
|
||||
return errorResponse.error().message();
|
||||
}
|
||||
}
|
||||
catch (final JacksonException e) {
|
||||
LOGGER.debug("Failed to parse error response body", e);
|
||||
}
|
||||
return errorBody;
|
||||
}
|
||||
|
||||
private Map<String, Object> parseToolInput(final String arguments) {
|
||||
if (arguments == null || arguments.isBlank()) {
|
||||
return Map.of();
|
||||
}
|
||||
try {
|
||||
final var parsed = this.objectMapper.readValue(arguments, TOOL_INPUT_TYPE);
|
||||
return parsed == null ? Map.of() : parsed;
|
||||
}
|
||||
catch (final JacksonException e) {
|
||||
LOGGER.warn("Tool call arguments are not a valid JSON object, sending empty input: {}", arguments);
|
||||
return Map.of();
|
||||
}
|
||||
}
|
||||
|
||||
private String serializeToolInput(final Map<String, Object> input) {
|
||||
if (input == null) {
|
||||
return null;
|
||||
}
|
||||
try {
|
||||
return this.objectMapper.writeValueAsString(input);
|
||||
}
|
||||
catch (final JacksonException e) {
|
||||
throw new IllegalStateException("Cannot serialize tool call arguments", e);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Normalizes the configured base URL by removing a trailing
|
||||
* {@code /v1/messages} endpoint suffix (the request path is appended separately).
|
||||
*/
|
||||
static String stripMessagesSuffix(final String url) {
|
||||
if (url != null && url.endsWith(MESSAGES_PATH)) {
|
||||
final var stripped = url.substring(0, url.length() - MESSAGES_PATH.length());
|
||||
return stripped.isEmpty() ? url : stripped;
|
||||
}
|
||||
return url;
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolves the messages endpoint path relative to the (suffix-normalized)
|
||||
* configured base URL.
|
||||
*/
|
||||
static String messagesPath(final String url) {
|
||||
if (url == null) {
|
||||
return MESSAGES_PATH;
|
||||
}
|
||||
if (url.endsWith("/v1/") || url.endsWith("/v1")) {
|
||||
return "/messages";
|
||||
}
|
||||
return MESSAGES_PATH;
|
||||
}
|
||||
|
||||
private AnthropicDialect getAnthropicDialect() {
|
||||
return this.anthropicDialect == null ? DEFAULT_ANTHROPIC_DIALECT : this.anthropicDialect;
|
||||
}
|
||||
|
||||
private static final class StreamBlock {
|
||||
|
||||
private String type;
|
||||
private String id;
|
||||
private String name;
|
||||
private String signature;
|
||||
private String data;
|
||||
private final StringBuilder text = new StringBuilder();
|
||||
private final StringBuilder thinking = new StringBuilder();
|
||||
private final StringBuilder inputJson = new StringBuilder();
|
||||
}
|
||||
|
||||
@Override
|
||||
public String toString() {
|
||||
return "AnthropicClient []";
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,50 @@
|
||||
package de.extio.lmlib.client.anthropic;
|
||||
|
||||
import java.util.Map;
|
||||
|
||||
import com.fasterxml.jackson.annotation.JsonProperty;
|
||||
|
||||
/**
|
||||
* Anthropic content block used for both request building and response parsing.
|
||||
* <p>
|
||||
* Depending on {@code type}, only a subset of the fields is used:
|
||||
* <ul>
|
||||
* <li>{@code text}: {@code text}</li>
|
||||
* <li>{@code tool_use}: {@code id}, {@code name}, {@code input}</li>
|
||||
* <li>{@code tool_result}: {@code toolUseId}, {@code content}, {@code isError}</li>
|
||||
* <li>{@code thinking}: {@code thinking}, {@code signature}</li>
|
||||
* <li>{@code redacted_thinking}: {@code data}</li>
|
||||
* </ul>
|
||||
*/
|
||||
public record AnthropicContentBlock(String type,
|
||||
String text,
|
||||
String id,
|
||||
String name,
|
||||
Map<String, Object> input,
|
||||
@JsonProperty("tool_use_id") String toolUseId,
|
||||
Object content,
|
||||
@JsonProperty("is_error") Boolean isError,
|
||||
String thinking,
|
||||
String signature,
|
||||
String data) {
|
||||
|
||||
public static AnthropicContentBlock text(final String text) {
|
||||
return new AnthropicContentBlock("text", text, null, null, null, null, null, null, null, null, null);
|
||||
}
|
||||
|
||||
public static AnthropicContentBlock toolUse(final String id, final String name, final Map<String, Object> input) {
|
||||
return new AnthropicContentBlock("tool_use", null, id, name, input, null, null, null, null, null, null);
|
||||
}
|
||||
|
||||
public static AnthropicContentBlock toolResult(final String toolUseId, final String content) {
|
||||
return new AnthropicContentBlock("tool_result", null, null, null, null, toolUseId, content, null, null, null, null);
|
||||
}
|
||||
|
||||
public static AnthropicContentBlock thinking(final String thinking, final String signature) {
|
||||
return new AnthropicContentBlock("thinking", null, null, null, null, null, null, null, thinking, signature, null);
|
||||
}
|
||||
|
||||
public static AnthropicContentBlock redactedThinking(final String data) {
|
||||
return new AnthropicContentBlock("redacted_thinking", null, null, null, null, null, null, null, null, null, data);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,45 @@
|
||||
package de.extio.lmlib.client.anthropic;
|
||||
|
||||
import de.extio.lmlib.profile.ModelProfile;
|
||||
|
||||
/**
|
||||
* Optional dialect hook for the Anthropic Messages API client.
|
||||
* <p>
|
||||
* Applications can register this as a Spring bean to adapt request shaping for
|
||||
* Anthropic-compatible providers that deviate from the official API surface.
|
||||
* <p>
|
||||
* If no bean is present, lm-lib uses the defaults defined on this interface.
|
||||
*/
|
||||
public interface AnthropicDialect {
|
||||
|
||||
/**
|
||||
* The minimum extended thinking budget accepted by the Anthropic API.
|
||||
*/
|
||||
int MIN_THINKING_BUDGET_TOKENS = 1024;
|
||||
|
||||
/**
|
||||
* Returns the {@code anthropic-version} request header value.
|
||||
*
|
||||
* @param modelProfile the resolved model profile
|
||||
* @return the API version header value
|
||||
*/
|
||||
default String anthropicVersion(final ModelProfile modelProfile) {
|
||||
return "2023-06-01";
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns the extended thinking budget in tokens.
|
||||
* <p>
|
||||
* Thinking is disabled by default. When a budget is returned, the request
|
||||
* includes {@code thinking: {"type": "enabled", "budget_tokens": N}} with the
|
||||
* budget clamped to the API limits (at least {@value #MIN_THINKING_BUDGET_TOKENS}
|
||||
* and below {@code max_tokens}). Thinking blocks returned by the model are
|
||||
* exposed as reasoning and replayed unchanged in follow-up tool-call rounds.
|
||||
*
|
||||
* @param modelProfile the resolved model profile
|
||||
* @return the thinking budget in tokens, or {@code null} to disable thinking
|
||||
*/
|
||||
default Integer thinkingBudgetTokens(final ModelProfile modelProfile) {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,11 @@
|
||||
package de.extio.lmlib.client.anthropic;
|
||||
|
||||
/**
|
||||
* Anthropic API error detail.
|
||||
*
|
||||
* @param type the error type (e.g. {@code invalid_request_error})
|
||||
* @param message the human-readable error message
|
||||
*/
|
||||
public record AnthropicError(String type, String message) {
|
||||
|
||||
}
|
||||
@@ -0,0 +1,11 @@
|
||||
package de.extio.lmlib.client.anthropic;
|
||||
|
||||
/**
|
||||
* Anthropic API error response body.
|
||||
*
|
||||
* @param type the response type ({@code error})
|
||||
* @param error the error detail
|
||||
*/
|
||||
public record AnthropicErrorResponse(String type, AnthropicError error) {
|
||||
|
||||
}
|
||||
@@ -0,0 +1,28 @@
|
||||
package de.extio.lmlib.client.anthropic;
|
||||
|
||||
/**
|
||||
* A single input message of the Messages API.
|
||||
* <p>
|
||||
* {@code content} is either a plain string or a list of content blocks.
|
||||
*
|
||||
* @param role the message role ({@code user} or {@code assistant})
|
||||
* @param content string or list of content blocks
|
||||
*/
|
||||
public record AnthropicMessage(String role, Object content) {
|
||||
|
||||
public static AnthropicMessage user(final String text) {
|
||||
return new AnthropicMessage("user", text);
|
||||
}
|
||||
|
||||
public static AnthropicMessage userBlocks(final Object contentBlocks) {
|
||||
return new AnthropicMessage("user", contentBlocks);
|
||||
}
|
||||
|
||||
public static AnthropicMessage assistant(final String text) {
|
||||
return new AnthropicMessage("assistant", text);
|
||||
}
|
||||
|
||||
public static AnthropicMessage assistantBlocks(final Object contentBlocks) {
|
||||
return new AnthropicMessage("assistant", contentBlocks);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,34 @@
|
||||
package de.extio.lmlib.client.anthropic;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
import com.fasterxml.jackson.annotation.JsonProperty;
|
||||
|
||||
/**
|
||||
* Anthropic Messages API create-message request body.
|
||||
*
|
||||
* @param model the model name to generate the completion with
|
||||
* @param maxTokens the maximum number of tokens to generate (required by the API)
|
||||
* @param system the top-level system prompt (Anthropic has no system message role)
|
||||
* @param messages the conversational turns (user/assistant only)
|
||||
* @param temperature the sampling temperature
|
||||
* @param topP the nucleus sampling probability
|
||||
* @param stopSequences custom stop sequences
|
||||
* @param stream whether to stream the response via server-sent events
|
||||
* @param tools the tool definitions the model may use
|
||||
* @param toolChoice how the model should use the provided tools
|
||||
* @param thinking the extended thinking configuration
|
||||
*/
|
||||
public record AnthropicMessageRequest(String model,
|
||||
@JsonProperty("max_tokens") Integer maxTokens,
|
||||
String system,
|
||||
List<AnthropicMessage> messages,
|
||||
Double temperature,
|
||||
@JsonProperty("top_p") Double topP,
|
||||
@JsonProperty("stop_sequences") List<String> stopSequences,
|
||||
Boolean stream,
|
||||
List<AnthropicTool> tools,
|
||||
@JsonProperty("tool_choice") AnthropicToolChoice toolChoice,
|
||||
AnthropicThinkingConfig thinking) {
|
||||
|
||||
}
|
||||
@@ -0,0 +1,34 @@
|
||||
package de.extio.lmlib.client.anthropic;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
import com.fasterxml.jackson.annotation.JsonProperty;
|
||||
|
||||
/**
|
||||
* Anthropic Messages API response body.
|
||||
*
|
||||
* @param id the message id
|
||||
* @param type the response type ({@code message})
|
||||
* @param role the message role ({@code assistant})
|
||||
* @param model the model that generated the response
|
||||
* @param content the generated content blocks
|
||||
* @param stopReason why generation stopped
|
||||
* @param stopSequence the matched custom stop sequence, if any
|
||||
* @param usage the token usage information
|
||||
*/
|
||||
public record AnthropicMessageResponse(String id,
|
||||
String type,
|
||||
String role,
|
||||
String model,
|
||||
List<AnthropicContentBlock> content,
|
||||
@JsonProperty("stop_reason") String stopReason,
|
||||
@JsonProperty("stop_sequence") String stopSequence,
|
||||
AnthropicUsage usage) {
|
||||
|
||||
public static final String STOP_REASON_END_TURN = "end_turn";
|
||||
public static final String STOP_REASON_MAX_TOKENS = "max_tokens";
|
||||
public static final String STOP_REASON_STOP_SEQUENCE = "stop_sequence";
|
||||
public static final String STOP_REASON_TOOL_USE = "tool_use";
|
||||
public static final String STOP_REASON_PAUSE_TURN = "pause_turn";
|
||||
public static final String STOP_REASON_REFUSAL = "refusal";
|
||||
}
|
||||
@@ -0,0 +1,69 @@
|
||||
package de.extio.lmlib.client.anthropic;
|
||||
|
||||
import com.fasterxml.jackson.annotation.JsonProperty;
|
||||
|
||||
/**
|
||||
* One server-sent event of a streamed Anthropic Messages API response.
|
||||
* <p>
|
||||
* Depending on {@code type}, only a subset of the fields is populated:
|
||||
* <ul>
|
||||
* <li>{@code message_start}: {@code message}</li>
|
||||
* <li>{@code content_block_start}: {@code index}, {@code contentBlock}</li>
|
||||
* <li>{@code content_block_delta}: {@code index}, {@code delta}</li>
|
||||
* <li>{@code content_block_stop}: {@code index}</li>
|
||||
* <li>{@code message_delta}: {@code delta}, {@code usage}</li>
|
||||
* <li>{@code error}: {@code error}</li>
|
||||
* </ul>
|
||||
*
|
||||
* @param type the event type
|
||||
* @param index the content block index
|
||||
* @param contentBlock the started content block
|
||||
* @param delta the incremental payload
|
||||
* @param message the initial message payload
|
||||
* @param usage the usage payload of {@code message_delta} events
|
||||
* @param error the error payload of {@code error} events
|
||||
*/
|
||||
public record AnthropicStreamEvent(String type,
|
||||
Integer index,
|
||||
@JsonProperty("content_block") AnthropicContentBlock contentBlock,
|
||||
AnthropicStreamEvent.Delta delta,
|
||||
AnthropicMessageResponse message,
|
||||
AnthropicUsage usage,
|
||||
AnthropicError error) {
|
||||
|
||||
public static final String EVENT_MESSAGE_START = "message_start";
|
||||
public static final String EVENT_CONTENT_BLOCK_START = "content_block_start";
|
||||
public static final String EVENT_CONTENT_BLOCK_DELTA = "content_block_delta";
|
||||
public static final String EVENT_CONTENT_BLOCK_STOP = "content_block_stop";
|
||||
public static final String EVENT_MESSAGE_DELTA = "message_delta";
|
||||
public static final String EVENT_MESSAGE_STOP = "message_stop";
|
||||
public static final String EVENT_PING = "ping";
|
||||
public static final String EVENT_ERROR = "error";
|
||||
|
||||
/**
|
||||
* Incremental payload of delta events.
|
||||
*
|
||||
* @param type the delta type ({@code text_delta}, {@code thinking_delta},
|
||||
* {@code input_json_delta}, {@code signature_delta}; {@code message_delta}
|
||||
* events carry no type)
|
||||
* @param text the appended text for {@code text_delta}
|
||||
* @param thinking the appended thinking text for {@code thinking_delta}
|
||||
* @param partialJson the appended tool input JSON fragment for {@code input_json_delta}
|
||||
* @param signature the block signature for {@code signature_delta}
|
||||
* @param stopReason the final stop reason for {@code message_delta}
|
||||
* @param stopSequence the matched stop sequence for {@code message_delta}
|
||||
*/
|
||||
public record Delta(String type,
|
||||
String text,
|
||||
String thinking,
|
||||
@JsonProperty("partial_json") String partialJson,
|
||||
String signature,
|
||||
@JsonProperty("stop_reason") String stopReason,
|
||||
@JsonProperty("stop_sequence") String stopSequence) {
|
||||
|
||||
public static final String DELTA_TYPE_TEXT = "text_delta";
|
||||
public static final String DELTA_TYPE_THINKING = "thinking_delta";
|
||||
public static final String DELTA_TYPE_INPUT_JSON = "input_json_delta";
|
||||
public static final String DELTA_TYPE_SIGNATURE = "signature_delta";
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
package de.extio.lmlib.client.anthropic;
|
||||
|
||||
import com.fasterxml.jackson.annotation.JsonProperty;
|
||||
|
||||
/**
|
||||
* Anthropic extended thinking configuration.
|
||||
*
|
||||
* @param type the thinking mode ({@code enabled} or {@code disabled})
|
||||
* @param budgetTokens the thinking budget in tokens (at least 1024, below max_tokens)
|
||||
*/
|
||||
public record AnthropicThinkingConfig(String type, @JsonProperty("budget_tokens") Integer budgetTokens) {
|
||||
|
||||
public static AnthropicThinkingConfig enabled(final Integer budgetTokens) {
|
||||
return new AnthropicThinkingConfig("enabled", budgetTokens);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
package de.extio.lmlib.client.anthropic;
|
||||
|
||||
import java.util.Map;
|
||||
|
||||
import com.fasterxml.jackson.annotation.JsonProperty;
|
||||
|
||||
/**
|
||||
* Anthropic client tool definition.
|
||||
*
|
||||
* @param name the tool name
|
||||
* @param description the tool description
|
||||
* @param inputSchema the JSON schema for the tool input
|
||||
*/
|
||||
public record AnthropicTool(String name, String description, @JsonProperty("input_schema") Map<String, Object> inputSchema) {
|
||||
|
||||
}
|
||||
@@ -0,0 +1,29 @@
|
||||
package de.extio.lmlib.client.anthropic;
|
||||
|
||||
import com.fasterxml.jackson.annotation.JsonProperty;
|
||||
|
||||
/**
|
||||
* Anthropic tool choice configuration.
|
||||
*
|
||||
* @param type one of {@code auto}, {@code any}, {@code tool} or {@code none}
|
||||
* @param name the tool name for {@code type=tool}
|
||||
* @param disableParallelToolUse whether to disable parallel tool use
|
||||
*/
|
||||
public record AnthropicToolChoice(String type, String name, @JsonProperty("disable_parallel_tool_use") Boolean disableParallelToolUse) {
|
||||
|
||||
public static AnthropicToolChoice auto() {
|
||||
return new AnthropicToolChoice("auto", null, null);
|
||||
}
|
||||
|
||||
public static AnthropicToolChoice any() {
|
||||
return new AnthropicToolChoice("any", null, null);
|
||||
}
|
||||
|
||||
public static AnthropicToolChoice none() {
|
||||
return new AnthropicToolChoice("none", null, null);
|
||||
}
|
||||
|
||||
public static AnthropicToolChoice tool(final String name) {
|
||||
return new AnthropicToolChoice("tool", name, null);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,35 @@
|
||||
package de.extio.lmlib.client.anthropic;
|
||||
|
||||
import com.fasterxml.jackson.annotation.JsonProperty;
|
||||
|
||||
/**
|
||||
* Anthropic token usage information.
|
||||
* <p>
|
||||
* {@code inputTokens} excludes cached token counts. Thinking tokens are billed as
|
||||
* part of {@code outputTokens}.
|
||||
*
|
||||
* @param inputTokens the number of non-cached input tokens
|
||||
* @param outputTokens the number of output tokens (including thinking)
|
||||
* @param cacheCreationInputTokens the number of input tokens written to the prompt cache
|
||||
* @param cacheReadInputTokens the number of input tokens read from the prompt cache
|
||||
*/
|
||||
public record AnthropicUsage(@JsonProperty("input_tokens") Integer inputTokens,
|
||||
@JsonProperty("output_tokens") Integer outputTokens,
|
||||
@JsonProperty("cache_creation_input_tokens") Integer cacheCreationInputTokens,
|
||||
@JsonProperty("cache_read_input_tokens") Integer cacheReadInputTokens) {
|
||||
|
||||
/**
|
||||
* Merges a later usage snapshot onto this one, preferring non-null values from
|
||||
* the later snapshot.
|
||||
*/
|
||||
public AnthropicUsage mergedWith(final AnthropicUsage later) {
|
||||
if (later == null) {
|
||||
return this;
|
||||
}
|
||||
return new AnthropicUsage(
|
||||
later.inputTokens() != null ? later.inputTokens() : this.inputTokens,
|
||||
later.outputTokens() != null ? later.outputTokens() : this.outputTokens,
|
||||
later.cacheCreationInputTokens() != null ? later.cacheCreationInputTokens() : this.cacheCreationInputTokens,
|
||||
later.cacheReadInputTokens() != null ? later.cacheReadInputTokens() : this.cacheReadInputTokens);
|
||||
}
|
||||
}
|
||||
@@ -42,6 +42,6 @@ public record ModelProfile(
|
||||
}
|
||||
|
||||
public static enum ModelProvider {
|
||||
OAI_TEXT_COMPLETION, OAI_CHAT_COMPLETION, OAI_RESPONSES, OPENROUTER, OLLAMA;
|
||||
OAI_TEXT_COMPLETION, OAI_CHAT_COMPLETION, OAI_RESPONSES, OPENROUTER, OLLAMA, ANTHROPIC;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,356 @@
|
||||
package de.extio.lmlib.client.anthropic;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.assertEquals;
|
||||
import static org.junit.jupiter.api.Assertions.assertInstanceOf;
|
||||
import static org.junit.jupiter.api.Assertions.assertNull;
|
||||
import static org.junit.jupiter.api.Assertions.assertThrows;
|
||||
import static org.junit.jupiter.api.Assertions.assertTrue;
|
||||
|
||||
import java.io.ByteArrayInputStream;
|
||||
import java.math.BigDecimal;
|
||||
import java.nio.charset.StandardCharsets;
|
||||
import java.time.LocalDateTime;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
import de.extio.lmlib.client.Chunk;
|
||||
import de.extio.lmlib.client.CompletionFinishReason;
|
||||
import de.extio.lmlib.client.Conversation;
|
||||
import de.extio.lmlib.client.Conversation.Turn;
|
||||
import de.extio.lmlib.client.Conversation.TurnType;
|
||||
import de.extio.lmlib.client.ToolCall;
|
||||
import de.extio.lmlib.client.ToolCallData;
|
||||
import de.extio.lmlib.client.ToolDefinition;
|
||||
import de.extio.lmlib.profile.ModelProfile;
|
||||
import de.extio.lmlib.profile.ModelProfile.ModelProvider;
|
||||
import de.extio.lmlib.token.TokenizerResolver;
|
||||
import tools.jackson.databind.json.JsonMapper;
|
||||
|
||||
class AnthropicClientRequestTest {
|
||||
|
||||
@Test
|
||||
void sendsSystemAsTopLevelParameterAndPlainMessages() {
|
||||
final var client = new TestAnthropicClient();
|
||||
final var conversation = Conversation.create("answer briefly", "hello");
|
||||
conversation.addTurn(new Turn(TurnType.ASSISTANT, "hi there"));
|
||||
conversation.addTurn(new Turn(TurnType.USER, "again"));
|
||||
|
||||
final var request = client.createRequest(conversation, this.createProfile(2048), false, null);
|
||||
|
||||
assertEquals("answer briefly", request.system());
|
||||
assertEquals("test-model", request.model());
|
||||
assertEquals(Integer.valueOf(2048), request.maxTokens());
|
||||
assertEquals(Boolean.FALSE, request.stream());
|
||||
assertEquals(3, request.messages().size());
|
||||
assertEquals("user", request.messages().get(0).role());
|
||||
assertEquals("hello", request.messages().get(0).content());
|
||||
assertEquals("assistant", request.messages().get(1).role());
|
||||
assertEquals("hi there", request.messages().get(1).content());
|
||||
assertEquals("again", request.messages().get(2).content());
|
||||
assertNull(request.tools());
|
||||
assertNull(request.toolChoice());
|
||||
assertNull(request.thinking());
|
||||
}
|
||||
|
||||
@Test
|
||||
void coalescesToolResultsIntoSingleUserMessage() {
|
||||
final var client = new TestAnthropicClient();
|
||||
final var firstCall = new ToolCall("toolu_1", "tool_use", "get_weather", "{\"city\":\"Berlin\"}");
|
||||
final var secondCall = new ToolCall("toolu_2", "tool_use", "get_time", "{\"zone\":\"CET\"}");
|
||||
final var conversation = Conversation.create("first?");
|
||||
conversation.addTurn(new Turn(TurnType.ASSISTANT, "", List.of(firstCall, secondCall), null));
|
||||
conversation.addTurn(new Turn(TurnType.TOOL, "sunny", null, "toolu_1"));
|
||||
conversation.addTurn(new Turn(TurnType.TOOL, "noon", null, "toolu_2"));
|
||||
|
||||
final var messages = client.createMessages(conversation, new StringBuilder());
|
||||
|
||||
assertEquals(3, messages.size());
|
||||
assertEquals("user", messages.get(0).role());
|
||||
|
||||
final var assistantBlocks = this.blocks(messages.get(1));
|
||||
assertEquals(2, assistantBlocks.size());
|
||||
final var toolUse = assertInstanceOf(AnthropicContentBlock.class, assistantBlocks.get(0));
|
||||
assertEquals("tool_use", toolUse.type());
|
||||
assertEquals("toolu_1", toolUse.id());
|
||||
assertEquals(Map.of("city", "Berlin"), toolUse.input());
|
||||
final var secondToolUse = assertInstanceOf(AnthropicContentBlock.class, assistantBlocks.get(1));
|
||||
assertEquals("toolu_2", secondToolUse.id());
|
||||
|
||||
final var toolResultBlocks = this.blocks(messages.get(2));
|
||||
assertEquals(2, toolResultBlocks.size());
|
||||
final var firstResult = assertInstanceOf(AnthropicContentBlock.class, toolResultBlocks.get(0));
|
||||
assertEquals("tool_result", firstResult.type());
|
||||
assertEquals("toolu_1", firstResult.toolUseId());
|
||||
assertEquals("sunny", firstResult.content());
|
||||
assertEquals("toolu_2", assertInstanceOf(AnthropicContentBlock.class, toolResultBlocks.get(1)).toolUseId());
|
||||
}
|
||||
|
||||
@Test
|
||||
void replaysThinkingBlocksBeforeToolUseBlocks() {
|
||||
final var client = new TestAnthropicClient();
|
||||
final var toolCall = new ToolCall("toolu_1", "tool_use", "get_weather", "{}");
|
||||
final Map<String, Object> thinkingItem = Map.of("type", "thinking", "thinking", "hmm", "signature", "sig");
|
||||
final var conversation = Conversation.create("q");
|
||||
conversation.addTurn(new Turn(TurnType.ASSISTANT, "", List.of(toolCall), null, null, List.of(thinkingItem)));
|
||||
|
||||
final var messages = client.createMessages(conversation, new StringBuilder());
|
||||
|
||||
final var blocks = this.blocks(messages.get(1));
|
||||
assertEquals(2, blocks.size());
|
||||
assertEquals(thinkingItem, blocks.get(0));
|
||||
final var toolUse = assertInstanceOf(AnthropicContentBlock.class, blocks.get(1));
|
||||
assertEquals("tool_use", toolUse.type());
|
||||
}
|
||||
|
||||
@Test
|
||||
void mapsToolChoiceVariants() {
|
||||
final var client = new TestAnthropicClient();
|
||||
final var tools = List.of(new ToolDefinition("get_weather", "weather", Map.of("type", "object"), null));
|
||||
|
||||
final var required = client.createToolChoice(new ToolCallData(tools, ToolCallData.TOOL_CHOICE_REQUIRED, null, null));
|
||||
assertEquals("any", required.type());
|
||||
|
||||
final var forced = client.createToolChoice(new ToolCallData(tools, ToolCallData.TOOL_CHOICE_AUTO, "get_weather", null));
|
||||
assertEquals("tool", forced.type());
|
||||
assertEquals("get_weather", forced.name());
|
||||
|
||||
final var none = client.createToolChoice(new ToolCallData(tools, ToolCallData.TOOL_CHOICE_NONE, null, null));
|
||||
assertEquals("none", none.type());
|
||||
|
||||
final var auto = client.createToolChoice(new ToolCallData(tools, ToolCallData.TOOL_CHOICE_AUTO, null, null));
|
||||
assertEquals("auto", auto.type());
|
||||
assertNull(auto.disableParallelToolUse());
|
||||
|
||||
final var sequential = client.createToolChoice(new ToolCallData(tools, null, null, Boolean.FALSE));
|
||||
assertEquals("auto", sequential.type());
|
||||
assertEquals(Boolean.TRUE, sequential.disableParallelToolUse());
|
||||
|
||||
final var unset = client.createToolChoice(new ToolCallData(tools, null, null, null));
|
||||
assertNull(unset);
|
||||
}
|
||||
|
||||
@Test
|
||||
void clampsThinkingBudgetToTokenLimit() {
|
||||
final var client = new TestAnthropicClient();
|
||||
|
||||
client.anthropicDialect = new AnthropicDialect() {
|
||||
@Override
|
||||
public Integer thinkingBudgetTokens(final ModelProfile modelProfile) {
|
||||
return Integer.valueOf(4096);
|
||||
}
|
||||
};
|
||||
final var clamped = client.createThinkingConfig(this.createProfile(2500));
|
||||
assertEquals("enabled", clamped.type());
|
||||
assertEquals(Integer.valueOf(2499), clamped.budgetTokens());
|
||||
|
||||
client.anthropicDialect = new AnthropicDialect() {
|
||||
@Override
|
||||
public Integer thinkingBudgetTokens(final ModelProfile modelProfile) {
|
||||
return Integer.valueOf(1500);
|
||||
}
|
||||
};
|
||||
final var fitted = client.createThinkingConfig(this.createProfile(2500));
|
||||
assertEquals(Integer.valueOf(1500), fitted.budgetTokens());
|
||||
|
||||
client.anthropicDialect = new AnthropicDialect() {
|
||||
@Override
|
||||
public Integer thinkingBudgetTokens(final ModelProfile modelProfile) {
|
||||
return Integer.valueOf(512);
|
||||
}
|
||||
};
|
||||
assertNull(client.createThinkingConfig(this.createProfile(2500)));
|
||||
|
||||
client.anthropicDialect = null;
|
||||
assertNull(client.createThinkingConfig(this.createProfile(2500)));
|
||||
}
|
||||
|
||||
@Test
|
||||
void mapsNonStreamResponseToCompletion() {
|
||||
final var client = new TestAnthropicClient();
|
||||
client.tokenizerResolver = new CountingTokenizerResolver();
|
||||
final var json = """
|
||||
{
|
||||
"id": "msg_1",
|
||||
"type": "message",
|
||||
"role": "assistant",
|
||||
"model": "test-model",
|
||||
"content": [
|
||||
{"type": "thinking", "thinking": "reasoning text", "signature": "sig1"},
|
||||
{"type": "text", "text": "Hello "},
|
||||
{"type": "text", "text": "world"},
|
||||
{"type": "tool_use", "id": "toolu_1", "name": "get_weather", "input": {"city": "Berlin"}}
|
||||
],
|
||||
"stop_reason": "tool_use",
|
||||
"stop_sequence": null,
|
||||
"usage": {"input_tokens": 100, "output_tokens": 42, "cache_creation_input_tokens": 5, "cache_read_input_tokens": 7}
|
||||
}
|
||||
""";
|
||||
final var response = JsonMapper.builder().build().readValue(json, AnthropicMessageResponse.class);
|
||||
|
||||
final var completion = client.mapResponse(response, this.createProfile(2048), LocalDateTime.now(), Conversation.create("hi"));
|
||||
|
||||
assertEquals("Hello world", completion.response());
|
||||
assertEquals("reasoning text", completion.reasoning());
|
||||
assertEquals(CompletionFinishReason.TOOL_CALLS, completion.finishReason());
|
||||
assertEquals(1, completion.toolCalls().size());
|
||||
final var toolCall = completion.toolCalls().get(0);
|
||||
assertEquals("toolu_1", toolCall.id());
|
||||
assertEquals("get_weather", toolCall.name());
|
||||
assertEquals("{\"city\":\"Berlin\"}", toolCall.arguments());
|
||||
assertEquals(1, completion.outputItems().size());
|
||||
assertEquals("thinking", completion.outputItems().get(0).get("type"));
|
||||
assertEquals("reasoning text", completion.outputItems().get(0).get("thinking"));
|
||||
assertEquals("sig1", completion.outputItems().get(0).get("signature"));
|
||||
// input_tokens + cache_creation_input_tokens
|
||||
assertEquals(105, completion.statistics().inTokens());
|
||||
assertEquals(7, completion.statistics().cachedInTokens());
|
||||
// output_tokens minus tokenizer-estimated reasoning (fixed count of 3)
|
||||
assertEquals(39, completion.statistics().outTokens());
|
||||
assertEquals(3, completion.statistics().reasoningOutTokens());
|
||||
}
|
||||
|
||||
@Test
|
||||
void mapsFinishReasons() {
|
||||
final var client = new TestAnthropicClient();
|
||||
assertEquals(CompletionFinishReason.DONE, client.mapFinishReason("end_turn", List.of()));
|
||||
assertEquals(CompletionFinishReason.DONE, client.mapFinishReason("stop_sequence", List.of()));
|
||||
assertEquals(CompletionFinishReason.DONE, client.mapFinishReason(null, List.of()));
|
||||
assertEquals(CompletionFinishReason.TOKEN_LIMIT_REACHED, client.mapFinishReason("max_tokens", List.of()));
|
||||
assertEquals(CompletionFinishReason.CONTENT_FILTERED, client.mapFinishReason("refusal", List.of()));
|
||||
assertEquals(CompletionFinishReason.TOOL_CALLS, client.mapFinishReason("tool_use", List.of(new ToolCall("id", "tool_use", "n", "{}"))));
|
||||
assertEquals(CompletionFinishReason.TOOL_CALLS, client.mapFinishReason("end_turn", List.of(new ToolCall("id", "tool_use", "n", "{}"))));
|
||||
}
|
||||
|
||||
@Test
|
||||
void readsStreamedResponseAndEmitsChunks() throws Exception {
|
||||
final var client = new TestAnthropicClient();
|
||||
client.tokenizerResolver = new CountingTokenizerResolver();
|
||||
final var sse = """
|
||||
event: message_start
|
||||
data: {"type": "message_start", "message": {"id": "msg_1", "type": "message", "role": "assistant", "model": "test-model", "content": [], "usage": {"input_tokens": 57, "output_tokens": 0}}}
|
||||
|
||||
: keep-alive
|
||||
|
||||
event: content_block_start
|
||||
data: {"type": "content_block_start", "index": 0, "content_block": {"type": "thinking", "thinking": "", "signature": ""}}
|
||||
|
||||
event: content_block_delta
|
||||
data: {"type": "content_block_delta", "index": 0, "delta": {"type": "thinking_delta", "thinking": "hmm "}}
|
||||
|
||||
event: content_block_delta
|
||||
data: {"type": "content_block_delta", "index": 0, "delta": {"type": "thinking_delta", "thinking": "yes"}}
|
||||
|
||||
event: content_block_stop
|
||||
data: {"type": "content_block_stop", "index": 0}
|
||||
|
||||
event: content_block_start
|
||||
data: {"type": "content_block_start", "index": 1, "content_block": {"type": "text", "text": ""}}
|
||||
|
||||
event: content_block_delta
|
||||
data: {"type": "content_block_delta", "index": 1, "delta": {"type": "text_delta", "text": "hello stream"}}
|
||||
|
||||
event: content_block_stop
|
||||
data: {"type": "content_block_stop", "index": 1}
|
||||
|
||||
event: message_delta
|
||||
data: {"type": "message_delta", "delta": {"stop_reason": "end_turn", "stop_sequence": null}, "usage": {"input_tokens": 57, "cache_read_input_tokens": 0, "output_tokens": 30}}
|
||||
|
||||
event: message_stop
|
||||
data: {"type": "message_stop"}
|
||||
|
||||
""";
|
||||
final var chunks = new ArrayList<Chunk>();
|
||||
|
||||
final var response = client.readStream(new ByteArrayInputStream(sse.getBytes(StandardCharsets.UTF_8)), chunks::add);
|
||||
|
||||
final var reasoningChunks = chunks.stream().filter(chunk -> chunk.reasoningContent() != null).toList();
|
||||
final var textChunks = chunks.stream().filter(chunk -> chunk.content() != null).toList();
|
||||
assertEquals(2, reasoningChunks.size());
|
||||
assertEquals(1, textChunks.size());
|
||||
assertEquals("hello stream", textChunks.get(0).content());
|
||||
|
||||
assertEquals("msg_1", response.id());
|
||||
assertEquals("end_turn", response.stopReason());
|
||||
assertEquals(Integer.valueOf(30), response.usage().outputTokens());
|
||||
assertEquals(2, response.content().size());
|
||||
assertEquals("hmm yes", response.content().get(0).thinking());
|
||||
assertEquals("hello stream", response.content().get(1).text());
|
||||
|
||||
final var completion = client.mapResponse(response, this.createProfile(2048), LocalDateTime.now(), Conversation.create("hi"));
|
||||
assertEquals("hello stream", completion.response());
|
||||
assertEquals("hmm yes", completion.reasoning());
|
||||
assertEquals(CompletionFinishReason.DONE, completion.finishReason());
|
||||
}
|
||||
|
||||
@Test
|
||||
void failsOnStreamErrorEvent() {
|
||||
final var client = new TestAnthropicClient();
|
||||
final var sse = """
|
||||
event: error
|
||||
data: {"type": "error", "error": {"type": "overloaded_error", "message": "Overloaded"}}
|
||||
|
||||
""";
|
||||
final var exception = assertThrows(IllegalStateException.class,
|
||||
() -> client.readStream(new ByteArrayInputStream(sse.getBytes(StandardCharsets.UTF_8)), chunk -> {
|
||||
}));
|
||||
assertTrue(exception.getMessage().contains("Overloaded"));
|
||||
}
|
||||
|
||||
@Test
|
||||
void requiresPositiveMaxTokens() {
|
||||
final var client = new TestAnthropicClient();
|
||||
assertThrows(IllegalStateException.class, () -> client.createRequest(Conversation.create("hi"), this.createProfile(0), false, null));
|
||||
}
|
||||
|
||||
@Test
|
||||
void resolvesMessagesPathVariants() {
|
||||
assertEquals("/v1/messages", AnthropicClient.messagesPath("http://127.0.0.1:5001"));
|
||||
assertEquals("/messages", AnthropicClient.messagesPath("http://127.0.0.1:5001/v1"));
|
||||
assertEquals("/messages", AnthropicClient.messagesPath("http://127.0.0.1:5001/v1/"));
|
||||
assertEquals("http://127.0.0.1:5001", AnthropicClient.stripMessagesSuffix("http://127.0.0.1:5001/v1/messages"));
|
||||
assertEquals("http://127.0.0.1:5001", AnthropicClient.stripMessagesSuffix("http://127.0.0.1:5001"));
|
||||
}
|
||||
|
||||
@SuppressWarnings("unchecked")
|
||||
private List<Object> blocks(final AnthropicMessage message) {
|
||||
return (List<Object>) message.content();
|
||||
}
|
||||
|
||||
private ModelProfile createProfile(final int maxTokens) {
|
||||
return new ModelProfile(
|
||||
"",
|
||||
"fallback",
|
||||
"cl100k_base",
|
||||
maxTokens,
|
||||
8192,
|
||||
0.4,
|
||||
1.0,
|
||||
ModelProvider.ANTHROPIC,
|
||||
"test-model",
|
||||
"http://localhost:5001",
|
||||
"test-key",
|
||||
BigDecimal.ZERO,
|
||||
BigDecimal.ZERO,
|
||||
BigDecimal.ZERO,
|
||||
BigDecimal.ZERO,
|
||||
"TEST");
|
||||
}
|
||||
|
||||
private static final class CountingTokenizerResolver extends TokenizerResolver {
|
||||
|
||||
private CountingTokenizerResolver() {
|
||||
super(List.of());
|
||||
}
|
||||
|
||||
@Override
|
||||
public int count(final String txt, final ModelProfile modelProfile) {
|
||||
return 3;
|
||||
}
|
||||
}
|
||||
|
||||
private static final class TestAnthropicClient extends AnthropicClient {
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,150 @@
|
||||
package de.extio.lmlib.client.anthropic;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.assertEquals;
|
||||
import static org.junit.jupiter.api.Assertions.assertFalse;
|
||||
import static org.junit.jupiter.api.Assertions.assertTrue;
|
||||
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
|
||||
import org.junit.jupiter.api.Disabled;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.boot.SpringBootConfiguration;
|
||||
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;
|
||||
import org.springframework.boot.test.context.SpringBootTest;
|
||||
import org.springframework.boot.test.context.SpringBootTest.WebEnvironment;
|
||||
import org.springframework.test.context.TestPropertySource;
|
||||
|
||||
import de.extio.lmlib.client.ClientService;
|
||||
import de.extio.lmlib.client.CompletionFinishReason;
|
||||
import de.extio.lmlib.client.Conversation;
|
||||
import de.extio.lmlib.client.ToolCall;
|
||||
import de.extio.lmlib.client.ToolCallData;
|
||||
import de.extio.lmlib.client.ToolDefinition;
|
||||
import de.extio.lmlib.client.ToolParameters;
|
||||
import de.extio.lmlib.grader.Grader2;
|
||||
import de.extio.lmlib.profile.ModelProfile;
|
||||
import de.extio.lmlib.profile.ModelProfileService;
|
||||
|
||||
@Disabled("This test requires a running Anthropic-compatible server at the configured URL")
|
||||
@SpringBootTest(webEnvironment = WebEnvironment.NONE)
|
||||
@SpringBootConfiguration
|
||||
@EnableAutoConfiguration
|
||||
@TestPropertySource("classpath:/application-test.properties")
|
||||
public class AnthropicClientTest {
|
||||
|
||||
private static final Logger LOGGER = LoggerFactory.getLogger(AnthropicClientTest.class);
|
||||
|
||||
@Autowired
|
||||
private AnthropicClient anthropicClient;
|
||||
|
||||
@Autowired
|
||||
private ModelProfileService modelProfileService;
|
||||
|
||||
@Autowired
|
||||
private ClientService clientService;
|
||||
|
||||
@Test
|
||||
void routesProfileToAnthropicClient() throws Exception {
|
||||
final var modelProfile = this.modelProfile();
|
||||
assertEquals(ModelProfile.ModelProvider.ANTHROPIC, this.clientService.getClient(modelProfile).getModelProvider());
|
||||
assertTrue(this.anthropicClient.supportsToolCalling(modelProfile));
|
||||
}
|
||||
|
||||
@Test
|
||||
void chatCompletion() throws Exception {
|
||||
final var modelProfile = this.modelProfile();
|
||||
final var completion = this.anthropicClient.conversation(
|
||||
modelProfile,
|
||||
Conversation.create("You are a helpful assistant", "Say the color is green"),
|
||||
null,
|
||||
false);
|
||||
|
||||
LOGGER.info("Reasoning: {}", completion.reasoning());
|
||||
LOGGER.info("Response: {}", completion.response());
|
||||
|
||||
assertFalse(completion.response().isBlank());
|
||||
assertTrue(Grader2.assessScoreBinary("Does the text mention the color green?", completion.response(), modelProfile, this.clientService));
|
||||
}
|
||||
|
||||
@Test
|
||||
void streamChatCompletion() throws Exception {
|
||||
final var modelProfile = this.modelProfile();
|
||||
final var conversation = Conversation.create("You are a helpful assistant", "I would like to play a game of chess.");
|
||||
conversation.addTurn(new Conversation.Turn(Conversation.TurnType.ASSISTANT, "Which opening do you choose?"));
|
||||
conversation.addTurn(new Conversation.Turn(Conversation.TurnType.USER, "You choose, but explain the opening to me."));
|
||||
final var responseBuilder = new StringBuilder();
|
||||
final var reasoningBuilder = new StringBuilder();
|
||||
final var completion = this.anthropicClient.streamConversation(modelProfile, conversation, chunk -> {
|
||||
if (chunk.reasoningContent() != null) {
|
||||
System.out.print(chunk.reasoningContent());
|
||||
reasoningBuilder.append(chunk.reasoningContent());
|
||||
}
|
||||
if (chunk.content() != null) {
|
||||
System.out.print(chunk.content());
|
||||
responseBuilder.append(chunk.content());
|
||||
}
|
||||
}, null, false);
|
||||
|
||||
assertEquals(completion.response(), responseBuilder.toString());
|
||||
if (!completion.reasoning().isBlank()) {
|
||||
assertEquals(completion.reasoning(), reasoningBuilder.toString());
|
||||
}
|
||||
assertTrue(Grader2.assessScoreBinary("Does the text explain a chess opening?", completion.response(), modelProfile, this.clientService));
|
||||
}
|
||||
|
||||
@Test
|
||||
void toolCallingChatCompletion() throws Exception {
|
||||
final var modelProfile = this.modelProfile();
|
||||
final var weatherTool = new ToolDefinition(
|
||||
"get_weather",
|
||||
"Gets the current weather for a given location.",
|
||||
ToolParameters.create(Map.of("location", "City and country, for example Berlin, Germany")),
|
||||
null);
|
||||
final var conversation = Conversation.create(
|
||||
"You are a helpful assistant. Use the provided tools when the user asks for location information. After receiving tool results, answer the question from the tool output.",
|
||||
"What is the weather in Berlin, Germany? Use the tool.");
|
||||
final var completion = this.anthropicClient.conversation(
|
||||
modelProfile,
|
||||
conversation,
|
||||
ToolCallData.required(List.of(weatherTool)),
|
||||
false);
|
||||
|
||||
LOGGER.info("Reasoning: {}", completion.reasoning());
|
||||
LOGGER.info("Response: {}", completion.response());
|
||||
LOGGER.info("Tool calls: {}", completion.toolCalls());
|
||||
|
||||
assertEquals(CompletionFinishReason.TOOL_CALLS, completion.finishReason());
|
||||
assertFalse(completion.toolCalls().isEmpty());
|
||||
final var weatherToolCall = this.findToolCall(completion.toolCalls(), "get_weather");
|
||||
assertTrue(weatherToolCall.arguments().contains("Berlin"));
|
||||
|
||||
conversation.appendToolCallRound(completion, List.of(
|
||||
new Conversation.ToolResult(ToolParameters.create()
|
||||
.add("forecast", "Berlin is rainy and 21C")
|
||||
.json(), weatherToolCall.id())));
|
||||
LOGGER.info("Conversation after tool call: {}", conversation.getConversation());
|
||||
|
||||
final var finalCompletion = this.anthropicClient.conversation(modelProfile, conversation, ToolCallData.auto(List.of(weatherTool)), false);
|
||||
LOGGER.info("Reasoning: {}", finalCompletion.reasoning());
|
||||
LOGGER.info("Response: {}", finalCompletion.response());
|
||||
LOGGER.info("Tool calls: {}", finalCompletion.toolCalls());
|
||||
assertEquals(CompletionFinishReason.DONE, finalCompletion.finishReason());
|
||||
assertTrue(Grader2.assessScoreBinary("Does the text say that Berlin is rainy and 21C?", finalCompletion.response(), modelProfile, this.clientService));
|
||||
}
|
||||
|
||||
private ModelProfile modelProfile() {
|
||||
return this.modelProfileService.getModelProfile("profile.model.anthropic");
|
||||
}
|
||||
|
||||
private ToolCall findToolCall(final List<ToolCall> toolCalls, final String toolName) {
|
||||
return toolCalls.stream()
|
||||
.filter(toolCall -> toolName.equals(toolCall.name()))
|
||||
.findFirst()
|
||||
.orElseThrow(() -> new IllegalStateException("Missing tool call: " + toolName));
|
||||
}
|
||||
|
||||
}
|
||||
@@ -9,6 +9,7 @@ profile.model.chatcompletion=local-chatcompletion
|
||||
profile.model.responses=local-responses
|
||||
profile.model.ollama=local-ollama-chatcompletion
|
||||
profile.model.openrouter=local-openrouter-chatcompletion
|
||||
profile.model.anthropic=local-anthropic
|
||||
#profile.model.azure=llama3.3-70b-azure
|
||||
|
||||
agent.threads=2
|
||||
|
||||
16
src/test/resources/local-anthropic.properties
Normal file
16
src/test/resources/local-anthropic.properties
Normal file
@@ -0,0 +1,16 @@
|
||||
category=ANTHROPIC
|
||||
prompts=
|
||||
tokenizer=fallback
|
||||
tokenEncoding=cl100k_base
|
||||
maxTokens=2500
|
||||
maxContextLength=180000
|
||||
temperature=0.3
|
||||
topP=0.95
|
||||
modelProvider=ANTHROPIC
|
||||
modelName=qwen3.8-flash-next-unsloth-ud-iq4_xs
|
||||
url=http://127.0.0.1:5001
|
||||
apiKey=test
|
||||
cost1MInTokens=0
|
||||
cost1MCachedInTokens=0
|
||||
cost1MOutTokens=0
|
||||
cost1MReasoningOutTokens=0
|
||||
Reference in New Issue
Block a user