diff --git a/README.md b/README.md index 378c517..6bd90f7 100644 --- a/README.md +++ b/README.md @@ -13,13 +13,13 @@ lm-lib is published under the Apache License 2.0. lm-lib is designed for applications that need more than single prompt-response calls. Its center of gravity is the agent framework: structured multi-step flows such as classification, planning, fan-out analysis, synthesis, grading, feedback loops, and other graph-based execution patterns. -It also supports direct completion use cases through a client abstraction layer. The built-in clients currently cover OpenAI-compatible chat and text completion endpoints as well as Ollama's native chat API, and the abstraction remains open for additional provider implementations over time. Applications can also provide their own client implementations by implementing the relevant interfaces. +It also supports direct completion use cases through a client abstraction layer. The built-in clients currently cover OpenAI-compatible chat and text completion endpoints, OpenRouter's OpenAI-compatible routing API, and Ollama's native chat API, and the abstraction remains open for additional provider implementations over time. Applications can also provide their own client implementations by implementing the relevant interfaces. -The current built-in client set supports OpenAI-compatible chat completions, legacy text completions, and native Ollama chat completions, with model-specific prompt strategies for text completion models. +The current built-in client set supports OpenAI-compatible chat completions, OpenRouter chat completions with cached metadata discovery, legacy text completions, and native Ollama chat completions, with model-specific prompt strategies for text completion models. ## Key Capabilities -- OpenAI-compatible chat and text completions plus native Ollama chat completions +- OpenAI-compatible chat and text completions, OpenRouter chat completions, plus native Ollama chat completions - Chat-completion tool calling with tool definitions, returned tool calls, and tool-result follow-up rounds - Streaming and non-streaming APIs - Separate reasoning capture for models that expose reasoning output @@ -242,7 +242,7 @@ Each profile file is a `.properties` resource on the classpath. | `maxContextLength` | Integer | Yes | Maximum total context length. | `16000`, `32768`, `128000` | | `temperature` | Double | Yes | Sampling temperature. Must be greater than zero. | `0.2`, `0.4`, `0.7` | | `topP` | Double | Yes | Nucleus sampling parameter. Must be greater than zero. | `1.0`, `0.9` | -| `modelProvider` | Enum | Yes | Completion client type. | `OAI_CHAT_COMPLETION`, `OAI_TEXT_COMPLETION`, `OLLAMA` | +| `modelProvider` | Enum | Yes | Completion client type. | `OAI_CHAT_COMPLETION`, `OAI_TEXT_COMPLETION`, `OPENROUTER`, `OLLAMA` | | `modelName` | String | No | Model name or deployment. Can be empty for endpoints where the model should be auto-detected. | `gpt-4`, `Llama-3.3-70B-Instruct`, `` | | `url` | String | Yes | Base API URL. | `https://api.openai.com/v1`, `http://localhost:5001` | | `apiKey` | String | No | Bearer token or Spring placeholder. Can be empty for local servers. | `sk-...`, `${apikey}`, `` | @@ -330,6 +330,7 @@ The built-in provider implementations currently support these types: ```properties modelProvider=OAI_CHAT_COMPLETION modelProvider=OAI_TEXT_COMPLETION +modelProvider=OPENROUTER modelProvider=OLLAMA ``` @@ -391,6 +392,40 @@ OpenAiProviderDialect legacyOpenAiProviderDialect() { } ``` +### OpenRouter Provider + +`OPENROUTER` targets OpenRouter's chat-completions API and its richer model discovery endpoints. + +The OpenRouter provider uses `/v1/chat/completions` for completions and caches metadata from `/v1/models`, `/v1/models/user`, and `/v1/models/{author}/{slug}/endpoints`. The cached metadata is used to avoid repeating discovery calls on every request and to derive model-aware routing behavior such as tool support, reasoning support, token-limit parameter selection, context limits, and pricing. + +Use `OPENROUTER` when you want OpenRouter's model catalog, user-filtered model availability, or provider-aggregated pricing and capability metadata while keeping the standard lm-lib chat-completion API. + +Typical profile: + +```properties +category=L +prompts= +tokenizer=jTokkit +tokenEncoding=cl100k_base +maxTokens=4000 +maxContextLength=128000 +temperature=0.7 +topP=0.95 +modelProvider=OPENROUTER +modelName= +url= +apiKey=${openrouter.apiKey} +cost1MInTokens=0 +cost1MCachedInTokens=0 +cost1MOutTokens=0 +cost1MReasoningOutTokens=0 +``` + +* `modelName` is the OpenRouter model id (example: `openai/gpt-chat-latest`). If `modelName` is left empty, lm-lib will use the first available free model. +* The `url` field is ignored by the OpenRouter client and always targets `https://openrouter.ai/api`. +* Token pricing is overridden by OpenRouter's model metadata +* `maxTokens` and `maxContextLength` are clamped at the actual model capabilities. Leave it 0 to use the model limits from OpenRouter's metadata. + ### Ollama Provider `OLLAMA` targets Ollama's native API surface rather than the OpenAI-compatible transport. @@ -405,7 +440,7 @@ lm-lib caches `/api/show` model details per resolved model. Thinking and tool de Tool calling is exposed through the same lm-lib API across providers that support it. The capability is model-aware rather than purely provider-wide, so callers should check `supportsToolCalling(modelProfile)` or `supportsToolCalling(modelCategory)` on `Client` or `ClientService` before enabling tool-aware flows. -In practice, text-completion clients do not support tools via api (but you have full control over the raw prompt), while chat-oriented clients may support them depending on the resolved backend and model. For Ollama, lm-lib probes cached `/api/show` capability metadata. For OpenAI-compatible chat providers, support is determined by the selected client type. +In practice, text-completion clients do not support tools via api (but you have full control over the raw prompt), while chat-oriented clients may support them depending on the resolved backend and model. For Ollama, lm-lib probes cached `/api/show` capability metadata. For OpenRouter, lm-lib probes cached model and endpoint metadata from the OpenRouter model APIs. For OpenAI-compatible chat providers, support is determined by the selected client type. The library keeps the caller-facing tool-calling flow uniform even when providers differ internally. You still pass `ToolDefinition` and `ToolCallData`, inspect `CompletionFinishReason.TOOL_CALLS`, and append tool results through the same conversation API. If a backend does not expose every request-side control, lm-lib keeps the public API stable and applies those controls on a best-effort basis. diff --git a/src/main/java/de/extio/lmlib/client/ClientAutoConfiguration.java b/src/main/java/de/extio/lmlib/client/ClientAutoConfiguration.java index 04d9cf1..ea2d19e 100644 --- a/src/main/java/de/extio/lmlib/client/ClientAutoConfiguration.java +++ b/src/main/java/de/extio/lmlib/client/ClientAutoConfiguration.java @@ -6,6 +6,8 @@ import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty; import org.springframework.context.annotation.Bean; +import de.extio.lmlib.client.openrouter.OpenRouterClient; +import de.extio.lmlib.client.openrouter.OpenRouterModelDiscovery; import de.extio.lmlib.client.ollama.OllamaClient; import de.extio.lmlib.client.oai.completion.chat.ChatCompletionClient; import de.extio.lmlib.client.oai.completion.text.TextCompletionClient; @@ -35,6 +37,18 @@ public class ClientAutoConfiguration { return new ChatCompletionClient(); } + @Bean + @ConditionalOnMissingBean + OpenRouterModelDiscovery openRouterModelDiscovery() { + return new OpenRouterModelDiscovery(); + } + + @Bean + @ConditionalOnMissingBean + OpenRouterClient openRouterClient() { + return new OpenRouterClient(); + } + @Bean @ConditionalOnMissingBean OllamaClient ollamaClient() { diff --git a/src/main/java/de/extio/lmlib/client/oai/completion/AbstractCompletionClient.java b/src/main/java/de/extio/lmlib/client/oai/completion/AbstractCompletionClient.java index 19b480a..1073650 100755 --- a/src/main/java/de/extio/lmlib/client/oai/completion/AbstractCompletionClient.java +++ b/src/main/java/de/extio/lmlib/client/oai/completion/AbstractCompletionClient.java @@ -265,13 +265,10 @@ public abstract class AbstractCompletionClient implements Client, DisposableBean outTokens = this.tokenizerResolver.count(response, modelProfile); } - // Calculate cost - inTokens and outTokens are already exclusive of cached/reasoning tokens - BigDecimal cost = BigDecimal.ZERO; - cost = cost.add(new BigDecimal(inTokens).multiply(modelProfile.costPerInToken())); - cost = cost.add(new BigDecimal(cachedInTokens).multiply(modelProfile.costPerCachedInToken())); - cost = cost.add(new BigDecimal(outTokens).multiply(modelProfile.costPerOutToken())); - cost = cost.add(new BigDecimal(reasoningOutTokens).multiply(modelProfile.costPerReasoningOutToken())); - + var cost = this.getUsageCost(usage); + if (cost.compareTo(BigDecimal.ZERO) <= 0) { + cost = this.calculateTokenCost(modelProfile, inTokens, cachedInTokens, outTokens, reasoningOutTokens); + } final var statistics = new de.extio.lmlib.client.CompletionStatistics( 1, Duration.between(start, LocalDateTime.now()), @@ -287,6 +284,51 @@ public abstract class AbstractCompletionClient implements Client, DisposableBean } return statistics; } + + private BigDecimal getUsageCost(final Usage usage) { + if (usage == null) { + return BigDecimal.ZERO; + } + + if (usage.getCost() != null && usage.getCost().compareTo(BigDecimal.ZERO) > 0) { + return usage.getCost(); + } + + final var costDetails = usage.getCostDetails(); + if (costDetails != null) { + if (costDetails.getUpstreamInferenceCost() != null) { + return costDetails.getUpstreamInferenceCost(); + } + + var cost = BigDecimal.ZERO; + if (costDetails.getUpstreamInferencePromptCost() != null) { + cost = cost.add(costDetails.getUpstreamInferencePromptCost()); + } + if (costDetails.getUpstreamInferenceCompletionsCost() != null) { + cost = cost.add(costDetails.getUpstreamInferenceCompletionsCost()); + } + if (costDetails.getUpstreamInferenceInputCost() != null) { + cost = cost.add(costDetails.getUpstreamInferenceInputCost()); + } + if (costDetails.getUpstreamInferenceOutputCost() != null) { + cost = cost.add(costDetails.getUpstreamInferenceOutputCost()); + } + if (cost.compareTo(BigDecimal.ZERO) > 0) { + return cost; + } + } + + return BigDecimal.ZERO; + } + + private BigDecimal calculateTokenCost(final ModelProfile modelProfile, int inTokens, int cachedInTokens, int outTokens, int reasoningOutTokens) { + var cost = BigDecimal.ZERO; + cost = cost.add(new BigDecimal(inTokens).multiply(modelProfile.costPerInToken())); + cost = cost.add(new BigDecimal(cachedInTokens).multiply(modelProfile.costPerCachedInToken())); + cost = cost.add(new BigDecimal(outTokens).multiply(modelProfile.costPerOutToken())); + cost = cost.add(new BigDecimal(reasoningOutTokens).multiply(modelProfile.costPerReasoningOutToken())); + return cost; + } @Override public void destroy() throws Exception { diff --git a/src/main/java/de/extio/lmlib/client/oai/completion/Usage.java b/src/main/java/de/extio/lmlib/client/oai/completion/Usage.java index db191a3..80a1bc2 100644 --- a/src/main/java/de/extio/lmlib/client/oai/completion/Usage.java +++ b/src/main/java/de/extio/lmlib/client/oai/completion/Usage.java @@ -1,5 +1,7 @@ package de.extio.lmlib.client.oai.completion; +import java.math.BigDecimal; + import com.fasterxml.jackson.annotation.*; public final class Usage { @@ -13,6 +15,12 @@ public final class Usage { private InputTokensDetails inputTokensDetails; private OutputTokensDetails outputTokensDetails; + + private BigDecimal cost; + + private Boolean byok; + + private CostDetails costDetails; @JsonProperty("completion_tokens") public int getCompletionTokens() { @@ -59,6 +67,11 @@ public final class Usage { public void setInputTokensDetails(final InputTokensDetails inputTokensDetails) { this.inputTokensDetails = inputTokensDetails; } + + @JsonProperty("prompt_tokens_details") + public void setPromptTokensDetails(final InputTokensDetails inputTokensDetails) { + this.inputTokensDetails = inputTokensDetails; + } @JsonProperty("output_tokens_details") public OutputTokensDetails getOutputTokensDetails() { @@ -68,6 +81,38 @@ public final class Usage { public void setOutputTokensDetails(final OutputTokensDetails outputTokensDetails) { this.outputTokensDetails = outputTokensDetails; } + + @JsonProperty("completion_tokens_details") + public void setCompletionTokensDetails(final OutputTokensDetails outputTokensDetails) { + this.outputTokensDetails = outputTokensDetails; + } + + @JsonProperty("cost") + public BigDecimal getCost() { + return this.cost; + } + + public void setCost(final BigDecimal cost) { + this.cost = cost; + } + + @JsonProperty("is_byok") + public Boolean getByok() { + return this.byok; + } + + public void setByok(final Boolean byok) { + this.byok = byok; + } + + @JsonProperty("cost_details") + public CostDetails getCostDetails() { + return this.costDetails; + } + + public void setCostDetails(final CostDetails costDetails) { + this.costDetails = costDetails; + } public int getCachedTokens() { return this.inputTokensDetails != null ? this.inputTokensDetails.getCachedTokens() : 0; @@ -79,7 +124,70 @@ public final class Usage { @Override public String toString() { - return "Usage [completionTokens=" + this.completionTokens + ", promptTokens=" + this.promptTokens + ", totalTokens=" + this.totalTokens + ", inputTokensDetails=" + this.inputTokensDetails + ", outputTokensDetails=" + this.outputTokensDetails + "]"; + return "Usage [completionTokens=" + this.completionTokens + ", promptTokens=" + this.promptTokens + ", totalTokens=" + this.totalTokens + ", inputTokensDetails=" + this.inputTokensDetails + ", outputTokensDetails=" + this.outputTokensDetails + ", cost=" + this.cost + ", byok=" + this.byok + ", costDetails=" + this.costDetails + "]"; + } + + public static final class CostDetails { + + private BigDecimal upstreamInferenceCost; + + private BigDecimal upstreamInferencePromptCost; + + private BigDecimal upstreamInferenceCompletionsCost; + + private BigDecimal upstreamInferenceInputCost; + + private BigDecimal upstreamInferenceOutputCost; + + @JsonProperty("upstream_inference_cost") + public BigDecimal getUpstreamInferenceCost() { + return this.upstreamInferenceCost; + } + + public void setUpstreamInferenceCost(final BigDecimal upstreamInferenceCost) { + this.upstreamInferenceCost = upstreamInferenceCost; + } + + @JsonProperty("upstream_inference_prompt_cost") + public BigDecimal getUpstreamInferencePromptCost() { + return this.upstreamInferencePromptCost; + } + + public void setUpstreamInferencePromptCost(final BigDecimal upstreamInferencePromptCost) { + this.upstreamInferencePromptCost = upstreamInferencePromptCost; + } + + @JsonProperty("upstream_inference_completions_cost") + public BigDecimal getUpstreamInferenceCompletionsCost() { + return this.upstreamInferenceCompletionsCost; + } + + public void setUpstreamInferenceCompletionsCost(final BigDecimal upstreamInferenceCompletionsCost) { + this.upstreamInferenceCompletionsCost = upstreamInferenceCompletionsCost; + } + + @JsonProperty("upstream_inference_input_cost") + public BigDecimal getUpstreamInferenceInputCost() { + return this.upstreamInferenceInputCost; + } + + public void setUpstreamInferenceInputCost(final BigDecimal upstreamInferenceInputCost) { + this.upstreamInferenceInputCost = upstreamInferenceInputCost; + } + + @JsonProperty("upstream_inference_output_cost") + public BigDecimal getUpstreamInferenceOutputCost() { + return this.upstreamInferenceOutputCost; + } + + public void setUpstreamInferenceOutputCost(final BigDecimal upstreamInferenceOutputCost) { + this.upstreamInferenceOutputCost = upstreamInferenceOutputCost; + } + + @Override + public String toString() { + return "CostDetails [upstreamInferenceCost=" + this.upstreamInferenceCost + ", upstreamInferencePromptCost=" + this.upstreamInferencePromptCost + ", upstreamInferenceCompletionsCost=" + this.upstreamInferenceCompletionsCost + ", upstreamInferenceInputCost=" + this.upstreamInferenceInputCost + ", upstreamInferenceOutputCost=" + this.upstreamInferenceOutputCost + "]"; + } } public static final class InputTokensDetails { diff --git a/src/main/java/de/extio/lmlib/client/oai/completion/chat/ChatCompletionClient.java b/src/main/java/de/extio/lmlib/client/oai/completion/chat/ChatCompletionClient.java index 8f5416e..8cba38d 100755 --- a/src/main/java/de/extio/lmlib/client/oai/completion/chat/ChatCompletionClient.java +++ b/src/main/java/de/extio/lmlib/client/oai/completion/chat/ChatCompletionClient.java @@ -101,6 +101,9 @@ public class ChatCompletionClient extends AbstractCompletionClient { if (line.startsWith("data: ")) { line = line.substring(6).trim(); } + if (line.startsWith(":")) { + continue; + } if (line.isEmpty()) { continue; } diff --git a/src/main/java/de/extio/lmlib/client/oai/completion/chat/ChatMessage.java b/src/main/java/de/extio/lmlib/client/oai/completion/chat/ChatMessage.java index 5c8c457..efb0f94 100644 --- a/src/main/java/de/extio/lmlib/client/oai/completion/chat/ChatMessage.java +++ b/src/main/java/de/extio/lmlib/client/oai/completion/chat/ChatMessage.java @@ -44,10 +44,12 @@ final class ChatMessage { } @JsonProperty("reasoning_content") + @JsonAlias("reasoning") public String getReasoningContent() { return reasoningContent; } + @JsonProperty("reasoning") public void setReasoningContent(final String reasoningContent) { this.reasoningContent = reasoningContent; } diff --git a/src/main/java/de/extio/lmlib/client/openrouter/OpenRouterClient.java b/src/main/java/de/extio/lmlib/client/openrouter/OpenRouterClient.java new file mode 100644 index 0000000..4db916d --- /dev/null +++ b/src/main/java/de/extio/lmlib/client/openrouter/OpenRouterClient.java @@ -0,0 +1,185 @@ +package de.extio.lmlib.client.openrouter; + +import java.util.List; +import java.util.function.Consumer; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.beans.factory.annotation.Autowired; + +import de.extio.lmlib.client.Chunk; +import de.extio.lmlib.client.Completion; +import de.extio.lmlib.client.Conversation; +import de.extio.lmlib.client.ToolCallData; +import de.extio.lmlib.client.oai.OpenAiProviderDialect; +import de.extio.lmlib.client.oai.completion.chat.ChatCompletionClient; +import de.extio.lmlib.profile.ModelCategory; +import de.extio.lmlib.profile.ModelProfile; +import de.extio.lmlib.profile.ModelProfile.ModelProvider; + +public class OpenRouterClient extends ChatCompletionClient { + + private static final Logger LOGGER = LoggerFactory.getLogger(OpenRouterClient.class); + + @Autowired + private OpenRouterModelDiscovery openRouterModelDiscovery; + + @Override + public Completion conversation(final ModelCategory modelCategory, final Conversation conversation, final ToolCallData toolCallData, final boolean skipCache) { + return this.streamConversation(modelCategory, conversation, null, toolCallData, skipCache); + } + + @Override + public Completion conversation(final ModelProfile modelProfile, final Conversation conversation, final ToolCallData toolCallData, final boolean skipCache) { + return this.streamConversation(modelProfile, conversation, null, toolCallData, skipCache); + } + + @Override + public Completion streamConversation(final ModelCategory modelCategory, final Conversation conversation, final Consumer chunkConsumer, final ToolCallData toolCallData, final boolean skipCache) { + final var modelProfile = this.modelProfileService.getModelProfile(modelCategory.getModelProfile(), modelCategory); + return this.streamConversation(modelProfile, conversation, chunkConsumer, toolCallData, skipCache); + } + + @Override + public Completion streamConversation(final ModelProfile modelProfile, final Conversation conversation, final Consumer chunkConsumer, final ToolCallData toolCallData, final boolean skipCache) { + if (modelProfile == null || modelProfile.modelProvider() != ModelProvider.OPENROUTER) { + throw new IllegalArgumentException("Invalid ModelProfile"); + } + return this.requestCompletion(conversation, modelProfile, chunkConsumer, toolCallData); + } + + @Override + public List getModelNames(final ModelProfile modelProfile, final boolean forceReload) { + if (modelProfile == null || modelProfile.modelProvider() != ModelProvider.OPENROUTER) { + return List.of(); + } + if (modelProfile.modelName() != null && !modelProfile.modelName().isBlank()) { + return List.of(modelProfile.modelName()); + } + return this.openRouterModelDiscovery.listModels(modelProfile, new OpenRouterModelQuery(true, null, List.of(), List.of()), forceReload).stream() + .map(OpenRouterModelsResponse.OpenRouterModel::getId) + .filter(modelName -> modelName != null && !modelName.isBlank()) + .toList(); + } + + @Override + public boolean supportsToolCalling(final ModelCategory modelCategory) { + final var modelProfile = this.modelProfileService.getModelProfile(modelCategory.getModelProfile(), modelCategory); + return this.supportsToolCalling(modelProfile); + } + + @Override + public boolean supportsToolCalling(final ModelProfile modelProfile) { + if (modelProfile == null || modelProfile.modelProvider() != ModelProvider.OPENROUTER) { + return false; + } + final var metadata = this.resolveMetadata(modelProfile); + return metadata != null && metadata.supportsToolCalling(); + } + + @Override + public ModelProvider getModelProvider() { + return ModelProvider.OPENROUTER; + } + + @Override + protected Completion requestCompletion(final Conversation conversation, final ModelProfile modelProfile, final Consumer chunkConsumer, final ToolCallData toolCallData) { + final var effectiveModelProfile = this.toEffectiveModelProfile(modelProfile); + LOGGER.debug("Requesting completion with OpenRouter model {} ", effectiveModelProfile.modelName()); + return super.requestCompletion(conversation, effectiveModelProfile, chunkConsumer, toolCallData); + } + + @Override + protected boolean sendUsage(final ModelProfile modelProfile) { + return false; + } + + @Override + protected boolean sendReasoning(final ModelProfile modelProfile) { + final var metadata = this.resolveMetadata(modelProfile); + return metadata != null && metadata.supportsReasoning(); + } + + @Override + protected OpenAiProviderDialect.ChatTokenLimitParameterMode chatTokenLimitParameterMode(final ModelProfile modelProfile) { + final var metadata = this.resolveMetadata(modelProfile); + return metadata == null ? OpenAiProviderDialect.ChatTokenLimitParameterMode.MAX_COMPLETION_TOKENS : metadata.tokenLimitParameterMode(); + } + + @Override + protected OpenAiProviderDialect.ReasoningEffort reasoningEffort(final ModelProfile modelProfile) { + if (!this.sendReasoning(modelProfile)) { + return null; + } + final var reasoningEffort = super.reasoningEffort(modelProfile); + return reasoningEffort == null ? OpenAiProviderDialect.ReasoningEffort.MEDIUM : reasoningEffort; + } + + private OpenRouterModelDiscovery.ResolvedOpenRouterModelMetadata resolveMetadata(final ModelProfile modelProfile) { + if (modelProfile == null || modelProfile.modelProvider() != ModelProvider.OPENROUTER) { + return null; + } + var modelName = modelProfile.modelName(); + if (modelName == null || modelName.isBlank()) { + modelName = this.openRouterModelDiscovery.listModels(modelProfile, new OpenRouterModelQuery(true, null, List.of(), List.of()), false).stream() + .filter(model -> model.getPricing() != null && "0".equals(model.getPricing().getPrompt()) && "0".equals(model.getPricing().getCompletion())) // Prefer free models if no model name specified + .map(OpenRouterModelsResponse.OpenRouterModel::getId) + .filter(id -> id != null && !id.isBlank()) + .findFirst() + .orElse(""); + } + if (modelName.isBlank()) { + return null; + } + return this.openRouterModelDiscovery.resolveModelMetadata(modelProfile, modelName, false); + } + + private ModelProfile toEffectiveModelProfile(final ModelProfile modelProfile) { + final var metadata = this.resolveMetadata(modelProfile); + if (metadata == null || metadata.modelId() == null || metadata.modelId().isBlank()) { + return modelProfile; + } + final var promptCost = metadata.promptCost(); + final var cachedPromptCost = metadata.cachedPromptCost(); + final var completionCost = metadata.completionCost(); + final var reasoningCost = metadata.reasoningCost(); + int maxContextLength; + if (modelProfile.maxContextLength() <= 0 && metadata.contextLength() <= 0) { + throw new IllegalArgumentException("OpenRouter metadata don't specify a valid context length. Therefore the model profile must specify a valid context length."); + } + else if (modelProfile.maxContextLength() > 0 && metadata.contextLength() > 0) { + maxContextLength = Math.min(modelProfile.maxContextLength(), metadata.contextLength()); + } + else { + maxContextLength = modelProfile.maxContextLength() > 0 ? modelProfile.maxContextLength() : metadata.contextLength(); + } + int maxTokens; + if (modelProfile.maxTokens() <= 0 && metadata.maxCompletionTokens() <= 0) { + throw new IllegalArgumentException("OpenRouter metadata don't specify a valid max completion tokens. Therefore the model profile must specify a valid max tokens."); + } + else if (modelProfile.maxTokens() > 0 && metadata.maxCompletionTokens() > 0) { + maxTokens = Math.min(modelProfile.maxTokens(), metadata.maxCompletionTokens()); + } + else { + maxTokens = modelProfile.maxTokens() > 0 ? modelProfile.maxTokens() : metadata.maxCompletionTokens(); + } + return new ModelProfile( + modelProfile.prompt(), + modelProfile.tokenizer(), + modelProfile.tokenEncoding(), + maxTokens, + maxContextLength, + modelProfile.temperature(), + modelProfile.topP(), + ModelProvider.OPENROUTER, + metadata.modelId(), + OpenRouterModelDiscovery.OPENROUTER_API_URL, + modelProfile.apiKey(), + promptCost != null ? promptCost : modelProfile.costPerInToken(), + cachedPromptCost != null ? cachedPromptCost : modelProfile.costPerCachedInToken(), + completionCost != null ? completionCost : modelProfile.costPerOutToken(), + reasoningCost != null ? reasoningCost : modelProfile.costPerReasoningOutToken(), + modelProfile.category()); + } + +} \ No newline at end of file diff --git a/src/main/java/de/extio/lmlib/client/openrouter/OpenRouterModelDiscovery.java b/src/main/java/de/extio/lmlib/client/openrouter/OpenRouterModelDiscovery.java new file mode 100644 index 0000000..2ccdfc6 --- /dev/null +++ b/src/main/java/de/extio/lmlib/client/openrouter/OpenRouterModelDiscovery.java @@ -0,0 +1,351 @@ +package de.extio.lmlib.client.openrouter; + +import java.math.BigDecimal; +import java.net.URLEncoder; +import java.nio.charset.StandardCharsets; +import java.util.List; +import java.util.Locale; +import java.util.Map; +import java.util.Objects; +import java.util.concurrent.ConcurrentHashMap; +import java.util.stream.Collectors; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.beans.factory.annotation.Qualifier; +import org.springframework.http.HttpMethod; +import org.springframework.web.client.RestClient; +import org.springframework.web.client.RestClientException; +import org.springframework.web.client.RestClientResponseException; + +import de.extio.lmlib.client.oai.OpenAiProviderDialect; +import de.extio.lmlib.profile.ModelProfile; +import de.extio.lmlib.profile.ModelProfile.ModelProvider; + +public class OpenRouterModelDiscovery { + + static final String OPENROUTER_API_URL = "https://openrouter.ai/api"; + + private static final Logger LOGGER = LoggerFactory.getLogger(OpenRouterModelDiscovery.class); + + @Autowired + @Qualifier("lmLibRestClientBuilder") + private RestClient.Builder restClientBuilder; + + private final Map> modelListCache = new ConcurrentHashMap<>(); + + private final Map endpointCache = new ConcurrentHashMap<>(); + + private final Map metadataCache = new ConcurrentHashMap<>(); + + public List listModels(final ModelProfile modelProfile, final OpenRouterModelQuery query_, final boolean forceReload) { + this.validateProfile(modelProfile); + final var query = query_ == null ? new OpenRouterModelQuery(false, null, List.of(), List.of()) : query_; + if (query.userFiltered() && query.hasServerSideFilters()) { + throw new IllegalArgumentException("OpenRouter /models/user does not support category, supported_parameters, or output_modalities filters"); + } + final var cacheKey = this.modelListCacheKey(modelProfile, query); + if (forceReload) { + this.modelListCache.remove(cacheKey); + } + return this.modelListCache.computeIfAbsent(cacheKey, key -> this.loadModels(modelProfile, query)); + } + + public OpenRouterModelEndpointsResponse.OpenRouterModelEndpoints getModelEndpoints(final ModelProfile modelProfile, final String modelName, final boolean forceReload) { + this.validateProfile(modelProfile); + if (modelName == null || modelName.isBlank()) { + return new OpenRouterModelEndpointsResponse.OpenRouterModelEndpoints(); + } + final var cacheKey = this.endpointCacheKey(modelProfile, modelName); + if (forceReload) { + final var endpoints = this.loadModelEndpoints(modelProfile, modelName, true); + this.endpointCache.put(cacheKey, endpoints); + return endpoints; + } + return this.endpointCache.computeIfAbsent(cacheKey, key -> this.loadModelEndpoints(modelProfile, modelName, false)); + } + + public ResolvedOpenRouterModelMetadata resolveModelMetadata(final ModelProfile modelProfile, final String modelName_, final boolean forceReload) { + this.validateProfile(modelProfile); + final var modelName = modelName_ == null || modelName_.isBlank() ? this.listModels(modelProfile, new OpenRouterModelQuery(true, null, List.of(), List.of()), forceReload) + .stream() + .map(OpenRouterModelsResponse.OpenRouterModel::getId) + .filter(Objects::nonNull) + .filter(id -> !id.isBlank()) + .findFirst() + .orElse("") : modelName_; + if (modelName.isBlank()) { + return new ResolvedOpenRouterModelMetadata("", null, null, null, List.of(), 0, 0); + } + final var cacheKey = this.metadataCacheKey(modelProfile, modelName); + if (forceReload) { + final var metadata = this.loadResolvedMetadata(modelProfile, modelName, true); + this.metadataCache.put(cacheKey, metadata); + return metadata; + } + return this.metadataCache.computeIfAbsent(cacheKey, key -> this.loadResolvedMetadata(modelProfile, modelName, false)); + } + + private List loadModels(final ModelProfile modelProfile, final OpenRouterModelQuery query) { + try { + var requestSpec = this.restClientBuilder.baseUrl(OPENROUTER_API_URL).build() + .method(HttpMethod.GET) + .uri(uriBuilder -> { + uriBuilder.path(query.userFiltered() ? "/v1/models/user" : "/v1/models"); + if (!query.userFiltered() && query.category() != null && !query.category().isBlank()) { + uriBuilder.queryParam("category", query.category()); + } + final var supportedParameters = this.join(query.supportedParameters()); + if (!query.userFiltered() && !supportedParameters.isBlank()) { + uriBuilder.queryParam("supported_parameters", supportedParameters); + } + final var outputModalities = this.join(query.outputModalities()); + if (!query.userFiltered() && !outputModalities.isBlank()) { + uriBuilder.queryParam("output_modalities", outputModalities); + } + return uriBuilder.build(); + }); + if (modelProfile.apiKey() != null && !modelProfile.apiKey().isBlank()) { + requestSpec = requestSpec.header("Authorization", "Bearer " + modelProfile.apiKey()); + } + final var response = requestSpec.retrieve().body(OpenRouterModelsResponse.class); + return response == null || response.getData() == null ? List.of() : List.copyOf(response.getData()); + } + catch (final RestClientResponseException e) { + LOGGER.warn("Failed to load OpenRouter models: {} {}", e.getStatusCode(), e.getStatusText()); + return List.of(); + } + catch (final RestClientException e) { + LOGGER.warn("Failed to load OpenRouter models: {}", e.getMessage()); + return List.of(); + } + catch (final RuntimeException e) { + LOGGER.warn("Failed to load OpenRouter models", e); + return List.of(); + } + } + + private OpenRouterModelEndpointsResponse.OpenRouterModelEndpoints loadModelEndpoints(final ModelProfile modelProfile, final String modelName, final boolean forceReload) { + try { + final var model = this.findModel(modelProfile, modelName, forceReload); + var requestSpec = this.restClientBuilder.baseUrl(OPENROUTER_API_URL).build() + .method(HttpMethod.GET) + .uri(this.resolveDetailsPath(model, modelName)); + if (modelProfile.apiKey() != null && !modelProfile.apiKey().isBlank()) { + requestSpec = requestSpec.header("Authorization", "Bearer " + modelProfile.apiKey()); + } + final var response = requestSpec.retrieve().body(OpenRouterModelEndpointsResponse.class); + return response == null || response.getData() == null ? new OpenRouterModelEndpointsResponse.OpenRouterModelEndpoints() : response.getData(); + } + catch (final RestClientResponseException e) { + if (e.getStatusCode().value() != 404) { + LOGGER.warn("Failed to load OpenRouter endpoints for {}: {} {}", modelName, e.getStatusCode(), e.getStatusText()); + } + return new OpenRouterModelEndpointsResponse.OpenRouterModelEndpoints(); + } + catch (final RestClientException e) { + LOGGER.warn("Failed to load OpenRouter endpoints for {}: {}", modelName, e.getMessage()); + return new OpenRouterModelEndpointsResponse.OpenRouterModelEndpoints(); + } + catch (final RuntimeException e) { + LOGGER.warn("Failed to load OpenRouter endpoints for {}", modelName, e); + return new OpenRouterModelEndpointsResponse.OpenRouterModelEndpoints(); + } + } + + private ResolvedOpenRouterModelMetadata loadResolvedMetadata(final ModelProfile modelProfile, final String modelName, final boolean forceReload) { + final var model = this.findModel(modelProfile, modelName, forceReload); + final var endpoints = this.getModelEndpoints(modelProfile, modelName, forceReload); + final var endpoint = this.preferredEndpoint(endpoints); + final var pricing = endpoint != null && this.hasPricing(endpoint.getPricing()) ? endpoint.getPricing() : model != null ? model.getPricing() : null; + final List supportedParameters = endpoint != null && endpoint.getSupportedParameters() != null && !endpoint.getSupportedParameters().isEmpty() + ? List.copyOf(endpoint.getSupportedParameters()) + : model != null && model.getSupportedParameters() != null ? List.copyOf(model.getSupportedParameters()) : List.of(); + final var contextLength = this.firstPositive( + endpoint != null ? endpoint.getContextLength() : null, + model != null && model.getTopProvider() != null ? model.getTopProvider().getContextLength() : null, + model != null ? model.getContextLength() : null, + model != null && model.getPerRequestLimits() != null ? model.getPerRequestLimits().getPromptTokens() : null); + final var maxCompletionTokens = this.firstPositive( + endpoint != null ? endpoint.getMaxCompletionTokens() : null, + model != null && model.getTopProvider() != null ? model.getTopProvider().getMaxCompletionTokens() : null, + model != null && model.getPerRequestLimits() != null ? model.getPerRequestLimits().getCompletionTokens() : null); + return new ResolvedOpenRouterModelMetadata(modelName, model, endpoints, pricing, supportedParameters, contextLength, maxCompletionTokens); + } + + private OpenRouterModelsResponse.OpenRouterModel findModel(final ModelProfile modelProfile, final String modelName, final boolean forceReload) { + final var model = this.findMatchingModel(this.listModels(modelProfile, new OpenRouterModelQuery(true, null, List.of(), List.of()), forceReload), modelName); + if (model != null) { + return model; + } + return this.findMatchingModel(this.listModels(modelProfile, new OpenRouterModelQuery(false, null, List.of(), List.of()), forceReload), modelName); + } + + private OpenRouterModelsResponse.OpenRouterModel findMatchingModel(final List models, final String modelName) { + if (modelName == null || modelName.isBlank() || models == null || models.isEmpty()) { + return null; + } + return models.stream() + .filter(Objects::nonNull) + .filter(model -> modelName.equalsIgnoreCase(model.getId()) || modelName.equalsIgnoreCase(model.getCanonicalSlug())) + .findFirst() + .orElse(null); + } + + private OpenRouterModelEndpointsResponse.OpenRouterEndpoint preferredEndpoint(final OpenRouterModelEndpointsResponse.OpenRouterModelEndpoints endpoints) { + if (endpoints == null || endpoints.getEndpoints() == null || endpoints.getEndpoints().isEmpty()) { + return null; + } + return endpoints.getEndpoints().stream() + .filter(Objects::nonNull) + .filter(endpoint -> endpoint.getStatus() != null && endpoint.getStatus().equalsIgnoreCase("default")) + .findFirst() + .orElse(endpoints.getEndpoints().getFirst()); + } + + private String resolveDetailsPath(final OpenRouterModelsResponse.OpenRouterModel model, final String modelName) { + if (model != null && model.getLinks() != null && model.getLinks().getDetails() != null && !model.getLinks().getDetails().isBlank()) { + return model.getLinks().getDetails(); + } + final var separator = modelName.indexOf('/'); + if (separator < 0 || separator == modelName.length() - 1) { + return "/v1/models"; + } + return "/v1/models/" + this.encode(modelName.substring(0, separator)) + "/" + this.encode(modelName.substring(separator + 1)) + "/endpoints"; + } + + private String encode(final String value) { + return URLEncoder.encode(value, StandardCharsets.UTF_8).replace("+", "%20"); + } + + private int firstPositive(final Integer... values) { + for (final var value : values) { + if (value != null && value > 0) { + return value; + } + } + return 0; + } + + private boolean hasPricing(final OpenRouterModelsResponse.OpenRouterPricing pricing) { + return pricing != null + && ((pricing.getPrompt() != null && !pricing.getPrompt().isBlank()) + || (pricing.getCompletion() != null && !pricing.getCompletion().isBlank()) + || (pricing.getInternalReasoning() != null && !pricing.getInternalReasoning().isBlank()) + || (pricing.getInputCacheRead() != null && !pricing.getInputCacheRead().isBlank()) + || (pricing.getInputCacheWrite() != null && !pricing.getInputCacheWrite().isBlank())); + } + + private String join(final List values) { + if (values == null || values.isEmpty()) { + return ""; + } + return values.stream() + .filter(Objects::nonNull) + .map(String::trim) + .filter(value -> !value.isBlank()) + .collect(Collectors.joining(",")); + } + + private String modelListCacheKey(final ModelProfile modelProfile, final OpenRouterModelQuery query) { + return this.baseKey(modelProfile) + "|user=" + query.userFiltered() + "|category=" + Objects.toString(query.category(), "") + "|supported=" + this.join(query.supportedParameters()) + "|modalities=" + this.join(query.outputModalities()); + } + + private String endpointCacheKey(final ModelProfile modelProfile, final String modelName) { + return this.baseKey(modelProfile) + "|endpoints|" + modelName; + } + + private String metadataCacheKey(final ModelProfile modelProfile, final String modelName) { + return this.baseKey(modelProfile) + "|metadata|" + modelName; + } + + private String baseKey(final ModelProfile modelProfile) { + return modelProfile.category() + "|" + OPENROUTER_API_URL + "|" + modelProfile.apiKey(); + } + + private void validateProfile(final ModelProfile modelProfile) { + if (modelProfile == null || modelProfile.modelProvider() != ModelProvider.OPENROUTER) { + throw new IllegalArgumentException("Invalid OpenRouter ModelProfile"); + } + } + + static BigDecimal parsePricing(final String value) { + if (value == null || value.isBlank()) { + return null; + } + try { + return new BigDecimal(value).divide(new BigDecimal(1000000.0)); + } + catch (final NumberFormatException e) { + return null; + } + } + + public record ResolvedOpenRouterModelMetadata( + String modelId, + OpenRouterModelsResponse.OpenRouterModel model, + OpenRouterModelEndpointsResponse.OpenRouterModelEndpoints endpoints, + OpenRouterModelsResponse.OpenRouterPricing pricing, + List supportedParameters, + int contextLength, + int maxCompletionTokens) { + + public ResolvedOpenRouterModelMetadata { + supportedParameters = supportedParameters == null ? List.of() : List.copyOf(supportedParameters); + } + + public boolean supportsParameter(final String parameter) { + if (parameter == null || parameter.isBlank()) { + return false; + } + return this.supportedParameters.stream() + .filter(Objects::nonNull) + .map(value -> value.toLowerCase(Locale.ROOT).trim()) + .anyMatch(value -> value.equals(parameter.toLowerCase(Locale.ROOT))); + } + + public boolean supportsToolCalling() { + return this.supportsParameter("tools"); + } + + public boolean supportsReasoning() { + return this.supportsParameter("reasoning") || this.supportsParameter("reasoning_effort") || this.supportsParameter("include_reasoning"); + } + + public OpenAiProviderDialect.ChatTokenLimitParameterMode tokenLimitParameterMode() { + if (this.supportsParameter("max_completion_tokens")) { + return OpenAiProviderDialect.ChatTokenLimitParameterMode.MAX_COMPLETION_TOKENS; + } + if (this.supportsParameter("max_tokens")) { + return OpenAiProviderDialect.ChatTokenLimitParameterMode.MAX_TOKENS; + } + return OpenAiProviderDialect.ChatTokenLimitParameterMode.MAX_COMPLETION_TOKENS; + } + + public BigDecimal promptCost() { + return OpenRouterModelDiscovery.parsePricing(this.pricing != null ? this.pricing.getPrompt() : null); + } + + public BigDecimal completionCost() { + return OpenRouterModelDiscovery.parsePricing(this.pricing != null ? this.pricing.getCompletion() : null); + } + + public BigDecimal cachedPromptCost() { + if (this.pricing == null) { + return null; + } + final var inputCacheReadCost = OpenRouterModelDiscovery.parsePricing(this.pricing.getInputCacheRead()); + return inputCacheReadCost != null ? inputCacheReadCost : OpenRouterModelDiscovery.parsePricing(this.pricing.getInputCacheWrite()); + } + + public BigDecimal reasoningCost() { + return OpenRouterModelDiscovery.parsePricing(this.pricing != null ? this.pricing.getInternalReasoning() : null); + } + + public BigDecimal requestCost() { + return OpenRouterModelDiscovery.parsePricing(this.pricing != null ? this.pricing.getRequest() : null); + } + } + +} \ No newline at end of file diff --git a/src/main/java/de/extio/lmlib/client/openrouter/OpenRouterModelEndpointsResponse.java b/src/main/java/de/extio/lmlib/client/openrouter/OpenRouterModelEndpointsResponse.java new file mode 100644 index 0000000..999f837 --- /dev/null +++ b/src/main/java/de/extio/lmlib/client/openrouter/OpenRouterModelEndpointsResponse.java @@ -0,0 +1,224 @@ +package de.extio.lmlib.client.openrouter; + +import java.util.List; + +import com.fasterxml.jackson.annotation.JsonProperty; + +final class OpenRouterModelEndpointsResponse { + + private OpenRouterModelEndpoints data; + + @JsonProperty("data") + public OpenRouterModelEndpoints getData() { + return this.data; + } + + public void setData(final OpenRouterModelEndpoints data) { + this.data = data; + } + + static final class OpenRouterModelEndpoints { + + private String id; + + private String name; + + private Integer created; + + private String description; + + private OpenRouterModelsResponse.OpenRouterModelArchitecture architecture; + + private List endpoints; + + @JsonProperty("id") + public String getId() { + return this.id; + } + + public void setId(final String id) { + this.id = id; + } + + @JsonProperty("name") + public String getName() { + return this.name; + } + + public void setName(final String name) { + this.name = name; + } + + @JsonProperty("created") + public Integer getCreated() { + return this.created; + } + + public void setCreated(final Integer created) { + this.created = created; + } + + @JsonProperty("description") + public String getDescription() { + return this.description; + } + + public void setDescription(final String description) { + this.description = description; + } + + @JsonProperty("architecture") + public OpenRouterModelsResponse.OpenRouterModelArchitecture getArchitecture() { + return this.architecture; + } + + public void setArchitecture(final OpenRouterModelsResponse.OpenRouterModelArchitecture architecture) { + this.architecture = architecture; + } + + @JsonProperty("endpoints") + public List getEndpoints() { + return this.endpoints; + } + + public void setEndpoints(final List endpoints) { + this.endpoints = endpoints; + } + } + + static final class OpenRouterEndpoint { + + private String name; + + private String modelId; + + private String modelName; + + private String providerName; + + private String status; + + private String tag; + + private Integer contextLength; + + private Integer maxCompletionTokens; + + private Integer maxPromptTokens; + + private OpenRouterModelsResponse.OpenRouterPricing pricing; + + private List supportedParameters; + + private boolean supportsImplicitCaching; + + @JsonProperty("name") + public String getName() { + return this.name; + } + + public void setName(final String name) { + this.name = name; + } + + @JsonProperty("model_id") + public String getModelId() { + return this.modelId; + } + + public void setModelId(final String modelId) { + this.modelId = modelId; + } + + @JsonProperty("model_name") + public String getModelName() { + return this.modelName; + } + + public void setModelName(final String modelName) { + this.modelName = modelName; + } + + @JsonProperty("provider_name") + public String getProviderName() { + return this.providerName; + } + + public void setProviderName(final String providerName) { + this.providerName = providerName; + } + + @JsonProperty("status") + public String getStatus() { + return this.status; + } + + public void setStatus(final String status) { + this.status = status; + } + + @JsonProperty("tag") + public String getTag() { + return this.tag; + } + + public void setTag(final String tag) { + this.tag = tag; + } + + @JsonProperty("context_length") + public Integer getContextLength() { + return this.contextLength; + } + + public void setContextLength(final Integer contextLength) { + this.contextLength = contextLength; + } + + @JsonProperty("max_completion_tokens") + public Integer getMaxCompletionTokens() { + return this.maxCompletionTokens; + } + + public void setMaxCompletionTokens(final Integer maxCompletionTokens) { + this.maxCompletionTokens = maxCompletionTokens; + } + + @JsonProperty("max_prompt_tokens") + public Integer getMaxPromptTokens() { + return this.maxPromptTokens; + } + + public void setMaxPromptTokens(final Integer maxPromptTokens) { + this.maxPromptTokens = maxPromptTokens; + } + + @JsonProperty("pricing") + public OpenRouterModelsResponse.OpenRouterPricing getPricing() { + return this.pricing; + } + + public void setPricing(final OpenRouterModelsResponse.OpenRouterPricing pricing) { + this.pricing = pricing; + } + + @JsonProperty("supported_parameters") + public List getSupportedParameters() { + return this.supportedParameters; + } + + public void setSupportedParameters(final List supportedParameters) { + this.supportedParameters = supportedParameters; + } + + @JsonProperty("supports_implicit_caching") + public boolean isSupportsImplicitCaching() { + return this.supportsImplicitCaching; + } + + public void setSupportsImplicitCaching(final boolean supportsImplicitCaching) { + this.supportsImplicitCaching = supportsImplicitCaching; + } + } + +} \ No newline at end of file diff --git a/src/main/java/de/extio/lmlib/client/openrouter/OpenRouterModelQuery.java b/src/main/java/de/extio/lmlib/client/openrouter/OpenRouterModelQuery.java new file mode 100644 index 0000000..13d265e --- /dev/null +++ b/src/main/java/de/extio/lmlib/client/openrouter/OpenRouterModelQuery.java @@ -0,0 +1,16 @@ +package de.extio.lmlib.client.openrouter; + +import java.util.List; + +public record OpenRouterModelQuery(boolean userFiltered, String category, List supportedParameters, List outputModalities) { + + public OpenRouterModelQuery { + supportedParameters = supportedParameters == null ? List.of() : List.copyOf(supportedParameters); + outputModalities = outputModalities == null ? List.of() : List.copyOf(outputModalities); + } + + public boolean hasServerSideFilters() { + return this.category != null && !this.category.isBlank() || !this.supportedParameters.isEmpty() || !this.outputModalities.isEmpty(); + } + +} \ No newline at end of file diff --git a/src/main/java/de/extio/lmlib/client/openrouter/OpenRouterModelsResponse.java b/src/main/java/de/extio/lmlib/client/openrouter/OpenRouterModelsResponse.java new file mode 100644 index 0000000..2b8eb1f --- /dev/null +++ b/src/main/java/de/extio/lmlib/client/openrouter/OpenRouterModelsResponse.java @@ -0,0 +1,428 @@ +package de.extio.lmlib.client.openrouter; + +import java.util.List; + +import com.fasterxml.jackson.annotation.JsonProperty; + +final class OpenRouterModelsResponse { + + private List data; + + @JsonProperty("data") + public List getData() { + return this.data; + } + + public void setData(final List data) { + this.data = data; + } + + static final class OpenRouterModel { + + private String id; + + private String canonicalSlug; + + private String name; + + private Integer created; + + private Integer contextLength; + + private OpenRouterModelArchitecture architecture; + + private OpenRouterPricing pricing; + + private List supportedParameters; + + private OpenRouterTopProviderInfo topProvider; + + private OpenRouterPerRequestLimits perRequestLimits; + + private OpenRouterModelLinks links; + + private String description; + + private String knowledgeCutoff; + + private String expirationDate; + + @JsonProperty("id") + public String getId() { + return this.id; + } + + public void setId(final String id) { + this.id = id; + } + + @JsonProperty("canonical_slug") + public String getCanonicalSlug() { + return this.canonicalSlug; + } + + public void setCanonicalSlug(final String canonicalSlug) { + this.canonicalSlug = canonicalSlug; + } + + @JsonProperty("name") + public String getName() { + return this.name; + } + + public void setName(final String name) { + this.name = name; + } + + @JsonProperty("created") + public Integer getCreated() { + return this.created; + } + + public void setCreated(final Integer created) { + this.created = created; + } + + @JsonProperty("context_length") + public Integer getContextLength() { + return this.contextLength; + } + + public void setContextLength(final Integer contextLength) { + this.contextLength = contextLength; + } + + @JsonProperty("architecture") + public OpenRouterModelArchitecture getArchitecture() { + return this.architecture; + } + + public void setArchitecture(final OpenRouterModelArchitecture architecture) { + this.architecture = architecture; + } + + @JsonProperty("pricing") + public OpenRouterPricing getPricing() { + return this.pricing; + } + + public void setPricing(final OpenRouterPricing pricing) { + this.pricing = pricing; + } + + @JsonProperty("supported_parameters") + public List getSupportedParameters() { + return this.supportedParameters; + } + + public void setSupportedParameters(final List supportedParameters) { + this.supportedParameters = supportedParameters; + } + + @JsonProperty("top_provider") + public OpenRouterTopProviderInfo getTopProvider() { + return this.topProvider; + } + + public void setTopProvider(final OpenRouterTopProviderInfo topProvider) { + this.topProvider = topProvider; + } + + @JsonProperty("per_request_limits") + public OpenRouterPerRequestLimits getPerRequestLimits() { + return this.perRequestLimits; + } + + public void setPerRequestLimits(final OpenRouterPerRequestLimits perRequestLimits) { + this.perRequestLimits = perRequestLimits; + } + + @JsonProperty("links") + public OpenRouterModelLinks getLinks() { + return this.links; + } + + public void setLinks(final OpenRouterModelLinks links) { + this.links = links; + } + + @JsonProperty("description") + public String getDescription() { + return this.description; + } + + public void setDescription(final String description) { + this.description = description; + } + + @JsonProperty("knowledge_cutoff") + public String getKnowledgeCutoff() { + return this.knowledgeCutoff; + } + + public void setKnowledgeCutoff(final String knowledgeCutoff) { + this.knowledgeCutoff = knowledgeCutoff; + } + + @JsonProperty("expiration_date") + public String getExpirationDate() { + return this.expirationDate; + } + + public void setExpirationDate(final String expirationDate) { + this.expirationDate = expirationDate; + } + + @Override + public String toString() { + return "OpenRouterModel [id=" + id + ", canonicalSlug=" + canonicalSlug + ", name=" + name + ", created=" + created + ", contextLength=" + contextLength + ", architecture=" + architecture + ", pricing=" + pricing + ", supportedParameters=" + supportedParameters + ", topProvider=" + topProvider + ", perRequestLimits=" + perRequestLimits + ", links=" + links + ", description=" + description + ", knowledgeCutoff=" + knowledgeCutoff + ", expirationDate=" + expirationDate + "]"; + } + + } + + static final class OpenRouterModelArchitecture { + + private String tokenizer; + + private String instructType; + + private String modality; + + private List inputModalities; + + private List outputModalities; + + @JsonProperty("tokenizer") + public String getTokenizer() { + return this.tokenizer; + } + + public void setTokenizer(final String tokenizer) { + this.tokenizer = tokenizer; + } + + @JsonProperty("instruct_type") + public String getInstructType() { + return this.instructType; + } + + public void setInstructType(final String instructType) { + this.instructType = instructType; + } + + @JsonProperty("modality") + public String getModality() { + return this.modality; + } + + public void setModality(final String modality) { + this.modality = modality; + } + + @JsonProperty("input_modalities") + public List getInputModalities() { + return this.inputModalities; + } + + public void setInputModalities(final List inputModalities) { + this.inputModalities = inputModalities; + } + + @JsonProperty("output_modalities") + public List getOutputModalities() { + return this.outputModalities; + } + + public void setOutputModalities(final List outputModalities) { + this.outputModalities = outputModalities; + } + + @Override + public String toString() { + return "OpenRouterModelArchitecture [tokenizer=" + tokenizer + ", instructType=" + instructType + ", modality=" + modality + ", inputModalities=" + inputModalities + ", outputModalities=" + outputModalities + "]"; + } + } + + static final class OpenRouterPricing { + + private String prompt; + + private String completion; + + private String internalReasoning; + + private String inputCacheRead; + + private String inputCacheWrite; + + private String request; + + private String image; + + @JsonProperty("prompt") + public String getPrompt() { + return this.prompt; + } + + public void setPrompt(final String prompt) { + this.prompt = prompt; + } + + @JsonProperty("completion") + public String getCompletion() { + return this.completion; + } + + public void setCompletion(final String completion) { + this.completion = completion; + } + + @JsonProperty("internal_reasoning") + public String getInternalReasoning() { + return this.internalReasoning; + } + + public void setInternalReasoning(final String internalReasoning) { + this.internalReasoning = internalReasoning; + } + + @JsonProperty("input_cache_read") + public String getInputCacheRead() { + return this.inputCacheRead; + } + + public void setInputCacheRead(final String inputCacheRead) { + this.inputCacheRead = inputCacheRead; + } + + @JsonProperty("input_cache_write") + public String getInputCacheWrite() { + return this.inputCacheWrite; + } + + public void setInputCacheWrite(final String inputCacheWrite) { + this.inputCacheWrite = inputCacheWrite; + } + + @JsonProperty("request") + public String getRequest() { + return this.request; + } + + public void setRequest(final String request) { + this.request = request; + } + + @JsonProperty("image") + public String getImage() { + return this.image; + } + + public void setImage(final String image) { + this.image = image; + } + + @Override + public String toString() { + return "OpenRouterPricing [prompt=" + prompt + ", completion=" + completion + ", internalReasoning=" + internalReasoning + ", inputCacheRead=" + inputCacheRead + ", inputCacheWrite=" + inputCacheWrite + ", request=" + request + ", image=" + image + "]"; + } + } + + static final class OpenRouterTopProviderInfo { + + private Integer contextLength; + + private Integer maxCompletionTokens; + + private boolean moderated; + + @JsonProperty("context_length") + public Integer getContextLength() { + return this.contextLength; + } + + public void setContextLength(final Integer contextLength) { + this.contextLength = contextLength; + } + + @JsonProperty("max_completion_tokens") + public Integer getMaxCompletionTokens() { + return this.maxCompletionTokens; + } + + public void setMaxCompletionTokens(final Integer maxCompletionTokens) { + this.maxCompletionTokens = maxCompletionTokens; + } + + @JsonProperty("is_moderated") + public boolean isModerated() { + return this.moderated; + } + + public void setModerated(final boolean moderated) { + this.moderated = moderated; + } + + @Override + public String toString() { + return "OpenRouterTopProviderInfo [contextLength=" + contextLength + ", maxCompletionTokens=" + maxCompletionTokens + ", moderated=" + moderated + "]"; + } + } + + static final class OpenRouterPerRequestLimits { + + private Integer promptTokens; + + private Integer completionTokens; + + @JsonProperty("prompt_tokens") + public Integer getPromptTokens() { + return this.promptTokens; + } + + public void setPromptTokens(final Integer promptTokens) { + this.promptTokens = promptTokens; + } + + @JsonProperty("completion_tokens") + public Integer getCompletionTokens() { + return this.completionTokens; + } + + public void setCompletionTokens(final Integer completionTokens) { + this.completionTokens = completionTokens; + } + + @Override + public String toString() { + return "OpenRouterPerRequestLimits [promptTokens=" + promptTokens + ", completionTokens=" + completionTokens + "]"; + } + + } + + static final class OpenRouterModelLinks { + + private String details; + + @JsonProperty("details") + public String getDetails() { + return this.details; + } + + public void setDetails(final String details) { + this.details = details; + } + + @Override + public String toString() { + return "OpenRouterModelLinks [details=" + details + "]"; + } + + } + + @Override + public String toString() { + return "OpenRouterModelsResponse [data=" + data + "]"; + } + +} diff --git a/src/main/java/de/extio/lmlib/profile/ModelProfile.java b/src/main/java/de/extio/lmlib/profile/ModelProfile.java index eb54d59..27cee39 100755 --- a/src/main/java/de/extio/lmlib/profile/ModelProfile.java +++ b/src/main/java/de/extio/lmlib/profile/ModelProfile.java @@ -21,11 +21,11 @@ public record ModelProfile( String category) { public ModelProfile { - if (maxTokens <= 0) { - throw new IllegalArgumentException("Model profile maxTokens must be greater than zero"); + if (maxTokens < 0) { + throw new IllegalArgumentException("Model profile maxTokens must be greater than or equal to zero"); } - if (maxContextLength <= 0) { - throw new IllegalArgumentException("Model profile maxContextLength must be greater than zero"); + if (maxContextLength < 0) { + throw new IllegalArgumentException("Model profile maxContextLength must be greater than or equal to zero"); } if (temperature <= 0) { throw new IllegalArgumentException("Model profile temperature must be greater than zero"); @@ -36,15 +36,12 @@ public record ModelProfile( if (modelProvider == null) { throw new IllegalArgumentException("Model profile modelProvider is mandatory and cannot be null"); } - if (url == null || url.isBlank()) { - throw new IllegalArgumentException("Model profile url is mandatory and cannot be empty"); - } if (category == null || category.isBlank()) { throw new IllegalArgumentException("Model profile category is mandatory and cannot be empty"); } } public static enum ModelProvider { - OAI_TEXT_COMPLETION, OAI_CHAT_COMPLETION, OLLAMA; + OAI_TEXT_COMPLETION, OAI_CHAT_COMPLETION, OPENROUTER, OLLAMA; } } diff --git a/src/main/java/de/extio/lmlib/profile/ModelProfileService.java b/src/main/java/de/extio/lmlib/profile/ModelProfileService.java index d8027fa..8b9e9b1 100755 --- a/src/main/java/de/extio/lmlib/profile/ModelProfileService.java +++ b/src/main/java/de/extio/lmlib/profile/ModelProfileService.java @@ -49,11 +49,11 @@ public class ModelProfileService { final var maxTokens = Optional.ofNullable(resource.getProperty("maxTokens")) .map(Object::toString) .map(Integer::parseInt) - .orElseThrow(() -> new IllegalStateException("No model maxTokens found for model: " + modelName)); + .orElse(Integer.valueOf(0)); final var maxContextLength = Optional.ofNullable(resource.getProperty("maxContextLength")) .map(Object::toString) .map(Integer::parseInt) - .orElseThrow(() -> new IllegalStateException("No model maxContextLength found for model: " + modelName)); + .orElse(Integer.valueOf(0)); final var temperature = Optional.ofNullable(resource.getProperty("temperature")) .map(Object::toString) .map(Double::parseDouble) @@ -71,7 +71,7 @@ public class ModelProfileService { .orElse(null); final var url = Optional.ofNullable(resource.getProperty("url")) .map(Object::toString) - .orElseThrow(() -> new IllegalStateException("No model url found for model: " + modelName)); + .orElse(null); final var costPerInToken = Optional.ofNullable(resource.getProperty("cost1MInTokens")) .map(Object::toString) .map(value -> new BigDecimal(value)) diff --git a/src/test/java/de/extio/lmlib/client/openrouter/OpenRouterClientTest.java b/src/test/java/de/extio/lmlib/client/openrouter/OpenRouterClientTest.java new file mode 100644 index 0000000..7bceb57 --- /dev/null +++ b/src/test/java/de/extio/lmlib/client/openrouter/OpenRouterClientTest.java @@ -0,0 +1,117 @@ +package de.extio.lmlib.client.openrouter; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.util.List; + +import org.junit.jupiter.api.Disabled; +import org.junit.jupiter.api.Test; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.boot.SpringBootConfiguration; +import org.springframework.boot.autoconfigure.EnableAutoConfiguration; +import org.springframework.boot.test.context.SpringBootTest; +import org.springframework.boot.test.context.SpringBootTest.WebEnvironment; +import org.springframework.test.context.DynamicPropertyRegistry; +import org.springframework.test.context.DynamicPropertySource; +import org.springframework.test.context.TestPropertySource; + +import de.extio.lmlib.client.ClientService; +import de.extio.lmlib.client.Conversation; +import de.extio.lmlib.grader.Grader2; +import de.extio.lmlib.profile.ModelProfile; +import de.extio.lmlib.profile.ModelProfileService; + +@Disabled("This test requires a valid OpenRouter API key") +@SpringBootTest(webEnvironment = WebEnvironment.NONE) +@SpringBootConfiguration +@EnableAutoConfiguration +@TestPropertySource("classpath:/application-test.properties") +public class OpenRouterClientTest { + + private final static String API_KEY = ""; // Enter your OpenRouter API key here for testing + + private static final Logger LOGGER = LoggerFactory.getLogger(OpenRouterClientTest.class); + + @DynamicPropertySource + static void registerProperties(final DynamicPropertyRegistry registry) { + registry.add("OPENROUTER_API_KEY", () -> API_KEY); + } + + @Autowired + private OpenRouterClient openRouterClient; + + @Autowired + private OpenRouterModelDiscovery openRouterModelDiscovery; + + @Autowired + private ModelProfileService modelProfileService; + + @Autowired + private ClientService clientService; + + @Test + void modelDiscovery() throws Exception { + final var modelProfile = this.modelProfile(); + final var models = this.openRouterModelDiscovery.listModels(modelProfile, new OpenRouterModelQuery(true, null, List.of(), List.of()), true); + LOGGER.info("Models: {}", models); + assertFalse(models.isEmpty()); + assertFalse(models.getFirst().getId().isBlank()); + final var resolvedModelName = models.getFirst().getId(); + LOGGER.info("Resolved model name: {}", resolvedModelName); + final var metadata = this.openRouterModelDiscovery.resolveModelMetadata(modelProfile, resolvedModelName, true); + LOGGER.info("Metadata: {}", metadata); + assertEquals(resolvedModelName, metadata.modelId()); + assertTrue(metadata.contextLength() > 0); + assertEquals(ModelProfile.ModelProvider.OPENROUTER, this.clientService.getClient(modelProfile).getModelProvider()); + } + + @Test + void chatCompletion() throws Exception { + final var modelProfile = this.modelProfile(); + final var completion = this.openRouterClient.conversation( + modelProfile, + Conversation.create("You are a helpful assistant", "Say the color is green"), + null, + false); + + LOGGER.info(completion.reasoning()); + LOGGER.info(completion.response()); + + assertTrue(Grader2.assessScoreBinary("Does the text mention the color green?", completion.response(), modelProfile, this.clientService)); + } + + @Test + void streamChatCompletion() throws Exception { + final var modelProfile = this.modelProfile(); + final var conversation = Conversation.create("You are a helpful assistant", "I would like to play a game of chess."); + conversation.addTurn(new Conversation.Turn(Conversation.TurnType.ASSISTANT, "Which opening do you choose?")); + conversation.addTurn(new Conversation.Turn(Conversation.TurnType.USER, "You choose, but explain the opening to me.")); + final var responseBuilder = new StringBuilder(); + final var reasoningBuilder = new StringBuilder(); + final var completion = this.openRouterClient.streamConversation(modelProfile, conversation, chunk -> { + if (chunk.reasoningContent() != null) { + System.out.print(chunk.reasoningContent()); + reasoningBuilder.append(chunk.reasoningContent()); + } + if (chunk.content() != null) { + System.out.print(chunk.content()); + responseBuilder.append(chunk.content()); + } + }, null, false); + + assertEquals(completion.response(), responseBuilder.toString()); + if (completion.reasoning() != null) { + assertEquals(completion.reasoning(), reasoningBuilder.toString()); + } + assertTrue(Grader2.assessScoreBinary("Does the text explain a chess opening?", completion.response(), modelProfile, this.clientService)); + } + + private ModelProfile modelProfile() { + return this.modelProfileService.getModelProfile("profile.model.openrouter"); + } + +} \ No newline at end of file diff --git a/src/test/resources/application-test.properties b/src/test/resources/application-test.properties index 65307de..f14a582 100755 --- a/src/test/resources/application-test.properties +++ b/src/test/resources/application-test.properties @@ -7,6 +7,7 @@ profile.model.cold=local-chatcompletion profile.model.textcompletion=local-completion profile.model.chatcompletion=local-chatcompletion profile.model.ollama=local-ollama-chatcompletion +profile.model.openrouter=local-openrouter-chatcompletion #profile.model.azure=llama3.3-70b-azure agent.threads=2 diff --git a/src/test/resources/local-openrouter-chatcompletion.properties b/src/test/resources/local-openrouter-chatcompletion.properties new file mode 100644 index 0000000..b7516ed --- /dev/null +++ b/src/test/resources/local-openrouter-chatcompletion.properties @@ -0,0 +1,16 @@ +category=OPENROUTER +prompts= +tokenizer=fallback +tokenEncoding=cl100k_base +maxTokens=2500 +maxContextLength=32768 +temperature=0.5 +topP=0.95 +modelProvider=OPENROUTER +modelName= +url= +apiKey=${OPENROUTER_API_KEY:} +cost1MInTokens=0 +cost1MCachedInTokens=0 +cost1MOutTokens=0 +cost1MReasoningOutTokens=0 \ No newline at end of file