diff --git a/README.md b/README.md
index ee8cb42..46dbcb5 100644
--- a/README.md
+++ b/README.md
@@ -1,21 +1,151 @@
-This suite of Java packages provides a comprehensive framework for building and deploying conversational agents powered by language models.
+# Purpose
-**Purpose:**
+The purpose of this application is to provide a comprehensive conversational AI system that enables efficient and effective human-computer interactions through natural language processing, conversation management, and text completion capabilities, utilizing caching, tokenization, and integration with external AI models and services to deliver a robust and scalable solution for applications such as chatbots, virtual assistants, and customer service platforms.
-The primary purpose is to simplify the development and management of AI-driven chatbot applications.
+# Program Description
-**Functionality:**
+The program is a comprehensive conversational AI system that provides natural language processing functionality, conversation management, and text completion capabilities. The purpose of the program is to enable efficient and effective human-computer interactions through AI-powered conversations.
-* **Agent Framework:** The core package enables the creation of agents capable of interacting with language models, processing text, maintaining conversation context, tracking statistics, and executing in a structured manner. It supports chaining agents, handling responses, and flexible configuration.
-* **Language Model Interaction:** Several packages provide tools for interacting with various language models, including Azure AI's text generation service and remote text completion models. These tools handle tasks like retrieving models, managing conversations, generating completions, and tracking API usage.
-* **Prompt Engineering:** A dedicated package focuses on constructing and manipulating prompts tailored to specific language models. It supports different formatting strategies and allows for fine-grained control over prompt structure.
-* **Tokenization:** Another package handles text tokenization and detokenization, crucial for preparing text for input to language models and interpreting their outputs.
+The program consists of several packages that work together to achieve this goal. These packages include:
-**Overall, this collection of packages offers a powerful and flexible toolkit for developers looking to build sophisticated conversational agents leveraging the capabilities of modern language models.**
+1. **Caching layer (com.transporeon.exchange.lmlib.client.cached)**: Improves performance by reducing requests to the underlying client through caching and retrieving completion responses.
+2. **Natural Language Processing (com.transporeon.exchange.lmlib.client)**: Provides conversation management and text completion functionality, generating completions, processing conversations, and managing models.
+3. **Azure AI Integration (com.transporeon.exchange.lmlib.client.azureai)**: Integrates with Azure AI services, managing model profiles and handling conversation completions.
+4. **Text Completion (com.transporeon.exchange.lmlib.client.textcompletion)**: Provides text completion functionality using AI models, including model selection, prompt generation, and response handling.
+5. **Text Tokenization (com.transporeon.exchange.lmlib.token)**: Tokenizes text into numerical tokens and detokenizes numerical tokens back into human-readable text.
+6. **Conversation Prompts (com.transporeon.exchange.lmlib.prompt)**: Manages conversation prompts, creating new prompts, continuing conversations, and removing end-of-text markers.
+7. **Agent Executor (com.transporeon.exchange.lmlib.agent)**: Executes agent graphs concurrently, handles agent responses, and tracks request statistics.
-*gemma2-27b requests=50, requestDuration=PT3M36.921314511S, inTokens=26737, outTokens=2207, tps=10.217592592592593*
+The program's key features include:
-*[Below announcement has been generated by the unit test AgentTest#agenticFlow:]*
+* Agent graph execution
+* Conversation management and text completion
+* Integration with external AI models and services (e.g., Azure AI)
+* Support for multiple prompt strategies and tokenization implementations
+* Caching and performance optimization
+
+Overall, the program is designed to provide a robust and efficient conversational AI system that can be used in various applications, such as chatbots, virtual assistants, and customer service platforms.
+
+*AgentRequestStatistic [requests=43, cachedPrompts=2, inTokens=23437, outTokens=7403, requestDuration=PT2M31.7349284S, getEffectiveDuration()=PT1M37.2400495S, getTps()=317.15, getOutTps()=76.13, getCost()=0.01458674]*
+
+# Setup
+
+To keep this library lean and compatible with different dependency trees, all dependencies must be provided by the application.
+
+## Mandatory dependencies
+
+
+ org.springframework.boot
+ spring-boot-starter-webflux
+
+
+## Optional dependencies
+
+Some services benefit from Spring cache framework, e.g. the model profile service:
+
+
+ org.springframework.boot
+ spring-boot-starter-cache
+
+
+Don't forget to activate a CacheManager afterwards:
+
+ @Bean
+ public CacheManager cacheManager() {
+ return new CaffeineCacheManager();
+ }
+
+## Model profiles
+
+Models are managed with profiles. You can maintain multiple profiles for different models and parameters such as temperature.
+
+These profiles must be mapped to fixed categories with the following application properties:
+
+ profile.model.s=
+ profile.model.m=
+ profile.model.l=
+ profile.model.hot=
+ profile.model.cold=
+
+The value is the name of the model profile .properties file, for example "llama3.3-70b-azure".
+
+Profiles must be located in the main resources location.
+
+Examples can be found in test resources.
+
+## OpenAi compatible text completion client
+
+Use tokenizer strategy jtokkit or llamaServer (if the model is hosted on a llama.cpp server)
+
+Property:
+
+ tokenizer.strategy=llamaServer
+
+If you use jtokkit tokenizer:
+
+
+ com.knuddels
+ jtokkit
+
+
+It is required to select the right prompt template. See package com.transporeon.exchange.lmlib.prompt for available prompt templates or implement your own PromptStrategy (must be a bean in the spring context).
+
+Set the following property in your model profile:
+
+ prompts=llama3
+
+Example profile: src/test/resources/llama3-local.properties
+
+## Azure AI client
+
+Use tokenizer strategy jtokkit.
+
+Property:
+
+ tokenizer.strategy=jtokkit
+
+Example profile: src/test/resources/llama3.3-70b-azure.properties
+
+
+ com.azure
+ azure-ai-inference
+
+
+ com.squareup.okhttp3
+ okhttp
+
+
+ com.knuddels
+ jtokkit
+
+
+ com.github.ben-manes.caffeine
+ caffeine
+
+
+# Completions Cache
+
+All language model requests will be cached if you add a CachedClientRepository bean to your context.
+
+# Agentic Flows
+
+This library contains a graph executor for implementing comprehensive processes using agent-based design patterns. See class AgentTest for examples.
+
+# Usage
+
+1. Setup dependencies, see above
+2. Configure model profiles, see above
+3. Implement a CachedClientRepository if you need your requests cached
+4. For chats or completions, use ClientService service bean
+5. For agentic flows, create an AgentContext and use AgentExecutorService service bean
+
+# Examples
+
+Text completions: AzureAiClientTest#completion
+Chat: AzureAiClientTest#chat
+Agentic flow: AgentTest#agenticFlow
+
+# Announcement generated by AgentTest#agenticFlow
Title: Introducing a Powerful Framework for Building Intelligent Agents
diff --git a/pom.xml b/pom.xml
index 0b08f55..1368031 100755
--- a/pom.xml
+++ b/pom.xml
@@ -4,13 +4,13 @@
lm-lib
de.extio
lm-lib
- 1.0.0
+ 1.1.0
21
21
21
- 3.4.1
+ 3.4.2
1.0.0-beta.2
1.1.0
diff --git a/src/main/java/de/extio/lmlib/agent/AgentExecutorService.java b/src/main/java/de/extio/lmlib/agent/AgentExecutorService.java
index 1d26333..bbd1b5b 100755
--- a/src/main/java/de/extio/lmlib/agent/AgentExecutorService.java
+++ b/src/main/java/de/extio/lmlib/agent/AgentExecutorService.java
@@ -15,6 +15,7 @@ import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Service;
+import de.extio.lmlib.client.Client;
import de.extio.lmlib.client.ClientService;
@Service
@@ -47,10 +48,14 @@ public class AgentExecutorService implements InitializingBean, DisposableBean {
public List walkGraph(final Agent agent, final AgentContext context) {
LOGGER.info("Agent: {}", agent.name());
- final var client = this.clientService.getClient(agent.modelCategory());
- final var responses = Collections.synchronizedList(new ArrayList());
+ Client client = null;
+ if (agent.agentType() != AgentType.PROCESSING_ONLY) {
+ client = this.clientService.getClient(agent.modelCategory());
+ }
final var branches = agent.execute(client, this.agentExecutorService, context);
+
+ final var responses = Collections.synchronizedList(new ArrayList());
final var branchTasks = new ArrayList>(branches.size());
for (final var branchContext : branches) {
if (!branchContext.isError() && branchContext.getNextAgent() != null) {
diff --git a/src/main/java/de/extio/lmlib/agent/AgentRequestStatistic.java b/src/main/java/de/extio/lmlib/agent/AgentRequestStatistic.java
index 076f6b5..f7485e5 100755
--- a/src/main/java/de/extio/lmlib/agent/AgentRequestStatistic.java
+++ b/src/main/java/de/extio/lmlib/agent/AgentRequestStatistic.java
@@ -1,5 +1,6 @@
package de.extio.lmlib.agent;
+import java.math.BigDecimal;
import java.text.DecimalFormat;
import java.time.Duration;
import java.time.Instant;
@@ -12,18 +13,26 @@ public final class AgentRequestStatistic {
private final AtomicInteger requests = new AtomicInteger();
+ private final AtomicInteger cachedPrompts = new AtomicInteger();
+
private final AtomicLong inTokens = new AtomicLong();
private final AtomicLong outTokens = new AtomicLong();
- private final Instant start = Instant.now();
+ private volatile Instant start = Instant.now();
private volatile Duration requestDuration = Duration.ofMillis(0L);
+ private volatile BigDecimal cost = BigDecimal.ZERO;
+
public AtomicInteger getRequests() {
return this.requests;
}
+ public AtomicInteger getCachedPrompts() {
+ return this.cachedPrompts;
+ }
+
public AtomicLong getInTokens() {
return this.inTokens;
}
@@ -41,22 +50,44 @@ public final class AgentRequestStatistic {
}
public String getTps() {
- return new DecimalFormat("#.##").format((double) (this.inTokens.get() + this.outTokens.get()) / (double)this.getEffectiveDuration().toMillis() * 1000.0);
+ return new DecimalFormat("#.##").format((double) (this.inTokens.get() + this.outTokens.get()) / (double) this.getEffectiveDuration().toMillis() * 1000.0);
}
public String getOutTps() {
- return new DecimalFormat("#.##").format((double) this.outTokens.get() / (double)this.getEffectiveDuration().toMillis() * 1000.0);
+ return new DecimalFormat("#.##").format((double) this.outTokens.get() / (double) this.getEffectiveDuration().toMillis() * 1000.0);
+ }
+
+ public BigDecimal getCost() {
+ return this.cost;
}
public void add(final Completion completion) {
- this.getRequests().addAndGet(completion.requests());
+ if (completion.cached()) {
+ this.cachedPrompts.incrementAndGet();
+ }
+ else {
+ this.getRequests().addAndGet(completion.requests());
+ }
this.getInTokens().addAndGet(completion.inTokens());
this.getOutTokens().addAndGet(completion.outTokens());
- this.addRequestDuration(completion.duration());
+ synchronized (this) {
+ this.requestDuration = this.requestDuration.plus(completion.duration());
+ this.cost = this.cost.add(completion.cost());
+ }
}
- public synchronized void addRequestDuration(final Duration duration) {
- this.requestDuration = this.requestDuration.plus(duration);
+ public void add(final AgentRequestStatistic other) {
+ this.getRequests().addAndGet(other.getRequests().get());
+ this.getCachedPrompts().addAndGet(other.getCachedPrompts().get());
+ this.getInTokens().addAndGet(other.getInTokens().get());
+ this.getOutTokens().addAndGet(other.getOutTokens().get());
+ synchronized (this) {
+ if (other.start.isBefore(this.start)) {
+ this.start = other.start;
+ }
+ this.requestDuration = this.requestDuration.plus(other.getRequestDuration());
+ this.cost = this.cost.add(other.getCost());
+ }
}
@Override
@@ -64,6 +95,8 @@ public final class AgentRequestStatistic {
final StringBuilder builder = new StringBuilder();
builder.append("AgentRequestStatistic [requests=");
builder.append(this.requests);
+ builder.append(", cachedPrompts=");
+ builder.append(this.cachedPrompts);
builder.append(", inTokens=");
builder.append(this.inTokens);
builder.append(", outTokens=");
@@ -76,6 +109,8 @@ public final class AgentRequestStatistic {
builder.append(this.getTps());
builder.append(", getOutTps()=");
builder.append(this.getOutTps());
+ builder.append(", getCost()=");
+ builder.append(this.getCost());
builder.append("]");
return builder.toString();
}
diff --git a/src/main/java/de/extio/lmlib/client/ClientService.java b/src/main/java/de/extio/lmlib/client/ClientService.java
index e42933b..b8e15e9 100755
--- a/src/main/java/de/extio/lmlib/client/ClientService.java
+++ b/src/main/java/de/extio/lmlib/client/ClientService.java
@@ -3,8 +3,11 @@ package de.extio.lmlib.client;
import java.util.List;
import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.cache.annotation.Cacheable;
import org.springframework.stereotype.Service;
+import de.extio.lmlib.client.cached.CachedClient;
+import de.extio.lmlib.client.cached.CachedClientRepository;
import de.extio.lmlib.profile.ModelCategory;
import de.extio.lmlib.profile.ModelProfileService;
@@ -17,11 +20,16 @@ public class ClientService {
@Autowired
private ModelProfileService modelProfileService;
+ @Autowired(required = false)
+ private CachedClientRepository cachedClientRepository;
+
+ @Cacheable("clients")
public Client getClient(final ModelCategory category) {
final var modelProfile = this.modelProfileService.getModelProfile(category.getModelProfile());
return this.clients.stream()
.filter(client -> client.getModelProvider() == modelProfile.modelProvider())
.findFirst()
+ .map(client -> this.cachedClientRepository == null ? client : new CachedClient(this.cachedClientRepository, this.modelProfileService, client))
.orElseThrow(() -> new IllegalStateException("No client found for model profile " + category));
}
diff --git a/src/main/java/de/extio/lmlib/client/Completion.java b/src/main/java/de/extio/lmlib/client/Completion.java
index e7c8f81..dd092fc 100755
--- a/src/main/java/de/extio/lmlib/client/Completion.java
+++ b/src/main/java/de/extio/lmlib/client/Completion.java
@@ -1,7 +1,8 @@
package de.extio.lmlib.client;
+import java.math.BigDecimal;
import java.time.Duration;
-public final record Completion(String response, int requests, Duration duration, long inTokens, long outTokens, boolean cached) {
+public final record Completion(String response, int requests, Duration duration, long inTokens, long outTokens, BigDecimal cost, boolean cached) {
}
diff --git a/src/main/java/de/extio/lmlib/client/azureai/AzureAiClient.java b/src/main/java/de/extio/lmlib/client/azureai/AzureAiClient.java
index 9810dff..fa62d07 100755
--- a/src/main/java/de/extio/lmlib/client/azureai/AzureAiClient.java
+++ b/src/main/java/de/extio/lmlib/client/azureai/AzureAiClient.java
@@ -1,5 +1,6 @@
package de.extio.lmlib.client.azureai;
+import java.math.BigDecimal;
import java.time.Duration;
import java.time.LocalDateTime;
import java.util.ArrayDeque;
@@ -108,6 +109,7 @@ public final class AzureAiClient implements Client {
statistics.duration,
statistics.inTokens,
statistics.outTokens,
+ new BigDecimal(statistics.inTokens).multiply(modelProfile.costInToken()).add(new BigDecimal(statistics.outTokens).multiply(modelProfile.costOutToken())),
false);
}
@@ -120,7 +122,7 @@ public final class AzureAiClient implements Client {
Turn assistantTurn = null;
boolean first = true;
- final int maxLength = modelProfile.maxContextLength() - modelProfile.maxTokens() - 50; // 50 is a margin for EOT and other special tokens
+ final int maxLength = modelProfile.maxContextLength() - modelProfile.maxTokens() - 15; // 15 is a margin for EOT and other special tokens
final AtomicInteger tokens = new AtomicInteger();
while (!turns.isEmpty()) {
@@ -165,21 +167,19 @@ public final class AzureAiClient implements Client {
}
private boolean addChatRequestMessage(final String text, final int maxLength, final AtomicInteger tokens, final List chat, final ModelProfile modelProfile, final Function messageFactory) {
- final int textTokens = this.tokenizer.count(text, modelProfile);
-
- String trimmedText;
+ String prompt = text;
boolean trimmed = false;
+
+ final int textTokens = this.tokenizer.count(prompt, modelProfile);
if (tokens.addAndGet(textTokens) > maxLength) {
// TODO: Cut smarter, e.g. split conversation into parts while keeping system prompt
- LOGGER.warn("Prompt too long: {} tokens / {} max. Prompt will be cut!", tokens.get(), maxLength);
- trimmedText = text.substring(0, text.length() - (tokens.get() - maxLength) / 3);
+ List tokenized = this.tokenizer.tokenize(prompt, modelProfile);
+ LOGGER.warn("Prompt too long: {} tokens / {} max. Prompt will be cut!", tokenized.size(), maxLength);
+ tokenized = tokenized.subList(0, tokenized.size() - (tokens.get() - maxLength));
+ prompt = this.tokenizer.detokenize(tokenized, modelProfile);
trimmed = true;
- }
- else {
- trimmedText = text;
- }
-
- chat.add(messageFactory.apply(trimmedText));
+ }
+ chat.add(messageFactory.apply(prompt));
return trimmed;
}
@@ -246,7 +246,7 @@ public final class AzureAiClient implements Client {
long outTokens;
void add(final Duration duration, final int inTokens, final int outTokens) {
- LOGGER.info("Request duration: {}; in: {}; out: {}; tps: {}", duration, inTokens, outTokens, String.format("%.2f", (inTokens + outTokens) / (double) duration.toMillis() * 1000.0));
+ LOGGER.debug("Request duration: {}; in: {}; out: {}; tps: {}", duration, inTokens, outTokens, String.format("%.2f", (inTokens + outTokens) / (double) duration.toMillis() * 1000.0));
this.requests++;
this.duration = this.duration.plus(duration);
diff --git a/src/main/java/de/extio/lmlib/client/textcompletion/TextCompletionClient.java b/src/main/java/de/extio/lmlib/client/textcompletion/TextCompletionClient.java
index 3a5a663..5050048 100755
--- a/src/main/java/de/extio/lmlib/client/textcompletion/TextCompletionClient.java
+++ b/src/main/java/de/extio/lmlib/client/textcompletion/TextCompletionClient.java
@@ -1,5 +1,6 @@
package de.extio.lmlib.client.textcompletion;
+import java.math.BigDecimal;
import java.text.DecimalFormat;
import java.time.Duration;
import java.time.Instant;
@@ -122,6 +123,7 @@ public class TextCompletionClient implements Client, DisposableBean {
statistics.duration,
statistics.inTokens,
statistics.outTokens,
+ new BigDecimal(statistics.inTokens).multiply(modelProfile.costInToken()).add(new BigDecimal(statistics.outTokens).multiply(modelProfile.costOutToken())),
false);
}
@@ -207,9 +209,10 @@ public class TextCompletionClient implements Client, DisposableBean {
final var dur = Duration.between(start, LocalDateTime.now());
final var in = this.tokenizer.count(promptStr, modelProfile);
final var out = this.tokenizer.count(content, modelProfile);
- LOGGER.debug("Request duration: " + dur + "; in tokens: " + in + "; out tokens: " + out);
- statistics.add(dur, in, out);
- this.totalStatistics.add(dur, in, out);
+ final var cost = new BigDecimal(in).multiply(modelProfile.costInToken()).add(new BigDecimal(out).multiply(modelProfile.costOutToken()));
+ LOGGER.debug("Request duration: " + dur + "; in tokens: " + in + "; out tokens: " + out + "; cost: " + new DecimalFormat("0.####").format(cost));
+ statistics.add(dur, in, out, cost);
+ this.totalStatistics.add(dur, in, out, cost);
}
answer.append(content);
@@ -247,7 +250,7 @@ public class TextCompletionClient implements Client, DisposableBean {
}
private void summarize(final List answers, final ModelProfile modelProfile, final PromptStrategy promptStrategy, final CompletionStatistics statistics) {
- // TODO: Reactivate when multiple (split) prompts have been implemented again
+ // TODO: Reactivate when multiple (split) prompts have been implemented again in #createPrompts
// LOGGER.debug("Summarizing answers");
//
// final StringBuilder summary = new StringBuilder();
@@ -264,7 +267,7 @@ public class TextCompletionClient implements Client, DisposableBean {
@Override
public void destroy() throws Exception {
- if (this.collectStatistics) {
+ if (this.collectStatistics && this.totalStatistics.requests > 0) {
LOGGER.info("Client Statistics: {}", this.totalStatistics);
}
}
@@ -273,19 +276,22 @@ public class TextCompletionClient implements Client, DisposableBean {
private final Instant start = Instant.now();
- volatile int requests;
+ private int requests;
- volatile Duration duration = Duration.ofMillis(0l);
+ private Duration duration = Duration.ofMillis(0l);
- volatile long inTokens;
+ private long inTokens;
- volatile long outTokens;
+ private long outTokens;
- synchronized void add(final Duration duration, final int inTokens, final int outTokens) {
+ private BigDecimal cost = BigDecimal.ZERO;
+
+ synchronized void add(final Duration duration, final int inTokens, final int outTokens, final BigDecimal cost) {
this.requests++;
this.duration = this.duration.plus(duration);
this.inTokens += inTokens;
this.outTokens += outTokens;
+ this.cost = this.cost.add(cost);
}
@Override
@@ -300,16 +306,18 @@ public class TextCompletionClient implements Client, DisposableBean {
builder.append(", outTokens=");
builder.append(this.outTokens);
builder.append(", averageTps=");
- builder.append(new DecimalFormat("#.##").format(((double) (this.outTokens + this.inTokens) / (double) this.requests) / ((double) this.duration.toMillis() / 1000.0 / (double) this.requests)));
+ builder.append(new DecimalFormat("0.##").format(((double) (this.outTokens + this.inTokens) / (double) this.requests) / ((double) this.duration.toMillis() / 1000.0 / (double) this.requests)));
builder.append(", averageOutTps=");
- builder.append(new DecimalFormat("#.##").format((double) this.outTokens / ((double) this.duration.toMillis() / 1000.0)));
+ builder.append(new DecimalFormat("0.##").format((double) this.outTokens / ((double) this.duration.toMillis() / 1000.0)));
builder.append(", effectiveDuration=");
final var effectiveDuration = Duration.between(this.start, Instant.now());
builder.append(effectiveDuration);
builder.append(", effectiveTps=");
- builder.append(new DecimalFormat("#.##").format(((double) (this.outTokens + this.inTokens) / (double) this.requests) / ((double) effectiveDuration.toMillis() / 1000.0 / (double) this.requests)));
+ builder.append(new DecimalFormat("0.##").format(((double) (this.outTokens + this.inTokens) / (double) this.requests) / ((double) effectiveDuration.toMillis() / 1000.0 / (double) this.requests)));
builder.append(", effectiveOutTps=");
- builder.append(new DecimalFormat("#.##").format((double) this.outTokens / ((double) effectiveDuration.toMillis() / 1000.0)));
+ builder.append(new DecimalFormat("0.##").format((double) this.outTokens / ((double) effectiveDuration.toMillis() / 1000.0)));
+ builder.append(", cost=");
+ builder.append(new DecimalFormat("0.###").format(cost.doubleValue()));
builder.append("]");
return builder.toString();
}
diff --git a/src/main/java/de/extio/lmlib/profile/ModelProfile.java b/src/main/java/de/extio/lmlib/profile/ModelProfile.java
index 5bee02a..eccb73e 100755
--- a/src/main/java/de/extio/lmlib/profile/ModelProfile.java
+++ b/src/main/java/de/extio/lmlib/profile/ModelProfile.java
@@ -1,5 +1,7 @@
package de.extio.lmlib.profile;
+import java.math.BigDecimal;
+
public record ModelProfile(
String prompt,
String tokenEncoding,
@@ -11,7 +13,9 @@ public record ModelProfile(
ModelProvider modelProvider,
String modelName,
String url,
- String apiKey) {
+ String apiKey,
+ BigDecimal costInToken,
+ BigDecimal costOutToken) {
public static enum ModelProvider {
AZURE_AI, OAI_TEXT_COMPLETION;
diff --git a/src/main/java/de/extio/lmlib/profile/ModelProfileService.java b/src/main/java/de/extio/lmlib/profile/ModelProfileService.java
index 7a8a709..3d8a3dc 100755
--- a/src/main/java/de/extio/lmlib/profile/ModelProfileService.java
+++ b/src/main/java/de/extio/lmlib/profile/ModelProfileService.java
@@ -1,6 +1,7 @@
package de.extio.lmlib.profile;
import java.io.IOException;
+import java.math.BigDecimal;
import java.util.Collections;
import java.util.List;
import java.util.Objects;
@@ -12,6 +13,7 @@ import org.slf4j.LoggerFactory;
import org.springframework.boot.context.properties.bind.Bindable;
import org.springframework.boot.context.properties.bind.Binder;
import org.springframework.cache.annotation.Cacheable;
+import org.springframework.core.env.ConfigurableEnvironment;
import org.springframework.core.env.Environment;
import org.springframework.core.io.support.ResourcePropertySource;
import org.springframework.stereotype.Service;
@@ -38,7 +40,9 @@ public class ModelProfileService {
throw new IllegalStateException("No model found for profile: " + modelProfile);
}
try {
+
final var resource = new ResourcePropertySource("classpath:" + modelName + ".properties");
+
final var promptTemplate = Optional.ofNullable(resource.getProperty("prompts"))
.map(Object::toString)
.orElseThrow(() -> new IllegalStateException("No model name found for model: " + modelName));
@@ -75,11 +79,29 @@ public class ModelProfileService {
final var url = Optional.ofNullable(resource.getProperty("url"))
.map(Object::toString)
.orElseThrow(() -> new IllegalStateException("No model url found for model: " + modelName));
- final var apiKey = Optional.ofNullable(resource.getProperty("apiKey"))
+ final var costInToken = Optional.ofNullable(resource.getProperty("cost1MInTokens"))
.map(Object::toString)
- .orElseThrow(() -> new IllegalStateException("No model apiKey found for model: " + modelName));
+ .map(value -> new BigDecimal(value))
+ .orElse(BigDecimal.ZERO)
+ .divide(new BigDecimal(1000000.0));
+ final var costOutToken = Optional.ofNullable(resource.getProperty("cost1MOutTokens"))
+ .map(Object::toString)
+ .map(value -> new BigDecimal(value))
+ .orElse(BigDecimal.ZERO)
+ .divide(new BigDecimal(1000000.0));
- return new ModelProfile(promptTemplate, tokenEncoding, maxTokens, maxContextLength, temperature, topP, maxContinuations, modelProvider, modelNameCfg, url, apiKey);
+ //To resolve api key placeholder ${exchangeautodocumentation.azure.apikey} in model-name.properties file
+ String apiKey = null;
+ synchronized (this.environment) {
+ if (this.environment instanceof ConfigurableEnvironment configurableEnvironment) {
+ configurableEnvironment.getPropertySources().addLast(resource);
+ }
+ apiKey = Optional.ofNullable(this.environment.getProperty("apiKey"))
+ .map(Object::toString)
+ .orElseThrow(() -> new IllegalStateException("No model apiKey found for model: " + modelName));
+ }
+
+ return new ModelProfile(promptTemplate, tokenEncoding, maxTokens, maxContextLength, temperature, topP, maxContinuations, modelProvider, modelNameCfg, url, apiKey, costInToken, costOutToken);
}
catch (final IOException e) {
LOGGER.error("Error while reading model profile", e);
diff --git a/src/main/java/de/extio/lmlib/token/TokenizerAutoConfiguration.java b/src/main/java/de/extio/lmlib/token/TokenizerAutoConfiguration.java
index d4d706f..81d9872 100644
--- a/src/main/java/de/extio/lmlib/token/TokenizerAutoConfiguration.java
+++ b/src/main/java/de/extio/lmlib/token/TokenizerAutoConfiguration.java
@@ -14,7 +14,7 @@ public class TokenizerAutoConfiguration {
@Bean
@ConditionalOnProperty(name = "tokenizer.strategy", havingValue = "llamaServer")
- @ConditionalOnMissingBean
+ @ConditionalOnMissingBean(type = "de.extio.lmlib.token.Tokenizer")
Tokenizer llamaServerTokenizer(@Qualifier("lmLibWebClientBuilder") final WebClient.Builder webClientBuilder) {
return new LlamaServerTokenizer(webClientBuilder);
}
@@ -22,7 +22,7 @@ public class TokenizerAutoConfiguration {
@Configuration(proxyBeanMethods = false)
@ConditionalOnClass(name = "com.knuddels.jtokkit.Encodings")
@ConditionalOnProperty(name = "tokenizer.strategy", havingValue = "jTokkit")
- @ConditionalOnMissingBean
+ @ConditionalOnMissingBean(type = "de.extio.lmlib.token.Tokenizer")
public static class SomeServiceConfiguration {
@Bean
diff --git a/src/test/resources/llama3-local.properties b/src/test/resources/llama3-local.properties
index f56566c..931ac4e 100755
--- a/src/test/resources/llama3-local.properties
+++ b/src/test/resources/llama3-local.properties
@@ -9,3 +9,5 @@ modelProvider=OAI_TEXT_COMPLETION
modelName=
url=http://localhost:5001
apiKey=
+cost1MInTokens=0
+cost1MOutTokens=0
diff --git a/src/test/resources/llama3.3-70b-azure.properties b/src/test/resources/llama3.3-70b-azure.properties
new file mode 100644
index 0000000..d2866db
--- /dev/null
+++ b/src/test/resources/llama3.3-70b-azure.properties
@@ -0,0 +1,13 @@
+prompts=
+tokenEncoding=cl100k_base
+maxTokens=1500
+maxContextLength=32767
+temperature=0.2
+topP=1.0
+maxContinuations=1
+modelProvider=AZURE_AI
+modelName=Llama-3.3-70B-Instruct
+url=XXX
+apiKey=XXX
+cost1MInTokens=0.71
+cost1MOutTokens=0.71