feat: Add Spring Boot auto-configuration for lm-lib modules
- Introduced auto-configuration classes for profile, client, agent, and prompt modules. - Updated README to reflect new modular architecture and configuration properties. - Added selective enabling/disabling of modules via application properties. - Refactored existing services and strategies to remove unnecessary Spring annotations. - Enhanced test configurations to utilize auto-configuration. - Updated META-INF imports to include new auto-configuration classes.
This commit is contained in:
@@ -0,0 +1,6 @@
|
||||
de.extio.lmlib.profile.ProfileAutoConfiguration
|
||||
de.extio.lmlib.token.TokenizerAutoConfiguration
|
||||
de.extio.lmlib.client.RestClientConfiguration
|
||||
de.extio.lmlib.prompt.PromptAutoConfiguration
|
||||
de.extio.lmlib.client.ClientAutoConfiguration
|
||||
de.extio.lmlib.agent.AgentAutoConfiguration
|
||||
50
README.md
50
README.md
@@ -1,6 +1,6 @@
|
||||
# lm-lib
|
||||
|
||||
The purpose of lm-lib is to provide a modular, extensible Java framework for interacting with large language models (LLMs) from various providers (such as OpenAI and local models). It abstracts away the complexities of prompt construction, conversation management, and model configuration, enabling developers to easily switch between models and providers. The library supports advanced features like streaming completions, request caching, statistics tracking, and agentic flow execution—allowing for the orchestration of multi-step, branching, and parallel conversational workflows. Integration with Spring ensures flexible configuration and dependency management, making it suitable for building scalable, maintainable, and sophisticated LLM-powered applications.
|
||||
The purpose of lm-lib is to provide a modular, extensible Java framework for interacting with large language models (LLMs) from various providers (such as OpenAI and local models). It abstracts away the complexities of prompt construction, conversation management, and model configuration, enabling developers to easily switch between models and providers. The library supports advanced features like streaming completions, request caching, statistics tracking, and agentic flow execution, allowing for the orchestration of multi-step, branching, and parallel conversational workflows. Integration with Spring ensures flexible configuration and dependency management, making it suitable for building scalable, maintainable, and sophisticated LLM-powered applications.
|
||||
|
||||
The libraries's key features include:
|
||||
|
||||
@@ -14,7 +14,7 @@ The libraries's key features include:
|
||||
- Request caching and statistics tracking
|
||||
- Pluggable interceptors, e.g. for rate limiting
|
||||
- Modular architecture for easy extension and customization
|
||||
- Spring integration for dependency injection and configuration
|
||||
- Spring Boot auto-configuration with selective module enabling
|
||||
|
||||
# Requirements
|
||||
|
||||
@@ -34,7 +34,22 @@ To keep this library lean and compatible with different dependency trees, all de
|
||||
|
||||
## Configuration
|
||||
|
||||
The library can be configured using the following application properties:
|
||||
The library uses Spring Boot auto-configuration with a modular architecture. Each domain can be selectively enabled or disabled.
|
||||
|
||||
### Module Enable/Disable Properties
|
||||
|
||||
| Property | Type | Default | Description |
|
||||
|----------|------|---------|-------------|
|
||||
| `lmlib.profile.enabled` | Boolean | `true` | Enable model profile service for managing LLM configurations. |
|
||||
| `lmlib.tokenizer.enabled` | Boolean | `true` | Enable tokenizer service for token counting and encoding. |
|
||||
| `lmlib.restclient.enabled` | Boolean | `true` | Enable REST client configuration for HTTP communication with LLM providers. |
|
||||
| `lmlib.prompt.enabled` | Boolean | `true` | Enable prompt strategy services for text completion formatting. |
|
||||
| `lmlib.client.enabled` | Boolean | `true` | Enable LLM client services for completion requests. Depends on profile, tokenizer, and restclient modules. |
|
||||
| `lmlib.agent.enabled` | Boolean | `true` | Enable agent executor service for orchestrating agentic flows. Depends on client module. |
|
||||
|
||||
**Note:** Modules have dependencies. Disabling a module that another depends on may cause runtime errors. For example, disabling `lmlib.client.enabled` while keeping `lmlib.agent.enabled=true` requires to provide a custom ClientService implementation.
|
||||
|
||||
### REST Client Configuration
|
||||
|
||||
| Property | Type | Default | Description |
|
||||
|----------|------|---------|-------------|
|
||||
@@ -45,25 +60,48 @@ The library can be configured using the following application properties:
|
||||
| `lmlib.client.proxy.password` | String | | HTTP proxy password (optional). |
|
||||
| `lmlib.client.tls.verification.disabled` | Boolean | `false` | Disable TLS certificate verification (not recommended for production). |
|
||||
| `lmlib.client.retry.max-attempts` | Integer | `5` | Maximum number of retry attempts for failed requests. |
|
||||
| `lmlib.client.retry.backoff-interval-min` | Long | `200` | Minimum backoff interval in milliseconds between retry attempts. |
|
||||
| `lmlib.client.retry.backoff-interval-min` | Long | `125` | Minimum backoff interval in milliseconds between retry attempts. |
|
||||
| `lmlib.client.retry.backoff-interval-max` | Long | `2500` | Maximum backoff interval in milliseconds between retry attempts. |
|
||||
|
||||
### Tokenizer Configuration
|
||||
|
||||
| Property | Type | Default | Description |
|
||||
|----------|------|---------|-------------|
|
||||
| `tokenizer.strategy` | String | `jTokkit` | Token counting strategy. Options: `jTokkit`, `llamaServer`. |
|
||||
|
||||
## Optional dependencies
|
||||
|
||||
Some services benefit from Spring cache framework, e.g. the model profile service:
|
||||
### Caching Support
|
||||
|
||||
Some services benefit from Spring cache framework (e.g., model profile service, client service):
|
||||
|
||||
<dependency>
|
||||
<groupId>org.springframework.boot</groupId>
|
||||
<artifactId>spring-boot-starter-cache</artifactId>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>com.github.ben-manes.caffeine</groupId>
|
||||
<artifactId>caffeine</artifactId>
|
||||
</dependency>
|
||||
|
||||
Don't forget to activate a CacheManager afterwards:
|
||||
Don't forget to activate a CacheManager:
|
||||
|
||||
@Bean
|
||||
public CacheManager cacheManager() {
|
||||
return new CaffeineCacheManager();
|
||||
}
|
||||
|
||||
### Tokenizer Dependencies
|
||||
|
||||
For jTokkit tokenizer (recommended):
|
||||
|
||||
<dependency>
|
||||
<groupId>com.knuddels</groupId>
|
||||
<artifactId>jtokkit</artifactId>
|
||||
</dependency>
|
||||
|
||||
For llamaServer tokenizer, no additional dependencies are needed (uses REST client to query llama.cpp server).
|
||||
|
||||
## Model profiles
|
||||
|
||||
Models are managed with profiles. You can maintain multiple profiles for different models and parameters such as temperature.
|
||||
|
||||
155
src/main/java/de/extio/lmlib/LmLibAutoConfiguration.java
Normal file
155
src/main/java/de/extio/lmlib/LmLibAutoConfiguration.java
Normal file
@@ -0,0 +1,155 @@
|
||||
package de.extio.lmlib;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
import org.springframework.boot.autoconfigure.AutoConfiguration;
|
||||
import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean;
|
||||
import org.springframework.context.annotation.Bean;
|
||||
import org.springframework.core.env.Environment;
|
||||
|
||||
import de.extio.lmlib.agent.AgentExecutorService;
|
||||
import de.extio.lmlib.client.ClientService;
|
||||
import de.extio.lmlib.client.oai.ModelNameSupplier;
|
||||
import de.extio.lmlib.client.oai.completion.chat.ChatCompletionClient;
|
||||
import de.extio.lmlib.client.oai.completion.text.TextCompletionClient;
|
||||
import de.extio.lmlib.profile.ModelProfileService;
|
||||
import de.extio.lmlib.prompt.AlpacaShortPromptStrategy;
|
||||
import de.extio.lmlib.prompt.ChatMLPromptStrategy;
|
||||
import de.extio.lmlib.prompt.GemmaPromptStrategy;
|
||||
import de.extio.lmlib.prompt.GptOssPromptStrategy;
|
||||
import de.extio.lmlib.prompt.Llama3PromptStrategy;
|
||||
import de.extio.lmlib.prompt.Llama4PromptStrategy;
|
||||
import de.extio.lmlib.prompt.MistralNemoPromptStrategy;
|
||||
import de.extio.lmlib.prompt.MistralPromptStrategy;
|
||||
import de.extio.lmlib.prompt.NoPromptStrategy;
|
||||
import de.extio.lmlib.prompt.Phi3PromptStrategy;
|
||||
import de.extio.lmlib.prompt.Phi4PromptStrategy;
|
||||
import de.extio.lmlib.prompt.PromptStrategy;
|
||||
import de.extio.lmlib.prompt.PromptStrategyFactory;
|
||||
import de.extio.lmlib.prompt.Qwen3PromptStrategy;
|
||||
import de.extio.lmlib.prompt.VicunaPromptStrategy;
|
||||
|
||||
@AutoConfiguration
|
||||
public class LmLibAutoConfiguration {
|
||||
|
||||
@Bean
|
||||
@ConditionalOnMissingBean
|
||||
ModelProfileService modelProfileService(final Environment environment) {
|
||||
return new ModelProfileService(environment);
|
||||
}
|
||||
|
||||
@Bean
|
||||
@ConditionalOnMissingBean
|
||||
ClientService clientService() {
|
||||
return new ClientService();
|
||||
}
|
||||
|
||||
@Bean
|
||||
@ConditionalOnMissingBean
|
||||
AgentExecutorService agentExecutorService() {
|
||||
return new AgentExecutorService();
|
||||
}
|
||||
|
||||
@Bean
|
||||
@ConditionalOnMissingBean
|
||||
ModelNameSupplier modelNameSupplier() {
|
||||
return new ModelNameSupplier();
|
||||
}
|
||||
|
||||
@Bean
|
||||
@ConditionalOnMissingBean
|
||||
TextCompletionClient textCompletionClient() {
|
||||
return new TextCompletionClient();
|
||||
}
|
||||
|
||||
@Bean
|
||||
@ConditionalOnMissingBean
|
||||
ChatCompletionClient chatCompletionClient() {
|
||||
return new ChatCompletionClient();
|
||||
}
|
||||
|
||||
@Bean
|
||||
@ConditionalOnMissingBean
|
||||
PromptStrategyFactory promptStrategyFactory(final List<PromptStrategy> strategies) {
|
||||
return new PromptStrategyFactory(strategies);
|
||||
}
|
||||
|
||||
@Bean
|
||||
@ConditionalOnMissingBean
|
||||
AlpacaShortPromptStrategy alpacaShortPromptStrategy() {
|
||||
return new AlpacaShortPromptStrategy();
|
||||
}
|
||||
|
||||
@Bean
|
||||
@ConditionalOnMissingBean
|
||||
ChatMLPromptStrategy chatMLPromptStrategy() {
|
||||
return new ChatMLPromptStrategy();
|
||||
}
|
||||
|
||||
@Bean
|
||||
@ConditionalOnMissingBean
|
||||
GemmaPromptStrategy gemmaPromptStrategy() {
|
||||
return new GemmaPromptStrategy();
|
||||
}
|
||||
|
||||
@Bean
|
||||
@ConditionalOnMissingBean
|
||||
GptOssPromptStrategy gptOssPromptStrategy() {
|
||||
return new GptOssPromptStrategy();
|
||||
}
|
||||
|
||||
@Bean
|
||||
@ConditionalOnMissingBean
|
||||
Llama3PromptStrategy llama3PromptStrategy() {
|
||||
return new Llama3PromptStrategy();
|
||||
}
|
||||
|
||||
@Bean
|
||||
@ConditionalOnMissingBean
|
||||
Llama4PromptStrategy llama4PromptStrategy() {
|
||||
return new Llama4PromptStrategy();
|
||||
}
|
||||
|
||||
@Bean
|
||||
@ConditionalOnMissingBean
|
||||
MistralNemoPromptStrategy mistralNemoPromptStrategy() {
|
||||
return new MistralNemoPromptStrategy();
|
||||
}
|
||||
|
||||
@Bean
|
||||
@ConditionalOnMissingBean
|
||||
MistralPromptStrategy mistralPromptStrategy() {
|
||||
return new MistralPromptStrategy();
|
||||
}
|
||||
|
||||
@Bean
|
||||
@ConditionalOnMissingBean
|
||||
NoPromptStrategy noPromptStrategy() {
|
||||
return new NoPromptStrategy();
|
||||
}
|
||||
|
||||
@Bean
|
||||
@ConditionalOnMissingBean
|
||||
Phi3PromptStrategy phi3PromptStrategy() {
|
||||
return new Phi3PromptStrategy();
|
||||
}
|
||||
|
||||
@Bean
|
||||
@ConditionalOnMissingBean
|
||||
Phi4PromptStrategy phi4PromptStrategy() {
|
||||
return new Phi4PromptStrategy();
|
||||
}
|
||||
|
||||
@Bean
|
||||
@ConditionalOnMissingBean
|
||||
Qwen3PromptStrategy qwen3PromptStrategy() {
|
||||
return new Qwen3PromptStrategy();
|
||||
}
|
||||
|
||||
@Bean
|
||||
@ConditionalOnMissingBean
|
||||
VicunaPromptStrategy vicunaPromptStrategy() {
|
||||
return new VicunaPromptStrategy();
|
||||
}
|
||||
|
||||
}
|
||||
@@ -0,0 +1,22 @@
|
||||
package de.extio.lmlib.agent;
|
||||
|
||||
import org.springframework.boot.autoconfigure.AutoConfiguration;
|
||||
import org.springframework.boot.autoconfigure.AutoConfigureAfter;
|
||||
import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean;
|
||||
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
|
||||
import org.springframework.context.annotation.Bean;
|
||||
|
||||
import de.extio.lmlib.client.ClientAutoConfiguration;
|
||||
|
||||
@AutoConfiguration
|
||||
@AutoConfigureAfter(ClientAutoConfiguration.class)
|
||||
@ConditionalOnProperty(prefix = "lmlib.agent", name = "enabled", havingValue = "true", matchIfMissing = true)
|
||||
public class AgentAutoConfiguration {
|
||||
|
||||
@Bean
|
||||
@ConditionalOnMissingBean
|
||||
AgentExecutorService agentExecutorService() {
|
||||
return new AgentExecutorService();
|
||||
}
|
||||
|
||||
}
|
||||
@@ -15,12 +15,10 @@ import org.springframework.beans.factory.DisposableBean;
|
||||
import org.springframework.beans.factory.InitializingBean;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.beans.factory.annotation.Value;
|
||||
import org.springframework.stereotype.Service;
|
||||
|
||||
import de.extio.lmlib.client.Client;
|
||||
import de.extio.lmlib.client.ClientService;
|
||||
|
||||
@Service
|
||||
public class AgentExecutorService implements InitializingBean, DisposableBean {
|
||||
|
||||
private static final Logger LOGGER = LoggerFactory.getLogger(AgentExecutorService.class);
|
||||
|
||||
@@ -0,0 +1,44 @@
|
||||
package de.extio.lmlib.client;
|
||||
|
||||
import org.springframework.boot.autoconfigure.AutoConfiguration;
|
||||
import org.springframework.boot.autoconfigure.AutoConfigureAfter;
|
||||
import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean;
|
||||
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
|
||||
import org.springframework.context.annotation.Bean;
|
||||
|
||||
import de.extio.lmlib.client.oai.ModelNameSupplier;
|
||||
import de.extio.lmlib.client.oai.completion.chat.ChatCompletionClient;
|
||||
import de.extio.lmlib.client.oai.completion.text.TextCompletionClient;
|
||||
import de.extio.lmlib.profile.ProfileAutoConfiguration;
|
||||
import de.extio.lmlib.token.TokenizerAutoConfiguration;
|
||||
|
||||
@AutoConfiguration
|
||||
@AutoConfigureAfter({ProfileAutoConfiguration.class, TokenizerAutoConfiguration.class, RestClientConfiguration.class})
|
||||
@ConditionalOnProperty(prefix = "lmlib.client", name = "enabled", havingValue = "true", matchIfMissing = true)
|
||||
public class ClientAutoConfiguration {
|
||||
|
||||
@Bean
|
||||
@ConditionalOnMissingBean
|
||||
ClientService clientService() {
|
||||
return new ClientService();
|
||||
}
|
||||
|
||||
@Bean
|
||||
@ConditionalOnMissingBean
|
||||
ModelNameSupplier modelNameSupplier() {
|
||||
return new ModelNameSupplier();
|
||||
}
|
||||
|
||||
@Bean
|
||||
@ConditionalOnMissingBean
|
||||
TextCompletionClient textCompletionClient() {
|
||||
return new TextCompletionClient();
|
||||
}
|
||||
|
||||
@Bean
|
||||
@ConditionalOnMissingBean
|
||||
ChatCompletionClient chatCompletionClient() {
|
||||
return new ChatCompletionClient();
|
||||
}
|
||||
|
||||
}
|
||||
@@ -5,7 +5,6 @@ import java.util.function.Consumer;
|
||||
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.cache.annotation.Cacheable;
|
||||
import org.springframework.stereotype.Service;
|
||||
|
||||
import de.extio.lmlib.client.cached.CachedClient;
|
||||
import de.extio.lmlib.client.cached.CachedClientRepository;
|
||||
@@ -14,7 +13,6 @@ import de.extio.lmlib.profile.ModelProfile;
|
||||
import de.extio.lmlib.profile.ModelProfile.ModelProvider;
|
||||
import de.extio.lmlib.profile.ModelProfileService;
|
||||
|
||||
@Service
|
||||
public class ClientService {
|
||||
|
||||
@Autowired
|
||||
|
||||
@@ -19,8 +19,9 @@ import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.beans.factory.annotation.Qualifier;
|
||||
import org.springframework.beans.factory.annotation.Value;
|
||||
import org.springframework.boot.autoconfigure.AutoConfiguration;
|
||||
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
|
||||
import org.springframework.context.annotation.Bean;
|
||||
import org.springframework.context.annotation.Configuration;
|
||||
import org.springframework.http.HttpHeaders;
|
||||
import org.springframework.http.MediaType;
|
||||
import org.springframework.http.client.ClientHttpRequestInterceptor;
|
||||
@@ -30,7 +31,8 @@ import org.springframework.util.backoff.BackOffExecution;
|
||||
import org.springframework.util.backoff.ExponentialBackOff;
|
||||
import org.springframework.web.client.RestClient;
|
||||
|
||||
@Configuration
|
||||
@AutoConfiguration
|
||||
@ConditionalOnProperty(prefix = "lmlib.restclient", name = "enabled", havingValue = "true", matchIfMissing = true)
|
||||
public class RestClientConfiguration {
|
||||
|
||||
private static final Logger LOGGER = LoggerFactory.getLogger(RestClientConfiguration.class);
|
||||
|
||||
@@ -5,13 +5,11 @@ import java.util.concurrent.ConcurrentHashMap;
|
||||
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.beans.factory.annotation.Qualifier;
|
||||
import org.springframework.stereotype.Component;
|
||||
import org.springframework.web.client.RestClient;
|
||||
|
||||
import de.extio.lmlib.profile.ModelProfile;
|
||||
import de.extio.lmlib.profile.ModelProfile.ModelProvider;
|
||||
|
||||
@Component
|
||||
public class ModelNameSupplier {
|
||||
|
||||
@Autowired
|
||||
|
||||
@@ -14,7 +14,6 @@ import java.util.function.Consumer;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.http.HttpMethod;
|
||||
import org.springframework.stereotype.Component;
|
||||
import org.springframework.web.client.RestClient;
|
||||
|
||||
import tools.jackson.core.JacksonException;
|
||||
@@ -27,7 +26,6 @@ import de.extio.lmlib.client.oai.completion.AbstractCompletionClient;
|
||||
import de.extio.lmlib.profile.ModelProfile;
|
||||
import de.extio.lmlib.profile.ModelProfile.ModelProvider;
|
||||
|
||||
@Component
|
||||
public class ChatCompletionClient extends AbstractCompletionClient {
|
||||
|
||||
private static final Logger LOGGER = LoggerFactory.getLogger(ChatCompletionClient.class);
|
||||
|
||||
@@ -13,7 +13,6 @@ import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.http.HttpMethod;
|
||||
import org.springframework.stereotype.Component;
|
||||
import org.springframework.web.client.RestClient;
|
||||
|
||||
import tools.jackson.core.JacksonException;
|
||||
@@ -29,7 +28,6 @@ import de.extio.lmlib.profile.ModelProfile.ModelProvider;
|
||||
import de.extio.lmlib.prompt.PromptStrategy;
|
||||
import de.extio.lmlib.prompt.PromptStrategyFactory;
|
||||
|
||||
@Component
|
||||
public class TextCompletionClient extends AbstractCompletionClient {
|
||||
|
||||
private static final Logger LOGGER = LoggerFactory.getLogger(TextCompletionClient.class);
|
||||
|
||||
@@ -10,11 +10,9 @@ import org.springframework.cache.annotation.Cacheable;
|
||||
import org.springframework.core.env.ConfigurableEnvironment;
|
||||
import org.springframework.core.env.Environment;
|
||||
import org.springframework.core.io.support.ResourcePropertySource;
|
||||
import org.springframework.stereotype.Service;
|
||||
|
||||
import de.extio.lmlib.profile.ModelProfile.ModelProvider;
|
||||
|
||||
@Service
|
||||
public class ModelProfileService {
|
||||
|
||||
private final Environment environment;
|
||||
|
||||
@@ -0,0 +1,19 @@
|
||||
package de.extio.lmlib.profile;
|
||||
|
||||
import org.springframework.boot.autoconfigure.AutoConfiguration;
|
||||
import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean;
|
||||
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
|
||||
import org.springframework.context.annotation.Bean;
|
||||
import org.springframework.core.env.Environment;
|
||||
|
||||
@AutoConfiguration
|
||||
@ConditionalOnProperty(prefix = "lmlib.profile", name = "enabled", havingValue = "true", matchIfMissing = true)
|
||||
public class ProfileAutoConfiguration {
|
||||
|
||||
@Bean
|
||||
@ConditionalOnMissingBean
|
||||
ModelProfileService modelProfileService(final Environment environment) {
|
||||
return new ModelProfileService(environment);
|
||||
}
|
||||
|
||||
}
|
||||
@@ -1,8 +1,5 @@
|
||||
package de.extio.lmlib.prompt;
|
||||
|
||||
import org.springframework.stereotype.Component;
|
||||
|
||||
@Component
|
||||
public class ChatMLPromptStrategy implements PromptStrategy {
|
||||
|
||||
@Override
|
||||
|
||||
@@ -1,12 +1,5 @@
|
||||
package de.extio.lmlib.prompt;
|
||||
|
||||
import org.springframework.stereotype.Component;
|
||||
|
||||
/**
|
||||
* Works for both Gemma2 and Gemma3
|
||||
* @see https://ai.google.dev/gemma/docs/core/prompt-structure
|
||||
*/
|
||||
@Component
|
||||
public class GemmaPromptStrategy implements PromptStrategy {
|
||||
|
||||
@Override
|
||||
|
||||
@@ -1,8 +1,5 @@
|
||||
package de.extio.lmlib.prompt;
|
||||
|
||||
import org.springframework.stereotype.Component;
|
||||
|
||||
@Component
|
||||
public class GptOssPromptStrategy implements PromptStrategy {
|
||||
|
||||
private static final String finalMarker = "<|channel|>final<|message|>";
|
||||
|
||||
@@ -1,11 +1,5 @@
|
||||
package de.extio.lmlib.prompt;
|
||||
|
||||
import org.springframework.stereotype.Component;
|
||||
|
||||
/**
|
||||
* @see https://llama.meta.com/docs/model-cards-and-prompt-formats/meta-llama-3
|
||||
*/
|
||||
@Component
|
||||
public class Llama3PromptStrategy implements PromptStrategy {
|
||||
|
||||
@Override
|
||||
|
||||
@@ -1,11 +1,6 @@
|
||||
package de.extio.lmlib.prompt;
|
||||
|
||||
import org.springframework.stereotype.Component;
|
||||
|
||||
/**
|
||||
* @see https://www.llama.com/docs/model-cards-and-prompt-formats/llama4/
|
||||
*/
|
||||
@Component
|
||||
public class Llama4PromptStrategy implements PromptStrategy {
|
||||
|
||||
@Override
|
||||
|
||||
@@ -1,8 +1,5 @@
|
||||
package de.extio.lmlib.prompt;
|
||||
|
||||
import org.springframework.stereotype.Component;
|
||||
|
||||
@Component
|
||||
public class MistralNemoPromptStrategy implements PromptStrategy {
|
||||
|
||||
@Override
|
||||
|
||||
@@ -1,8 +1,5 @@
|
||||
package de.extio.lmlib.prompt;
|
||||
|
||||
import org.springframework.stereotype.Component;
|
||||
|
||||
@Component
|
||||
public class MistralPromptStrategy implements PromptStrategy {
|
||||
|
||||
@Override
|
||||
|
||||
@@ -1,8 +1,5 @@
|
||||
package de.extio.lmlib.prompt;
|
||||
|
||||
import org.springframework.stereotype.Component;
|
||||
|
||||
@Component
|
||||
public class NoPromptStrategy implements PromptStrategy {
|
||||
|
||||
@Override
|
||||
|
||||
@@ -1,8 +1,5 @@
|
||||
package de.extio.lmlib.prompt;
|
||||
|
||||
import org.springframework.stereotype.Component;
|
||||
|
||||
@Component
|
||||
public class Phi3PromptStrategy implements PromptStrategy {
|
||||
|
||||
@Override
|
||||
|
||||
@@ -1,8 +1,5 @@
|
||||
package de.extio.lmlib.prompt;
|
||||
|
||||
import org.springframework.stereotype.Component;
|
||||
|
||||
@Component
|
||||
public class Phi4PromptStrategy implements PromptStrategy {
|
||||
|
||||
@Override
|
||||
|
||||
@@ -0,0 +1,98 @@
|
||||
package de.extio.lmlib.prompt;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
import org.springframework.boot.autoconfigure.AutoConfiguration;
|
||||
import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean;
|
||||
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
|
||||
import org.springframework.context.annotation.Bean;
|
||||
|
||||
@AutoConfiguration
|
||||
@ConditionalOnProperty(prefix = "lmlib.prompt", name = "enabled", havingValue = "true", matchIfMissing = true)
|
||||
public class PromptAutoConfiguration {
|
||||
|
||||
@Bean
|
||||
@ConditionalOnMissingBean
|
||||
PromptStrategyFactory promptStrategyFactory(final List<PromptStrategy> strategies) {
|
||||
return new PromptStrategyFactory(strategies);
|
||||
}
|
||||
|
||||
@Bean
|
||||
@ConditionalOnMissingBean
|
||||
AlpacaShortPromptStrategy alpacaShortPromptStrategy() {
|
||||
return new AlpacaShortPromptStrategy();
|
||||
}
|
||||
|
||||
@Bean
|
||||
@ConditionalOnMissingBean
|
||||
ChatMLPromptStrategy chatMLPromptStrategy() {
|
||||
return new ChatMLPromptStrategy();
|
||||
}
|
||||
|
||||
@Bean
|
||||
@ConditionalOnMissingBean
|
||||
GemmaPromptStrategy gemmaPromptStrategy() {
|
||||
return new GemmaPromptStrategy();
|
||||
}
|
||||
|
||||
@Bean
|
||||
@ConditionalOnMissingBean
|
||||
GptOssPromptStrategy gptOssPromptStrategy() {
|
||||
return new GptOssPromptStrategy();
|
||||
}
|
||||
|
||||
@Bean
|
||||
@ConditionalOnMissingBean
|
||||
Llama3PromptStrategy llama3PromptStrategy() {
|
||||
return new Llama3PromptStrategy();
|
||||
}
|
||||
|
||||
@Bean
|
||||
@ConditionalOnMissingBean
|
||||
Llama4PromptStrategy llama4PromptStrategy() {
|
||||
return new Llama4PromptStrategy();
|
||||
}
|
||||
|
||||
@Bean
|
||||
@ConditionalOnMissingBean
|
||||
MistralNemoPromptStrategy mistralNemoPromptStrategy() {
|
||||
return new MistralNemoPromptStrategy();
|
||||
}
|
||||
|
||||
@Bean
|
||||
@ConditionalOnMissingBean
|
||||
MistralPromptStrategy mistralPromptStrategy() {
|
||||
return new MistralPromptStrategy();
|
||||
}
|
||||
|
||||
@Bean
|
||||
@ConditionalOnMissingBean
|
||||
NoPromptStrategy noPromptStrategy() {
|
||||
return new NoPromptStrategy();
|
||||
}
|
||||
|
||||
@Bean
|
||||
@ConditionalOnMissingBean
|
||||
Phi3PromptStrategy phi3PromptStrategy() {
|
||||
return new Phi3PromptStrategy();
|
||||
}
|
||||
|
||||
@Bean
|
||||
@ConditionalOnMissingBean
|
||||
Phi4PromptStrategy phi4PromptStrategy() {
|
||||
return new Phi4PromptStrategy();
|
||||
}
|
||||
|
||||
@Bean
|
||||
@ConditionalOnMissingBean
|
||||
Qwen3PromptStrategy qwen3PromptStrategy() {
|
||||
return new Qwen3PromptStrategy();
|
||||
}
|
||||
|
||||
@Bean
|
||||
@ConditionalOnMissingBean
|
||||
VicunaPromptStrategy vicunaPromptStrategy() {
|
||||
return new VicunaPromptStrategy();
|
||||
}
|
||||
|
||||
}
|
||||
@@ -4,9 +4,6 @@ import java.util.HashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
|
||||
import org.springframework.stereotype.Component;
|
||||
|
||||
@Component
|
||||
public class PromptStrategyFactory {
|
||||
|
||||
private final Map<String, PromptStrategy> strategies = new HashMap<>();
|
||||
|
||||
@@ -1,8 +1,5 @@
|
||||
package de.extio.lmlib.prompt;
|
||||
|
||||
import org.springframework.stereotype.Component;
|
||||
|
||||
@Component
|
||||
public class Qwen3PromptStrategy implements PromptStrategy {
|
||||
|
||||
private static final String IM_START = "<|im_start|>";
|
||||
|
||||
@@ -1,8 +1,5 @@
|
||||
package de.extio.lmlib.prompt;
|
||||
|
||||
import org.springframework.stereotype.Component;
|
||||
|
||||
@Component
|
||||
public class VicunaPromptStrategy implements PromptStrategy {
|
||||
|
||||
@Override
|
||||
|
||||
@@ -6,10 +6,10 @@ import org.springframework.boot.autoconfigure.condition.ConditionalOnClass;
|
||||
import org.springframework.boot.autoconfigure.condition.ConditionalOnMissingBean;
|
||||
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
|
||||
import org.springframework.context.annotation.Bean;
|
||||
import org.springframework.context.annotation.Configuration;
|
||||
import org.springframework.web.client.RestClient;
|
||||
|
||||
@AutoConfiguration
|
||||
@ConditionalOnProperty(prefix = "lmlib.tokenizer", name = "enabled", havingValue = "true", matchIfMissing = true)
|
||||
public class TokenizerAutoConfiguration {
|
||||
|
||||
@Bean
|
||||
@@ -19,17 +19,12 @@ public class TokenizerAutoConfiguration {
|
||||
return new LlamaServerTokenizer(restClientBuilder);
|
||||
}
|
||||
|
||||
@Configuration(proxyBeanMethods = false)
|
||||
@Bean
|
||||
@ConditionalOnClass(name = "com.knuddels.jtokkit.Encodings")
|
||||
@ConditionalOnProperty(name = "tokenizer.strategy", havingValue = "jTokkit")
|
||||
@ConditionalOnProperty(name = "tokenizer.strategy", havingValue = "jTokkit", matchIfMissing = true)
|
||||
@ConditionalOnMissingBean(type = "de.extio.lmlib.token.Tokenizer")
|
||||
public static class SomeServiceConfiguration {
|
||||
|
||||
@Bean
|
||||
Tokenizer jTokkitTokenizer() {
|
||||
return new JTokkitTokenizer();
|
||||
}
|
||||
|
||||
Tokenizer jTokkitTokenizer() {
|
||||
return new JTokkitTokenizer();
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
@@ -1 +1,6 @@
|
||||
de.extio.lmlib.profile.ProfileAutoConfiguration
|
||||
de.extio.lmlib.token.TokenizerAutoConfiguration
|
||||
de.extio.lmlib.client.RestClientConfiguration
|
||||
de.extio.lmlib.prompt.PromptAutoConfiguration
|
||||
de.extio.lmlib.client.ClientAutoConfiguration
|
||||
de.extio.lmlib.agent.AgentAutoConfiguration
|
||||
|
||||
@@ -17,9 +17,9 @@ import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.boot.SpringBootConfiguration;
|
||||
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;
|
||||
import org.springframework.boot.test.context.SpringBootTest;
|
||||
import org.springframework.boot.test.context.SpringBootTest.WebEnvironment;
|
||||
import org.springframework.context.annotation.ComponentScan;
|
||||
import org.springframework.context.annotation.PropertySource;
|
||||
|
||||
import de.extio.lmlib.agent.responsehandler.JsonAgentResponseHandler;
|
||||
@@ -34,7 +34,7 @@ import de.extio.lmlib.profile.ModelCategory;
|
||||
@Disabled("This test requires a running Llama server or a cloud subscription")
|
||||
@SpringBootTest(webEnvironment = WebEnvironment.NONE)
|
||||
@SpringBootConfiguration
|
||||
@ComponentScan(basePackages = "de.extio.lmlib")
|
||||
@EnableAutoConfiguration
|
||||
@PropertySource("classpath:/application-test.properties")
|
||||
public class AgentTest {
|
||||
|
||||
|
||||
@@ -12,9 +12,10 @@ import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.boot.SpringBootConfiguration;
|
||||
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;
|
||||
import org.springframework.boot.test.context.SpringBootTest;
|
||||
import org.springframework.boot.test.context.SpringBootTest.WebEnvironment;
|
||||
import org.springframework.context.annotation.ComponentScan;
|
||||
import org.springframework.context.annotation.Import;
|
||||
import org.springframework.context.annotation.PropertySource;
|
||||
|
||||
import de.extio.lmlib.client.ClientService;
|
||||
@@ -26,7 +27,8 @@ import de.extio.lmlib.profile.ModelCategory;
|
||||
@Disabled("This test requires a running Llama server or a cloud subscription (setup key in model profile)")
|
||||
@SpringBootTest(webEnvironment = WebEnvironment.NONE)
|
||||
@SpringBootConfiguration
|
||||
@ComponentScan(basePackages = "de.extio.lmlib")
|
||||
@EnableAutoConfiguration
|
||||
@Import({ TestCachedClientRepository.Config.class })
|
||||
@PropertySource("classpath:/application-test.properties")
|
||||
public class CachedClientTest {
|
||||
|
||||
|
||||
@@ -3,6 +3,8 @@ package de.extio.lmlib.client.cached;
|
||||
import java.util.concurrent.ConcurrentHashMap;
|
||||
import java.util.concurrent.ConcurrentMap;
|
||||
|
||||
import org.springframework.context.annotation.Bean;
|
||||
import org.springframework.context.annotation.Configuration;
|
||||
import org.springframework.stereotype.Component;
|
||||
|
||||
@Component
|
||||
@@ -10,6 +12,16 @@ class TestCachedClientRepository implements CachedClientRepository {
|
||||
|
||||
private final ConcurrentMap<String, CachedCompletion> cache = new ConcurrentHashMap<>();
|
||||
|
||||
@Configuration
|
||||
static class Config {
|
||||
|
||||
@Bean
|
||||
CachedClientRepository cachedClientRepository() {
|
||||
return new TestCachedClientRepository();
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
@Override
|
||||
public CachedCompletion get(final String key) {
|
||||
return this.cache.get(key);
|
||||
|
||||
@@ -15,9 +15,9 @@ import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.boot.SpringBootConfiguration;
|
||||
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;
|
||||
import org.springframework.boot.test.context.SpringBootTest;
|
||||
import org.springframework.boot.test.context.SpringBootTest.WebEnvironment;
|
||||
import org.springframework.context.annotation.ComponentScan;
|
||||
import org.springframework.context.annotation.PropertySource;
|
||||
|
||||
import de.extio.lmlib.client.ClientService;
|
||||
@@ -32,7 +32,7 @@ import de.extio.lmlib.profile.ModelProfileService;
|
||||
@Disabled("This test requires a running Llama server")
|
||||
@SpringBootTest(webEnvironment = WebEnvironment.NONE)
|
||||
@SpringBootConfiguration
|
||||
@ComponentScan(basePackages = "de.extio.lmlib")
|
||||
@EnableAutoConfiguration
|
||||
@PropertySource("classpath:/application-test.properties")
|
||||
public class CompletionClientTest {
|
||||
|
||||
|
||||
@@ -25,9 +25,9 @@ import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.boot.SpringBootConfiguration;
|
||||
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;
|
||||
import org.springframework.boot.test.context.SpringBootTest;
|
||||
import org.springframework.boot.test.context.SpringBootTest.WebEnvironment;
|
||||
import org.springframework.context.annotation.ComponentScan;
|
||||
import org.springframework.context.annotation.PropertySource;
|
||||
import org.junit.jupiter.api.TestInstance;
|
||||
import org.junit.jupiter.api.TestInstance.Lifecycle;
|
||||
@@ -44,7 +44,7 @@ import de.extio.lmlib.profile.ModelCategory;
|
||||
@Disabled("This test requires a running Llama server or a cloud subscription (setup key in model profile)")
|
||||
@SpringBootTest(webEnvironment = WebEnvironment.NONE)
|
||||
@SpringBootConfiguration
|
||||
@ComponentScan(basePackages = "de.extio.lmlib")
|
||||
@EnableAutoConfiguration
|
||||
@PropertySource("classpath:/application-test.properties")
|
||||
@TestInstance(Lifecycle.PER_CLASS)
|
||||
public class GraderTest {
|
||||
|
||||
@@ -7,9 +7,9 @@ import org.junit.jupiter.api.Test;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.beans.factory.annotation.Qualifier;
|
||||
import org.springframework.boot.SpringBootConfiguration;
|
||||
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;
|
||||
import org.springframework.boot.test.context.SpringBootTest;
|
||||
import org.springframework.boot.test.context.SpringBootTest.WebEnvironment;
|
||||
import org.springframework.context.annotation.ComponentScan;
|
||||
import org.springframework.context.annotation.PropertySource;
|
||||
import org.springframework.web.client.RestClient;
|
||||
|
||||
@@ -18,7 +18,7 @@ import de.extio.lmlib.profile.ModelProfileService;
|
||||
|
||||
@SpringBootTest(webEnvironment = WebEnvironment.NONE)
|
||||
@SpringBootConfiguration
|
||||
@ComponentScan(basePackages = "de.extio.lmlib")
|
||||
@EnableAutoConfiguration
|
||||
@PropertySource("classpath:/application-test.properties")
|
||||
public class TokenizerTest {
|
||||
|
||||
|
||||
Reference in New Issue
Block a user