The purpose of this library is to provide a comprehensive conversational AI system that enables efficient and effective human-computer interactions through natural language processing, conversation management, and text completion capabilities, utilizing caching, tokenization, and integration with external AI models and services to deliver a robust and scalable solution for applications such as chatbots, virtual assistants, and service platforms.

The program's key features include:

Agent graph execution (incl. template engine and branching capabilities)
Integration with external AI models and services (e.g., Azure AI and OpenAi Api compatible)
Model profiles (choose the right model and parameters for the task)
Conversation management and text completion with support for multiple prompt strategies and tokenization implementations
Prompt caching

Setup

To keep this library lean and compatible with different dependency trees, all dependencies must be provided by the application.

Mandatory dependencies

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-webflux</artifactId>
</dependency>

Optional dependencies

Some services benefit from Spring cache framework, e.g. the model profile service:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-cache</artifactId>
</dependency>

Don't forget to activate a CacheManager afterwards:

@Bean
public CacheManager cacheManager() {
    return new CaffeineCacheManager();
}

Model profiles

Models are managed with profiles. You can maintain multiple profiles for different models and parameters such as temperature.

These profiles must be mapped to fixed categories with the following application properties:

profile.model.s=
profile.model.m=
profile.model.l=
profile.model.hot=
profile.model.cold=

The value is the name of the model profile .properties file, for example "llama3.3-70b-azure".

Agent and completion requests will be invoked with these categories and will load the corresponding model profile.

Profiles must be located in the classpath resources location (e.g. src/main/resources).

Examples can be found in test resources.

OpenAi compatible client

Use tokenizer strategy jtokkit or llamaServer (if the model is hosted on a llama.cpp server)

Property:

tokenizer.strategy=llamaServer

If you use jtokkit tokenizer:

<dependency>
    <groupId>com.knuddels</groupId>
    <artifactId>jtokkit</artifactId>
</dependency>

The library supports both chat completions and (legacy) text completions. Choose one of these options in your model profile:

modelProvider=OAI_CHAT_COMPLETION
modelProvider=OAI_TEXT_COMPLETION

Chat completions are recommended for most use cases and easy to use, since the library only needs to send the actual conversation to the server, whereas for text completions, the library needs to create a raw prompt that includes the correct instruction formatting (e.g. special tokens). Text completions are still preferred if you need more control over the raw prompt or if you don't use an instruct model. For text completions, it is required to select the right prompt template. See package .lmlib.prompt for available prompt templates or implement your own PromptStrategy (must be a bean in the spring context). There is also the special NoPromptStrategy ("no") that does not add any special tokens. To activate a prompt template, set the following property in your model profile (corresponds to getPromptName()):

prompts=llama3

Example profile: src/test/resources/xxx-local.properties

Azure AI client

Use tokenizer strategy jtokkit.

Property:

tokenizer.strategy=jtokkit

Example profile: src/test/resources/llama3.3-70b-azure.properties

<dependency>
	<groupId>com.azure</groupId>
	<artifactId>azure-ai-inference</artifactId>
</dependency>
<dependency>
	<groupId>com.squareup.okhttp3</groupId>
	<artifactId>okhttp</artifactId>
</dependency>    
<dependency>
	<groupId>com.knuddels</groupId>
	<artifactId>jtokkit</artifactId>
</dependency>
<dependency>
	<groupId>com.github.ben-manes.caffeine</groupId>
	<artifactId>caffeine</artifactId>
</dependency>

Usage

Setup dependencies, see above
Configure model profiles, see above
Implement a CachedClientRepository if you need your requests cached
For chats or completions, use ClientService service bean
For agentic flows, create an AgentContext and use AgentExecutorService service bean

Usage Examples

Client service: clientService.getClient(ModelCategory.MEDIUM); Client usage details: CompletionClientTest and AzureAiClientTest Agentic flow: AgentTest#agenticFlow

Completions and Streaming

The library supports both streaming and non-streaming completions.

Streaming methods accept a Consumer to receive the next chunk (direct Client usage) or a callback on the next AgentContext update (Agents). They also return a full Completion object or the final AgentContext at the end of the execution that also contains the statistics.

Non-streaming methods only return the final objects at the end of the execution. This approach is much more resource-friendly and therefore recommended if you don't need to display or process the results in real-time.

Completions Cache

All language model requests will be cached if you add a CachedClientRepository bean to your context.

Completion Interceptors

You can intercept every Conversation and Completion by implementing CompletionInterceptor. This is for example useful for accounting purposes or to implement a circuit breaker. The client will autowire all implementations and intercept requests accordingly.

Agentic Flows

This library contains a graph executor for implementing comprehensive processes using agent-based design patterns. See class AgentTest for examples.

Grader

The library includes a grader component that evaluates the correctness of agent responses.

Example:

assertTrue(Grader.assessScoreBinary("Does the following blog post mention the feature " + feature, resultContext.getStringValue("blogPost"), this.clientService));

Announcement generated by AgentTest#agenticFlow

Introducing Advanced Agent Orchestration: Dynamic Conversations & Parallel Processing

We're excited to announce significant enhancements to our agent framework, designed to empower developers building sophisticated conversational AI applications. This release introduces a powerful new system for orchestrating multiple agents, enabling more dynamic, efficient, and complex interactions.

Here's a breakdown of what's new:

1. LLM Conversation Orchestration:

Our framework now provides a robust system for chaining multiple agents together. Each agent can perform a specific task, and the framework intelligently manages the flow of information between them, creating a pipeline for complex conversational workflows.

2. Dynamic Prompt Templating with Branching:

Say goodbye to rigid prompts! Our new templating engine allows for dynamic prompt generation based on the context. Critically, it now supports branching. This means a single input can trigger multiple prompt variations, allowing agents to explore different avenues of conversation or processing paths based on data within the input.

3. Asynchronous Task Execution for Parallel Processing:

Speed and efficiency are key. We've implemented asynchronous task execution using CompletableFuture and an ExecutorService. This allows agents to process multiple inputs (splits) in parallel, significantly reducing overall processing time and improving responsiveness.

4. Context Merging Across Multiple Agent Splits:

When branching occurs, the framework intelligently merges the results from different execution paths. This ensures a unified context is maintained, allowing subsequent agents to access a complete and consolidated view of the conversation or task.

5. Agent Chaining and Next Agent Selection:

The framework dynamically determines the next agent to execute based on the current context. This allows for creating complex, stateful conversations where the flow adapts based on user input and agent responses. The system can also signal the end of the chain, indicating completion.

What does this mean for you?

These improvements unlock a new level of flexibility and power for building conversational AI applications. You can now:

Create more complex and nuanced conversational flows.
Handle a wider range of user inputs and scenarios.
Improve the performance and responsiveness of your agents.
Build more scalable and maintainable AI systems.

We're confident that these advancements will significantly enhance your ability to build intelligent and engaging conversational experiences. Stay tuned for more updates and documentation!

gemma3-12b-it-q5_k_m M, CodeSummarizer, ↴, M, FeatureIdentifier, →, ⇉ 1/5, M, FeatureAnalyzer, 5↣1, →, M, AnswerGenerator AgentRequestStatistic [requests=8, cachedPrompts=0, inTokens=27393, outTokens=4861, requestDuration=PT3M30.415881394S, getEffectiveDuration()=PT2M22.102597491S, getTps()=226.98, getOutTps()=34.21, getCost()=0]