lm-lib/README.md

lm-lib

The purpose of this library is to provide a comprehensive conversational AI system that enables efficient and effective human-computer interactions through natural language processing, conversation management, and text completion capabilities, utilizing caching, tokenization, and integration with external AI models and services to deliver a robust and scalable solution for applications such as chatbots, virtual assistants, and service platforms.

The program's key features include:

• Agent graph execution (incl. template engine and branching capabilities)
• Integration with external AI models and services (e.g., Azure AI and OpenAi Api compatible)
• Model profiles (choose the right model and parameters for the task)
• Conversation management and text completion with support for multiple prompt strategies and tokenization implementations
• Prompt caching

Setup

To keep this library lean and compatible with different dependency trees, all dependencies must be provided by the application.

Mandatory dependencies

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-webflux</artifactId>
</dependency>

Optional dependencies

Some services benefit from Spring cache framework, e.g. the model profile service:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-cache</artifactId>
</dependency>		

Don't forget to activate a CacheManager afterwards:

@Bean
public CacheManager cacheManager() {
    return new CaffeineCacheManager();
}

Model profiles

Models are managed with profiles. You can maintain multiple profiles for different models and parameters such as temperature.

These profiles must be mapped to fixed categories with the following application properties:

profile.model.s=
profile.model.m=
profile.model.l=
profile.model.hot=
profile.model.cold=

The value is the name of the model profile .properties file, for example "llama3.3-70b-azure".

Profiles must be located in the main resources location.

Examples can be found in test resources.

OpenAi compatible client

Use tokenizer strategy jtokkit or llamaServer (if the model is hosted on a llama.cpp server)

Property:

tokenizer.strategy=llamaServer

If you use jtokkit tokenizer:

<dependency>
    <groupId>com.knuddels</groupId>
    <artifactId>jtokkit</artifactId>
</dependency>

It is required to select the right prompt template. See package .lmlib.prompt for available prompt templates or implement your own PromptStrategy (must be a bean in the spring context).

Set the following property in your model profile:

prompts=llama3

Example profile: src/test/resources/llama3-local.properties

Azure AI client

Use tokenizer strategy jtokkit.

Property:

tokenizer.strategy=jtokkit

Example profile: src/test/resources/llama3.3-70b-azure.properties

<dependency>
	<groupId>com.azure</groupId>
	<artifactId>azure-ai-inference</artifactId>
</dependency>
<dependency>
	<groupId>com.squareup.okhttp3</groupId>
	<artifactId>okhttp</artifactId>
</dependency>    
<dependency>
	<groupId>com.knuddels</groupId>
	<artifactId>jtokkit</artifactId>
</dependency>
<dependency>
	<groupId>com.github.ben-manes.caffeine</groupId>
	<artifactId>caffeine</artifactId>
</dependency>

Completions Cache

All language model requests will be cached if you add a CachedClientRepository bean to your context.

Completion Interceptors

You can intercept every Conversation and Completion by implementing CompletionInterceptor. This is for example useful for accounting purposes or to implement a circuit breaker. The client will autowire all implementations and intercept requests accordingly.

Agentic Flows

This library contains a graph executor for implementing comprehensive processes using agent-based design patterns. See class AgentTest for examples.

Grader

The library includes a grader component that evaluates the correctness of agent responses.

Example:

assertTrue(Grader.assessScoreBinary("Does the following blog post mention the feature " + feature, resultContext.getStringValue("blogPost"), this.clientService));

Usage

1. Setup dependencies, see above
2. Configure model profiles, see above
3. Implement a CachedClientRepository if you need your requests cached
4. For chats or completions, use ClientService service bean
5. For agentic flows, create an AgentContext and use AgentExecutorService service bean

Usage Examples

Text completions: AzureAiClientTest#completion
Chat: AzureAiClientTest#chat
Agentic flow: AgentTest#agenticFlow

Announcement generated by AgentTest#agenticFlow

Introducing Advanced Agent Orchestration: Dynamic Conversations & Parallel Processing

We're excited to announce significant enhancements to our agent framework, designed to empower developers building sophisticated conversational AI applications. This release introduces a powerful new system for orchestrating multiple agents, enabling more dynamic, efficient, and complex interactions.

Here's a breakdown of what's new:

1. LLM Conversation Orchestration:

Our framework now provides a robust system for chaining multiple agents together. Each agent can perform a specific task, and the framework intelligently manages the flow of information between them, creating a pipeline for complex conversational workflows.

2. Dynamic Prompt Templating with Branching:

Say goodbye to rigid prompts! Our new templating engine allows for dynamic prompt generation based on the context. Critically, it now supports branching. This means a single input can trigger multiple prompt variations, allowing agents to explore different avenues of conversation or processing paths based on data within the input.

3. Asynchronous Task Execution for Parallel Processing:

Speed and efficiency are key. We've implemented asynchronous task execution using CompletableFuture and an ExecutorService. This allows agents to process multiple inputs (splits) in parallel, significantly reducing overall processing time and improving responsiveness.

4. Context Merging Across Multiple Agent Splits:

When branching occurs, the framework intelligently merges the results from different execution paths. This ensures a unified context is maintained, allowing subsequent agents to access a complete and consolidated view of the conversation or task.

5. Agent Chaining and Next Agent Selection:

The framework dynamically determines the next agent to execute based on the current context. This allows for creating complex, stateful conversations where the flow adapts based on user input and agent responses. The system can also signal the end of the chain, indicating completion.

What does this mean for you?

These improvements unlock a new level of flexibility and power for building conversational AI applications. You can now:

• Create more complex and nuanced conversational flows.
• Handle a wider range of user inputs and scenarios.
• Improve the performance and responsiveness of your agents.
• Build more scalable and maintainable AI systems.

We're confident that these advancements will significantly enhance your ability to build intelligent and engaging conversational experiences. Stay tuned for more updates and documentation!

gemma3-12b-it-q5_k_m M, CodeSummarizer, ↴, M, FeatureIdentifier, →, ⇉ 1/5, M, FeatureAnalyzer, 5↣1, →, M, AnswerGenerator AgentRequestStatistic [requests=8, cachedPrompts=0, inTokens=27393, outTokens=4861, requestDuration=PT3M30.415881394S, getEffectiveDuration()=PT2M22.102597491S, getTps()=226.98, getOutTps()=34.21, getCost()=0]