183 lines
7.4 KiB
Markdown
183 lines
7.4 KiB
Markdown
# lm-lib
|
|
|
|
The purpose of this library is to provide a comprehensive conversational AI system that enables efficient and effective human-computer interactions through natural language processing, conversation management, and text completion capabilities, utilizing caching, tokenization, and integration with external AI models and services to deliver a robust and scalable solution for applications such as chatbots, virtual assistants, and service platforms.
|
|
|
|
The program's key features include:
|
|
|
|
* Agent graph execution
|
|
* Integration with external AI models and services (e.g., Azure AI and OpenAi Api compatible)
|
|
* Model profiles (choose the right model and parameters for the task)
|
|
* Conversation management and text completion with support for multiple prompt strategies and tokenization implementations
|
|
* Prompt caching
|
|
|
|
# Setup
|
|
|
|
To keep this library lean and compatible with different dependency trees, all dependencies must be provided by the application.
|
|
|
|
## Mandatory dependencies
|
|
|
|
<dependency>
|
|
<groupId>org.springframework.boot</groupId>
|
|
<artifactId>spring-boot-starter-webflux</artifactId>
|
|
</dependency>
|
|
|
|
## Optional dependencies
|
|
|
|
Some services benefit from Spring cache framework, e.g. the model profile service:
|
|
|
|
<dependency>
|
|
<groupId>org.springframework.boot</groupId>
|
|
<artifactId>spring-boot-starter-cache</artifactId>
|
|
</dependency>
|
|
|
|
Don't forget to activate a CacheManager afterwards:
|
|
|
|
@Bean
|
|
public CacheManager cacheManager() {
|
|
return new CaffeineCacheManager();
|
|
}
|
|
|
|
## Model profiles
|
|
|
|
Models are managed with profiles. You can maintain multiple profiles for different models and parameters such as temperature.
|
|
|
|
These profiles must be mapped to fixed categories with the following application properties:
|
|
|
|
profile.model.s=
|
|
profile.model.m=
|
|
profile.model.l=
|
|
profile.model.hot=
|
|
profile.model.cold=
|
|
|
|
The value is the name of the model profile .properties file, for example "llama3.3-70b-azure".
|
|
|
|
Profiles must be located in the main resources location.
|
|
|
|
Examples can be found in test resources.
|
|
|
|
## OpenAi compatible client
|
|
|
|
Use tokenizer strategy jtokkit or llamaServer (if the model is hosted on a llama.cpp server)
|
|
|
|
Property:
|
|
|
|
tokenizer.strategy=llamaServer
|
|
|
|
If you use jtokkit tokenizer:
|
|
|
|
<dependency>
|
|
<groupId>com.knuddels</groupId>
|
|
<artifactId>jtokkit</artifactId>
|
|
</dependency>
|
|
|
|
It is required to select the right prompt template. See package .lmlib.prompt for available prompt templates or implement your own PromptStrategy (must be a bean in the spring context).
|
|
|
|
Set the following property in your model profile:
|
|
|
|
prompts=llama3
|
|
|
|
Example profile: src/test/resources/llama3-local.properties
|
|
|
|
## Azure AI client
|
|
|
|
Use tokenizer strategy jtokkit.
|
|
|
|
Property:
|
|
|
|
tokenizer.strategy=jtokkit
|
|
|
|
Example profile: src/test/resources/llama3.3-70b-azure.properties
|
|
|
|
<dependency>
|
|
<groupId>com.azure</groupId>
|
|
<artifactId>azure-ai-inference</artifactId>
|
|
</dependency>
|
|
<dependency>
|
|
<groupId>com.squareup.okhttp3</groupId>
|
|
<artifactId>okhttp</artifactId>
|
|
</dependency>
|
|
<dependency>
|
|
<groupId>com.knuddels</groupId>
|
|
<artifactId>jtokkit</artifactId>
|
|
</dependency>
|
|
<dependency>
|
|
<groupId>com.github.ben-manes.caffeine</groupId>
|
|
<artifactId>caffeine</artifactId>
|
|
</dependency>
|
|
|
|
# Completions Cache
|
|
|
|
All language model requests will be cached if you add a CachedClientRepository bean to your context.
|
|
|
|
# Completion Interceptors
|
|
|
|
You can intercept every Conversation and Completion by implementing CompletionInterceptor.
|
|
This is for example useful for accounting purposes or to implement a circuit breaker.
|
|
The client will autowire all implementations and intercept requests accordingly.
|
|
|
|
# Agentic Flows
|
|
|
|
This library contains a graph executor for implementing comprehensive processes using agent-based design patterns. See class AgentTest for examples.
|
|
|
|
# Grader
|
|
|
|
The library includes a grader component that evaluates the correctness of agent responses.
|
|
|
|
Example:
|
|
|
|
assertTrue(Grader.assessScoreBinary("Does the following blog post mention the feature " + feature, resultContext.getStringValue("blogPost"), this.clientService));
|
|
|
|
# Usage
|
|
|
|
1. Setup dependencies, see above
|
|
2. Configure model profiles, see above
|
|
3. Implement a CachedClientRepository if you need your requests cached
|
|
4. For chats or completions, use ClientService service bean
|
|
5. For agentic flows, create an AgentContext and use AgentExecutorService service bean
|
|
|
|
# Usage Examples
|
|
|
|
Text completions: AzureAiClientTest#completion
|
|
Chat: AzureAiClientTest#chat
|
|
Agentic flow: AgentTest#agenticFlow
|
|
|
|
# Announcement generated by AgentTest#agenticFlow
|
|
|
|
Title: Introducing a Powerful Framework for Building Intelligent Agents
|
|
|
|
We're excited to announce the development of a new Java codebase designed to build intelligent agents capable of interacting with language models like GPT. This framework offers a flexible and extensible architecture, empowering developers to create sophisticated conversational agents for a wide range of applications.
|
|
|
|
**Key Features:**
|
|
|
|
* **Concurrent Text Processing:** Leveraging Java's `CompletableFuture` and `ExecutorService`, the framework enables parallel processing of text chunks, significantly speeding up agent execution.
|
|
|
|
* **Response-Driven Agent Selection:** The `chooseNextAgent` method intelligently selects the next agent in the chain based on the response received from the language model, allowing for dynamic and adaptive conversations.
|
|
|
|
* **Customizable Response Handling:** Each agent can be equipped with a custom `responseHandler` to parse and interpret responses from the language model, enabling tailored logic for specific tasks.
|
|
|
|
* **Dynamic Agent Chaining:** Agents can be chained together to create complex, multi-step interactions, with each agent specializing in a particular task and passing context to the next.
|
|
|
|
* **Contextualized Agent Execution:** The framework utilizes an `AgentContext` object to store and share information between agents, ensuring each agent has access to the relevant context for its task.
|
|
|
|
**How it Works:**
|
|
|
|
The framework revolves around the `Agent` class, which defines an agent's behavior and interaction with the language model. Each agent receives an `AgentContext` containing relevant information, processes text using its defined template, interacts with the language model, and then uses its `responseHandler` to interpret the response. Based on the response, the `chooseNextAgent` method determines the next agent in the chain.
|
|
|
|
**Benefits:**
|
|
|
|
This framework offers several advantages for developers:
|
|
|
|
* **Flexibility:** Easily customize agent behavior through configurable parameters and custom response handlers.
|
|
* **Extensibility:** Add new agent types and functionalities as needed.
|
|
* **Efficiency:** Parallel processing and dynamic agent chaining optimize performance.
|
|
* **Adaptability:** Agents can respond intelligently to changing conversation contexts.
|
|
|
|
**Looking Ahead:**
|
|
|
|
We believe this framework has the potential to revolutionize the way we build conversational agents. We're actively developing new features and functionalities, and we encourage the community to contribute and explore the possibilities.
|
|
|
|
Stay tuned for more updates and documentation!
|
|
|
|
*Graph: M, CodeSummarizer, ↴, M, FeatureIdentifier, →, ⇉ 3/5, M, FeatureAnalyzer, →, 5↣1, M, AnswerGenerator*
|
|
*gemma2-27b requests=8, inTokens=19646, outTokens=2784, requestDuration=PT7M38.309006603S, getAbsoluteDuration()=PT3M54.653718763S, getTps()=95.59, getOutTps()=11.86*
|