feat: add error handling and logging for tokenization and detokenization requests in LlamaServerTokenizer

README.md

@@ -121,7 +121,7 @@ public CacheManager cacheManager() {
 
 ### Tokenizer Dependencies
 
-For the recommended `jTokkit` tokenizer:
+For the `jTokkit` tokenizer:
 
 ```xml
 <dependency>
@@ -130,6 +130,14 @@ For the recommended `jTokkit` tokenizer:
 </dependency>
 ```
 
+## Tokenizers
+
+lm-lib registers tokenizer implementations as beans when their dependencies are available, then resolves the tokenizer at runtime from the active `ModelProfile`. Applications can implement the `Tokenizer` interface to inject their own Tokenizers.
+
+Tokenizer selection is configured per model profile with the `tokenizer` property. If a profile does not specify a tokenizer, lm-lib prefers `jTokkit` when the dependency is present and otherwise falls back to the built-in `fallback` tokenizer. The fallback tokenizer is deterministic and reversible, but it only provides approximate token counts.
+
+As an alternative, the library supports Hugging Face-style /tokenize and /detokenize endpoints (`llamaServer`). These endpoints must be available below base URL from the modelProfile.
+These endpoints are provided, for example, by llama.cpp llama-server and vLLM.
 For `llamaServer`, no extra dependency is required because tokenization is delegated to the remote server through the configured model endpoint.
 
 Available built-in tokenizer names:
@@ -138,12 +146,6 @@ Available built-in tokenizer names:
 - `llamaServer`
 - `fallback`
 
-### Tokenizer Configuration
-
-lm-lib registers tokenizer implementations as beans when their dependencies are available, then resolves the tokenizer at runtime from the active `ModelProfile`. Applications can implement the `Tokenizer` interface to inject their own Tokenizers.
-
-Tokenizer selection is configured per model profile with the `tokenizer` property. If a profile does not specify a tokenizer, lm-lib prefers `jTokkit` when the dependency is present and otherwise falls back to the built-in `fallback` tokenizer. The fallback tokenizer is deterministic and reversible, but it only provides approximate token counts.
-
 ## Model Profiles
 
 Model profiles describe how a request should be sent to a model and how its usage should be accounted for. They can be loaded from `.properties` files, selected dynamically at runtime, or created programmatically.