feat: add Anthropic tokenizer backed by count_tokens API

Adds the anthropic tokenizer that delegates token counting to the
Anthropic endpoint POST /v1/messages/count_tokens. Since the endpoint
only supports counting, tokenize and detokenize operate on a plain
codepoint list instead of real model tokens.

The local anthropic test profile now uses the new tokenizer.
This commit is contained in:
extio-de
2026-10-17 21:03:20 +02:00
parent cd99dfa360
commit 152c5845e6
5 changed files with 206 additions and 1 deletions

View File

@@ -141,10 +141,13 @@ As an alternative, the library supports Hugging Face-style /tokenize and /detoke
These endpoints are provided, for example, by llama.cpp llama-server and vLLM.
For `llamaServer`, no extra dependency is required because tokenization is delegated to the remote server through the configured model endpoint.
The `anthropic` tokenizer delegates counting to the Anthropic token counting endpoint (`POST /v1/messages/count_tokens`), which must be available below the base URL from the model profile. The endpoint only supports counting, so `tokenize` and `detokenize` operate on a plain codepoint list instead of real model tokens.
Available built-in tokenizer names:
- `jTokkit`
- `llamaServer`
- `anthropic`
- `fallback`
## Model Profiles