79 lines
3.2 KiB
Markdown
79 lines
3.2 KiB
Markdown
# Ollama OpenAI Bridge
|
|
|
|
`ollama-oai-bridge` is a Spring Boot-based proxy and translation layer designed to bridge the gap between OpenAI-compatible clients and Ollama (or other Llama-based) servers.
|
|
|
|
## Overview
|
|
|
|
The `ollama-oai-bridge` is specifically designed to work with a `llama-server` in the downstream. It simulates Ollama's model discovery endpoints while using OpenAI-compatible endpoints for the actual LLM requests.
|
|
|
|
This setup provides an easy way to use local models with VS Code GitHub Copilot BYOK (Bring Your Own Key) agents. By using `llama.cpp` as the upstream server, it leverages superior model and device support compared to Ollama, while still being compatible with tools that expect the Ollama API for model discovery.
|
|
|
|
## Key Features
|
|
|
|
- **OpenAI-to-Ollama Discovery**: Translates model lists and model details into the format expected by Ollama clients.
|
|
- **Model Metadata Normalization**: Automatically infers model families (Llama, Gemma, Mistral, etc.), quantization levels, and capabilities (Vision, Tools, Thinking) from model names and metadata.
|
|
- **Dynamic Modelfile Generation**: Can generate Ollama-compatible `Modelfile` content for any model served by the upstream.
|
|
- **Request Caching**: Includes a caching filter to allow inspection of request bodies without consuming the input stream.
|
|
- **Configurable Upstream**: Easily point the bridge to any upstream LLM server via configuration.
|
|
|
|
## Configuration
|
|
|
|
The application is configured via `src/main/resources/application.properties`.
|
|
|
|
| Property | Default | Description |
|
|
|----------|---------|-------------|
|
|
| `server.port` | `11434` | The port on which the bridge runs. |
|
|
| `bridge.upstream-base-url` | `http://localhost:5001` | The base URL of the upstream LLM server. |
|
|
| `bridge.version` | `0.24.0` | The ollama version to report. |
|
|
|
|
## API Endpoints
|
|
|
|
### Discovery API (Ollama Compatible)
|
|
|
|
- `GET /api/version`: Returns the bridge version.
|
|
- `GET /api/tags`: Returns a list of available models.
|
|
- `POST /api/show`: Returns detailed information about a specific model.
|
|
|
|
### Proxying
|
|
|
|
- All other requests are proxied to the `bridge.upstream-base-url`.
|
|
|
|
## Getting Started
|
|
|
|
### Prerequisites
|
|
|
|
- Java 25
|
|
- Maven
|
|
|
|
### Building and Running
|
|
|
|
1. **Clone the repository**:
|
|
```bash
|
|
git clone <repository-url>
|
|
cd ollama-oai-bridge
|
|
```
|
|
|
|
2. **Build the project**:
|
|
```bash
|
|
./mvnw clean package
|
|
```
|
|
|
|
3. **Run the application**:
|
|
You can run it using Maven:
|
|
```bash
|
|
./mvnw spring-boot:run
|
|
```
|
|
|
|
Or, after building, you can use the provided `start.sh` script. Copy the generated `.jar` file from the `target/` directory and `start.sh` into the same directory, then run:
|
|
```bash
|
|
./start.sh
|
|
```
|
|
|
|
## Architecture
|
|
|
|
- **`GatewayProxyConfiguration`**: Sets up the Spring Cloud Gateway routes to proxy requests.
|
|
- **`LlamaServerClient`**: Handles low-level communication with the upstream server and normalizes model data.
|
|
- **`OllamaDiscoveryService`**: Implements the logic for translating upstream data into Ollama-compatible JSON responses.
|
|
- **`OllamaDiscoveryController`**: Exposes the discovery endpoints.
|
|
- **`RequestBodyCachingFilter`**: Ensures request bodies can be read multiple times during the request lifecycle.
|