108 lines
4.4 KiB
Markdown
108 lines
4.4 KiB
Markdown
# Ollama OpenAI Bridge
|
|
|
|
`ollama-oai-bridge` is a Spring Boot-based proxy and translation layer designed to bridge the gap between OpenAI-compatible clients and Ollama (or other Llama-based) servers.
|
|
|
|
## Overview
|
|
|
|
The `ollama-oai-bridge` is specifically designed to work with a `llama-server` in the downstream. It simulates Ollama's model discovery endpoints while using OpenAI-compatible endpoints for the actual LLM requests.
|
|
|
|
This setup provides an easy way to use local models with VS Code GitHub Copilot BYOK (Bring Your Own Key) agents. By using `llama.cpp` as the upstream server, it leverages superior model and device support compared to Ollama, while still being compatible with tools that expect the Ollama API for model discovery.
|
|
|
|
## Key Features
|
|
|
|
- **OpenAI-to-Ollama Discovery**: Translates model lists and model details into the format expected by Ollama clients.
|
|
- **Model Metadata Normalization**: Automatically infers model families (Llama, Gemma, Mistral, etc.), quantization levels, and capabilities (Vision, Tools, Thinking) from model names and metadata.
|
|
- **Dynamic Modelfile Generation**: Can generate Ollama-compatible `Modelfile` content for any model served by the upstream.
|
|
- **Request Caching**: Includes a caching filter to allow inspection of request bodies without consuming the input stream.
|
|
- **Configurable Upstream**: Easily point the bridge to any upstream LLM server via configuration.
|
|
|
|
## Configuration
|
|
|
|
The application is configured via `src/main/resources/application.properties`.
|
|
|
|
| Property | Default | Description |
|
|
|----------|---------|-------------|
|
|
| `server.port` | `11434` | The port on which the bridge runs. |
|
|
| `bridge.upstream-base-url` | `http://localhost:5001` | The base URL of the upstream LLM server. |
|
|
| `bridge.version` | `0.24.0` | The ollama version to report. |
|
|
| `bridge.reported-context-offset` | `0` | Tokens to subtract from the downstream context length before reporting it to clients. |
|
|
|
|
## API Endpoints
|
|
|
|
### Discovery API (Ollama Compatible)
|
|
|
|
- `GET /api/version`: Returns the bridge version.
|
|
- `GET /api/tags`: Returns a list of available models.
|
|
- `POST /api/show`: Returns detailed information about a specific model.
|
|
|
|
### Proxying
|
|
|
|
- All other requests are proxied to the `bridge.upstream-base-url`.
|
|
|
|
## Getting Started
|
|
|
|
### License
|
|
|
|
This project is licensed under the Apache License 2.0. See the `LICENSE` file for details.
|
|
|
|
### Prerequisites
|
|
|
|
- Java 25
|
|
- Maven
|
|
|
|
### Building and Running
|
|
|
|
1. **Clone the repository**:
|
|
```bash
|
|
git clone <repository-url>
|
|
cd ollama-oai-bridge
|
|
```
|
|
|
|
2. **Build the project**:
|
|
```bash
|
|
./mvnw clean package
|
|
```
|
|
|
|
3. **Run the application**:
|
|
You can run it using Maven:
|
|
```bash
|
|
./mvnw spring-boot:run
|
|
```
|
|
|
|
You can also override any of the bridge settings with JVM system properties, for example:
|
|
```bash
|
|
java -Dserver.port=11435 -Dbridge.upstream-base-url=http://localhost:5002 -Dbridge.version=0.24.1 -Dbridge.reported-context-offset=5000 -jar target/ollama-oai-bridge.jar
|
|
```
|
|
|
|
If you use VS Code GitHub Copilot BYOK, configure the Ollama provider to point at the bridge endpoint and keep the upstream server behind it.
|
|
|
|
Or, after building, you can use the provided `start.sh` script. Copy the generated `.jar` file from the `target/` directory and `start.sh` into the same directory, then run:
|
|
```bash
|
|
./start.sh
|
|
```
|
|
|
|
The `start.sh` launcher also accepts JVM `-D` options directly, so this works too:
|
|
```bash
|
|
./start.sh -Dserver.port=11435 -Dbridge.upstream-base-url=http://localhost:5002 -Dbridge.version=0.24.1 -Dbridge.reported-context-offset=5000
|
|
```
|
|
|
|
### Docker
|
|
|
|
You can build and run the application as a Docker image:
|
|
|
|
```bash
|
|
cd docker
|
|
./build.sh
|
|
./start-docker.sh
|
|
```
|
|
|
|
The `build.sh` script builds a multi-stage Docker image that compiles the JAR inside Docker and packages it into a minimal runtime image. The `start-docker.sh` script runs the container with host networking.
|
|
|
|
## Architecture
|
|
|
|
- **`GatewayProxyConfiguration`**: Sets up the Spring Cloud Gateway routes to proxy requests.
|
|
- **`LlamaServerClient`**: Handles low-level communication with the upstream server and normalizes model data.
|
|
- **`OllamaDiscoveryService`**: Implements the logic for translating upstream data into Ollama-compatible JSON responses.
|
|
- **`OllamaDiscoveryController`**: Exposes the discovery endpoints.
|
|
- **`RequestBodyCachingFilter`**: Ensures request bodies can be read multiple times during the request lifecycle.
|