Add dockerized whisper.cpp inference server (Vulkan GPU, Tailscale)

- server/whisper-server: compose stack built from the pinned whisper.cpp
  v1.9.3 release, same as the Android JNI layer
- Vulkan GPU backend (AMD Radeon AI PRO R9700 / RADV) with transparent
  CPU fallback and NO_GPU override; GGML models auto-download on first
  start (MODEL env, default large-v3)
- API bound to the Tailscale interface only (100.103.83.12:8080) since
  whisper-server has no authentication; render-group GID passthrough for
  /dev/dri
- validated locally: image builds, entrypoint downloads tiny, POST
  /inference returns verbose_json with language + segments; GPU-less
  fallback confirmed
This commit is contained in:
2026-09-07 12:17:40 +02:00
parent 46b0f128bc
commit ef59fc0fb4
5 changed files with 204 additions and 0 deletions
+71
View File
@@ -0,0 +1,71 @@
# whisper.cpp server (Docker Compose)
The transcription backend for meetrec clients: the same pinned whisper.cpp
release (v1.9.3) as the Android app, exposed as an HTTP inference API with
**Vulkan GPU support** (AMD Radeon AI PRO R9700) and reachable over
Tailscale at `100.103.83.12:8080`.
The Fairphone 6 transcribes at roughly 0.60.8× realtime on-device; the
R9700 (Strix Halo, RDNA 3.5, ~256 GB/s shared memory) is bandwidth-bound
friendly for Whisper — expect large-v3 at many times realtime.
## Start
```sh
RENDER_GID=$(getent group render | cut -d: -f3) docker compose up -d --build
docker compose logs -f # watch the model download, then "running"
```
The `RENDER_GID` lookup passes the host's render group into the container
so the GPU device is accessible. Verify the GPU is actually used from the
startup log — it should print `ggml_vulkan: Found 1 Vulkan devices` (and
`VULKAN = 1` in the system info); if the GPU is unavailable the server
transparently falls back to CPU.
Configuration lives in `docker-compose.yml`:
| Env | Default | Meaning |
| ---------- | --------- | ------------------------------------------ |
| `MODEL` | `large-v3`| `tiny`/`base`/`small`/`medium`/`large-v3` (downloaded to `./models` on first start) |
| `THREADS` | `8` | CPU threads per inference |
| `PORT` | `8080` | Port inside the container |
## Try it
```sh
curl http://100.103.83.12:8080/inference \
-F file=@meeting.wav \
-F response_format=verbose_json \
-F language=auto
```
`POST /inference` accepts multipart fields `file` (PCM WAV, any rate),
`language` (`auto` supported), `response_format`
(`text`/`json`/`srt`/`vtt`/`verbose_json`), `temperature`. With
`verbose_json` the response carries the detected language and segments
with `start`/`end`/`text` (seconds).
Notes:
- **Beam size is a server-start setting** (v1.9.3 has no per-request
override), so the live/final beam split of the clients doesn't apply
here — one beam for all requests.
- The server also has a `/load` endpoint to swap models at runtime.
- **No authentication**: the compose file binds `100.103.83.12:8080`
(Tailscale interface only) for that reason. Do not switch this to
`0.0.0.0` unless the host is otherwise firewalled.
- The image builds with CPU feature auto-detection (`-march=native`):
build it on the machine that runs it (`--build` from the server, not
by exporting an image from another host).
- **GPU backend**: Vulkan via the RADV driver (mesa-vulkan-drivers in the
image). For maximum performance a ROCm/HIP build is the alternative
(heavier image, needs a ROCm base image and `gfx1151` target support
for Strix Halo) — add later if Vulkan benchmarks are insufficient.
- `NO_GPU=1` in the environment forces CPU-only inference.
## Client status
- Desktop `meetrec`: a `whisper-server` engine is planned
(`--engine whisper-server --server-url http://100.103.83.12:8080`).
- Android app: a remote engine option is planned (phone records, server
transcribes; local JNI stays as the offline fallback).