M1+M2: RAG-Pipeline mit verbindlichem Grounding
agent/-Paket: Ingest (601 Layer-2-Eintraege -> 3005 Chunks, FTS5-BM25 + Vektoren-Cache), Hybrid-Retrieval (RRF, Stand-Boost, cross_ref-Erweiterung), Ollama-Client (embed/chat, think-Flag-Fallback, kurzes Connect-Budget), Systemprompt mit Zitierpflicht, Post-Validierung (zitierte IDs gemaess Retrieved-Set, 1x Regenerierung, dann Verweigerung), FastAPI (/ask, /health, /reindex), CLI, Goldset (31 Fragen, IDs gegen kb.json verifiziert, inkl. ATZ-Konfliktfall + 4 Verweigerungsfaelle), Eval-Suite, Test-Chat. 41 Offline-Tests gruen. Baseline BM25-only: Hit-Rate 0,871 / Recall@8 0,855 / MRR 0,476. Hybrid-Messung, Antwortmodus-Eval und Modell-Bake-off (M3) auf dem Host ausstaendig (Ollama aus der Zed-Sandbox nicht erreichbar). MEMORY.md und planung.md Umsetzungsstand aktualisiert.
This commit is contained in:
+114
@@ -0,0 +1,114 @@
|
||||
# PV RAG Agent
|
||||
|
||||
Lokaler RAG-Agent für österreichische Personalverrechnung: beantwortet
|
||||
Fragen **ausschließlich** aus der kuratierten Wissensbasis (Layer 2,
|
||||
`wissensbasis/`, 601 Einträge) — mit ID- und Stand-Beleg, ohne
|
||||
Trainingswissen, ohne Web-Zugriff. Verbindliche Regeln:
|
||||
`.agents/skills/pv-rag-agent/SKILL.md`, Plan: `planung.md`.
|
||||
|
||||
## Architektur (Kurzfassung)
|
||||
|
||||
```
|
||||
wissensbasis/dokumente/*.md ──ingest──▶ data/index.db
|
||||
├─ chunks (FTS5, BM25, Umlaut-Folding)
|
||||
├─ vectors (bge-m3, Content-Hash-Cache)
|
||||
└─ Metadaten (stand, topic, tags, …)
|
||||
Frage ──retrieve──▶ Hybrid BM25+Dense (RRF) + cross_ref-Erweiterung
|
||||
──generate──▶ Ollama (Systemprompt, Zitierpflicht)
|
||||
──validate──▶ zitierte IDs ⊆ Retrieved-Set? sonst 1× regenerieren, dann verweigern
|
||||
```
|
||||
|
||||
- **Nur Layer 2** als Korpus (kuratiert, lizenzkonform). Layer-1-Volltexte
|
||||
(`.lexis360/`, `.wiku/`) bleiben außen vor (offener Lizenzpunkt).
|
||||
- **Kein Ausweg nach außen:** keine Tools, kein Browsing — der einzige
|
||||
HTTP-Client spricht mit Ollama.
|
||||
- **Verweigerungspflicht:** leeres/schwaches Retrieval → deterministische
|
||||
Antwort „Dazu enthält die Wissensbasis keine Aussage." (kein LLM-Call).
|
||||
|
||||
## Schnellstart
|
||||
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
|
||||
# 1) Index bauen (mit Embeddings, wenn Ollama erreichbar)
|
||||
python -m agent.cli ingest # --no-embed erzwingt BM25-only
|
||||
|
||||
# 2) Frage im Terminal
|
||||
python -m agent.cli ask "Wie hoch ist die AMS-Ersatzquote bei geblockter Altersteilzeit?"
|
||||
|
||||
# 3) Goldset-Evaluation (offline: Retrieval-Metriken)
|
||||
python -m agent.cli eval
|
||||
# inkl. Antworten + Verweigerungsfälle (benötigt Ollama):
|
||||
python -m agent.cli eval --answers --json-out data/eval-report.json
|
||||
|
||||
# 4) HTTP-API + Test-Chat
|
||||
python -m agent.cli serve # http://127.0.0.1:8080 (/ask, /health, /reindex)
|
||||
```
|
||||
|
||||
## Konfiguration (Umgebungsvariablen)
|
||||
|
||||
| Variable | Default | Bedeutung |
|
||||
|---|---|---|
|
||||
| `OLLAMA_URL` | `http://100.183.83.12:11435` | Ollama-Server (Custom-Port!) |
|
||||
| `PV_ANSWER_MODEL` | `qwen3.8:27b` | Antwortmodell (provisorisch bis Bake-off M3) |
|
||||
| `PV_EMBED_MODEL` | `bge-m3` | Embedding-Modell |
|
||||
| `PV_DB_PATH` | `data/index.db` | SQLite-Index |
|
||||
| `PV_KB_DIR` | `wissensbasis` | Wissensbasis-Verzeichnis |
|
||||
| `PV_THINK` | `false` | Thinking per Request (qwen3.8: default an) |
|
||||
| `PV_EMBED_OFF` | `false` | `true` = BM25-only |
|
||||
| `PV_CONTEXT_BLOCKS` | `8` | Kontextblöcke im Prompt |
|
||||
| `PV_PORT` | `8080` | API-Port |
|
||||
|
||||
## Deployment auf dem Host (Ollama-Maschine)
|
||||
|
||||
```bash
|
||||
# Modelle einmalig pullen
|
||||
ollama pull qwen3.8:27b
|
||||
ollama pull bge-m3
|
||||
curl http://100.183.83.12:11435/api/tags # Erreichbarkeit + Modelle
|
||||
|
||||
# Vollständiger Index (BM25 + Dense) — danach eval, Ziel: Recall@8 > 0,9
|
||||
python -m agent.cli ingest
|
||||
python -m agent.cli eval
|
||||
python -m agent.cli eval --answers # Zitier-Präzision, Verweigerungen, Latenz
|
||||
```
|
||||
|
||||
## Baseline (2026-09-14, BM25-only, ohne Dense)
|
||||
|
||||
Goldset (31 Fragen, `agent/eval/goldset.yaml`): Hit-Rate 0,871 ·
|
||||
Recall@8 0,855 · MRR 0,476. Die vier Fehltreffer sind klassische
|
||||
BM25-Schwächen (Komposita: „aliquotiert"↔„Aliquotierung";
|
||||
„Mindestlohngesetz") — genau die Fälle, die die Dense-Suche abdecken soll.
|
||||
Hybrid-Messung auf dem Host aussteht (Ollama aus der Zed-Sandbox nicht
|
||||
erreichbar).
|
||||
|
||||
## Dateien
|
||||
|
||||
```
|
||||
agent/
|
||||
config.py Env-Konfiguration
|
||||
kb.py Layer-2-Parsing + kb.json-Gate
|
||||
normalize.py Umlaut-Folding, FTS-Query-Bau
|
||||
ingest.py Index-Bau (chunks + FTS5 + Vektoren-Cache)
|
||||
retrieve.py Hybrid-Retrieval (BM25 + Dense, RRF, cross_refs)
|
||||
ollama_client.py Ollama-HTTP (embed + chat, think-Fallback)
|
||||
generate.py Systemprompt, Post-Validierung, Verweigerung
|
||||
api.py FastAPI (/ask, /health, /reindex)
|
||||
cli.py ingest | ask | eval | serve
|
||||
eval/ goldset.yaml + evaluate.py
|
||||
web/index.html Minimaler Test-Chat
|
||||
tests/ 41 Tests (offline, Fake-Ollama)
|
||||
data/ index.db (gitignored)
|
||||
```
|
||||
|
||||
## Tests
|
||||
|
||||
```bash
|
||||
python -m pytest -q # 41 Tests, alle offline
|
||||
```
|
||||
|
||||
## Lizenz-Disziplin
|
||||
|
||||
`.lexis360/`, `.wiku/`, `.firecrawl/`, `.ris/` sind lokal und unversioniert
|
||||
(`.gitignore`). Der Index enthält ausschließlich Layer-2-Kuratierung;
|
||||
Layer-1-Prompts wären ein Lizenzverstoß und sind im Code nicht vorgesehen.
|
||||
@@ -0,0 +1,8 @@
|
||||
"""PV RAG Agent — Wissensbasis-Copilot für österreichische Personalverrechnung.
|
||||
|
||||
Schlanke RAG-Pipeline über die kuratierte Wissensbasis (Layer 2):
|
||||
Ingest -> SQLite-Index (FTS5-BM25 + Dense-Vektoren) -> Hybrid-Retrieval ->
|
||||
Ollama-Generierung mit verbindlichen Grounding-Regeln.
|
||||
|
||||
Verbindliche Regeln: siehe .agents/skills/pv-rag-agent/SKILL.md und planung.md.
|
||||
"""
|
||||
+156
@@ -0,0 +1,156 @@
|
||||
"""FastAPI-Oberfläche des PV RAG Agent.
|
||||
|
||||
Endpunkte:
|
||||
POST /ask — Frage -> belegte Antwort (oder Verweigerung)
|
||||
GET /health — Index- und Ollama-Status
|
||||
POST /reindex — Index-Neuaufbau (nach neuem Wissensbasis-Batch)
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import threading
|
||||
from contextlib import asynccontextmanager
|
||||
|
||||
from fastapi import FastAPI, HTTPException
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from .config import Config
|
||||
from .generate import answer_question
|
||||
from .ingest import build_index
|
||||
from .ollama_client import OllamaClient
|
||||
from .retrieve import Retriever
|
||||
|
||||
|
||||
class AskRequest(BaseModel):
|
||||
question: str = Field(min_length=3, max_length=2000)
|
||||
top_k: int | None = Field(default=None, ge=1, le=20)
|
||||
|
||||
|
||||
class SourceOut(BaseModel):
|
||||
id: str
|
||||
title: str
|
||||
section: str | None = None
|
||||
stand: str | None = None
|
||||
work: str | None = None
|
||||
|
||||
|
||||
class AskResponse(BaseModel):
|
||||
question: str
|
||||
answer: str
|
||||
refused: bool
|
||||
verified: bool
|
||||
citations: list[str]
|
||||
sources: list[SourceOut]
|
||||
n_context: int
|
||||
model: str
|
||||
latency_ms: int
|
||||
regenerations: int = 0
|
||||
|
||||
|
||||
class AppState:
|
||||
def __init__(self) -> None:
|
||||
self.cfg: Config | None = None
|
||||
self.client: OllamaClient | None = None
|
||||
self.retriever: Retriever | None = None
|
||||
self.lock = threading.Lock()
|
||||
|
||||
def ensure(self) -> Config:
|
||||
if self.cfg is None:
|
||||
self.cfg = Config.from_env()
|
||||
return self.cfg
|
||||
|
||||
def get_client(self) -> OllamaClient:
|
||||
if self.client is None:
|
||||
cfg = self.ensure()
|
||||
self.client = OllamaClient(
|
||||
cfg.ollama_url,
|
||||
embed_timeout_s=cfg.embed_timeout_s,
|
||||
chat_timeout_s=cfg.chat_timeout_s,
|
||||
)
|
||||
return self.client
|
||||
|
||||
def get_retriever(self) -> Retriever:
|
||||
if self.retriever is None:
|
||||
cfg = self.ensure()
|
||||
self.retriever = Retriever(cfg)
|
||||
return self.retriever
|
||||
|
||||
def reset_retriever(self) -> None:
|
||||
if self.retriever is not None:
|
||||
self.retriever.close()
|
||||
self.retriever = None
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI):
|
||||
app.state.rag = AppState()
|
||||
yield
|
||||
rag: AppState = app.state.rag
|
||||
if rag.retriever:
|
||||
rag.retriever.close()
|
||||
if rag.client:
|
||||
rag.client.close()
|
||||
|
||||
|
||||
app = FastAPI(title="PV RAG Agent", version="0.1.0", lifespan=lifespan)
|
||||
|
||||
|
||||
@app.post("/ask", response_model=AskResponse)
|
||||
def ask(req: AskRequest) -> AskResponse:
|
||||
rag: AppState = app.state.rag
|
||||
cfg = rag.ensure()
|
||||
try:
|
||||
result = answer_question(
|
||||
req.question, cfg,
|
||||
client=rag.get_client(),
|
||||
retriever=rag.get_retriever(),
|
||||
top_k=req.top_k,
|
||||
)
|
||||
except RuntimeError as e: # Index fehlt
|
||||
raise HTTPException(status_code=503, detail=str(e)) from e
|
||||
except Exception as e: # Ollama nicht erreichbar o. Ä.
|
||||
raise HTTPException(
|
||||
status_code=503,
|
||||
detail=f"Antwortgenerierung fehlgeschlagen: {type(e).__name__}: {e}",
|
||||
) from e
|
||||
result.pop("draft", None)
|
||||
return AskResponse(**result)
|
||||
|
||||
|
||||
@app.get("/health")
|
||||
def health() -> dict:
|
||||
rag: AppState = app.state.rag
|
||||
cfg = rag.ensure()
|
||||
out: dict = {"service": "pv-rag-agent", "config": {
|
||||
"ollama_url": cfg.ollama_url,
|
||||
"answer_model": cfg.answer_model,
|
||||
"embed_model": cfg.embed_model,
|
||||
}}
|
||||
try:
|
||||
retriever = rag.get_retriever()
|
||||
out["index"] = retriever.stats()
|
||||
except RuntimeError as e:
|
||||
out["index"] = {"error": str(e)}
|
||||
client = rag.get_client()
|
||||
out["ollama_up"] = client.is_up()
|
||||
if out["ollama_up"]:
|
||||
try:
|
||||
out["ollama_models"] = client.list_models()
|
||||
except Exception:
|
||||
out["ollama_models"] = None
|
||||
return out
|
||||
|
||||
|
||||
@app.post("/reindex")
|
||||
def reindex() -> dict:
|
||||
rag: AppState = app.state.rag
|
||||
cfg = rag.ensure()
|
||||
with rag.lock:
|
||||
stats = build_index(cfg, client=rag.get_client())
|
||||
rag.reset_retriever()
|
||||
result = stats.as_dict()
|
||||
result["warning"] = (
|
||||
"Index ohne Dense-Vektoren aufgebaut (Ollama-Embedding nicht verfügbar) — "
|
||||
"BM25-only. 'ollama pull " + cfg.embed_model + "' prüfen und erneut reindexen."
|
||||
if stats.embed_error else None
|
||||
)
|
||||
return result
|
||||
+124
@@ -0,0 +1,124 @@
|
||||
"""CLI des PV RAG Agent.
|
||||
|
||||
python -m agent.cli ingest [--no-embed] Index (neu) aufbauen
|
||||
python -m agent.cli ask "Frage?" [--top-k N] [--json]
|
||||
python -m agent.cli eval [--answers] [--limit N] [--k 8] [--json-out FILE]
|
||||
python -m agent.cli serve [--host 0.0.0.0]
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import sys
|
||||
|
||||
from .config import Config
|
||||
|
||||
|
||||
def _cmd_ingest(args: argparse.Namespace, cfg: Config) -> int:
|
||||
import dataclasses
|
||||
|
||||
from .ingest import build_index
|
||||
client = None
|
||||
if args.no_embed:
|
||||
cfg = dataclasses.replace(cfg, embed_off=True)
|
||||
elif not cfg.embed_off:
|
||||
from .ollama_client import OllamaClient
|
||||
client = OllamaClient(
|
||||
cfg.ollama_url,
|
||||
embed_timeout_s=cfg.embed_timeout_s,
|
||||
chat_timeout_s=cfg.chat_timeout_s,
|
||||
)
|
||||
if not client.is_up():
|
||||
print(
|
||||
f"[warn] Ollama unter {cfg.ollama_url} nicht erreichbar — "
|
||||
"Index wird BM25-only aufgebaut.",
|
||||
file=sys.stderr,
|
||||
)
|
||||
try:
|
||||
stats = build_index(cfg, client=client)
|
||||
finally:
|
||||
if client is not None:
|
||||
client.close()
|
||||
print(json.dumps(stats.as_dict(), indent=2, ensure_ascii=False))
|
||||
if stats.embed_error:
|
||||
print(f"[warn] {stats.embed_error}", file=sys.stderr)
|
||||
return 0
|
||||
|
||||
|
||||
def _cmd_ask(args: argparse.Namespace, cfg: Config) -> int:
|
||||
from .generate import answer_question
|
||||
try:
|
||||
result = answer_question(args.question, cfg, top_k=args.top_k)
|
||||
except Exception as e:
|
||||
print(
|
||||
f"[Fehler] Antwortgenerierung fehlgeschlagen: "
|
||||
f"{type(e).__name__}: {e}\n"
|
||||
f"Ollama erreichbar unter {cfg.ollama_url}? 'curl {cfg.ollama_url}/api/tags'",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return 2
|
||||
if args.json:
|
||||
print(json.dumps(result, indent=2, ensure_ascii=False))
|
||||
return 0
|
||||
print(result["answer"])
|
||||
if result["sources"]:
|
||||
print("\nQuellen:")
|
||||
for s in result["sources"]:
|
||||
print(f" - {s['id']} — {s['title']} ({s['stand']})")
|
||||
status = "VERWEIGERT" if result["refused"] else (
|
||||
"OK" if result["verified"] else "UNVERIFIZIERT"
|
||||
)
|
||||
print(
|
||||
f"\n[{status} · {result['model']} · {result['latency_ms']} ms · "
|
||||
f"{result['n_context']} Kontextblöcke]"
|
||||
)
|
||||
return 0
|
||||
|
||||
|
||||
def _cmd_eval(args: argparse.Namespace, cfg: Config) -> int:
|
||||
from .eval.evaluate import run_eval
|
||||
return run_eval(cfg, args)
|
||||
|
||||
|
||||
def _cmd_serve(args: argparse.Namespace, cfg: Config) -> int:
|
||||
import uvicorn
|
||||
uvicorn.run("agent.api:app", host=args.host, port=cfg.port, log_level="info")
|
||||
return 0
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
cfg = Config.from_env()
|
||||
parser = argparse.ArgumentParser(prog="agent.cli", description=__doc__)
|
||||
sub = parser.add_subparsers(dest="cmd", required=True)
|
||||
|
||||
p_ingest = sub.add_parser("ingest", help="Index (neu) aufbauen")
|
||||
p_ingest.add_argument("--no-embed", action="store_true",
|
||||
help="Keine Embeddings erzeugen (BM25-only)")
|
||||
|
||||
p_ask = sub.add_parser("ask", help="Frage stellen")
|
||||
p_ask.add_argument("question")
|
||||
p_ask.add_argument("--top-k", type=int, default=None)
|
||||
p_ask.add_argument("--json", action="store_true")
|
||||
|
||||
p_eval = sub.add_parser("eval", help="Goldset-Evaluation")
|
||||
p_eval.add_argument("--answers", action="store_true",
|
||||
help="inkl. Antwortgenerierung (benötigt Ollama)")
|
||||
p_eval.add_argument("--limit", type=int, default=None)
|
||||
p_eval.add_argument("--k", type=int, default=8, help="K für Recall@k")
|
||||
p_eval.add_argument("--json-out", default=None)
|
||||
|
||||
p_serve = sub.add_parser("serve", help="HTTP-API starten")
|
||||
p_serve.add_argument("--host", default="127.0.0.1")
|
||||
|
||||
args = parser.parse_args(argv)
|
||||
handlers = {
|
||||
"ingest": _cmd_ingest,
|
||||
"ask": _cmd_ask,
|
||||
"eval": _cmd_eval,
|
||||
"serve": _cmd_serve,
|
||||
}
|
||||
return handlers[args.cmd](args, cfg)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,87 @@
|
||||
"""Konfiguration des PV RAG Agent (alle Werte per Umgebungsvariable übersteuerbar)."""
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from dataclasses import dataclass
|
||||
|
||||
|
||||
def _env_str(name: str, default: str) -> str:
|
||||
v = os.environ.get(name)
|
||||
return v if v not in (None, "") else default
|
||||
|
||||
|
||||
def _env_int(name: str, default: int) -> int:
|
||||
try:
|
||||
return int(os.environ.get(name, default))
|
||||
except (TypeError, ValueError):
|
||||
return default
|
||||
|
||||
|
||||
def _env_float(name: str, default: float) -> float:
|
||||
try:
|
||||
return float(os.environ.get(name, default))
|
||||
except (TypeError, ValueError):
|
||||
return default
|
||||
|
||||
|
||||
def _env_bool(name: str, default: bool) -> bool:
|
||||
v = os.environ.get(name)
|
||||
if v is None:
|
||||
return default
|
||||
return v.strip().lower() in ("1", "true", "yes", "on")
|
||||
|
||||
|
||||
@dataclass
|
||||
class Config:
|
||||
# Pfade (relativ zum Repo-Root, sofern nicht absolut)
|
||||
kb_dir: str = "wissensbasis"
|
||||
db_path: str = "data/index.db"
|
||||
|
||||
# Ollama (Custom-Port 11435 — nicht "korrigieren", s. Skill)
|
||||
ollama_url: str = "http://100.183.83.12:11435"
|
||||
embed_model: str = "bge-m3"
|
||||
answer_model: str = "qwen3.8:27b" # provisorisch bis Bake-off (M3)
|
||||
|
||||
# Generierung
|
||||
temperature: float = 0.1
|
||||
num_ctx: int = 16384
|
||||
num_predict: int = 1024
|
||||
think: bool = False # Thinking per Request abschalten (Latenz)
|
||||
chat_timeout_s: float = 300.0
|
||||
embed_timeout_s: float = 240.0
|
||||
|
||||
# Retrieval
|
||||
embed_off: bool = False # True = BM25-only (ohne Dense-Index/-Suche)
|
||||
candidate_pool: int = 50 # Kandidaten je Liste vor der Fusion
|
||||
context_blocks: int = 8 # Kontext-Blöcke im Prompt
|
||||
cross_ref_expand: int = 3 # Top-Einträge, deren cross_refs ergänzt werden
|
||||
cross_ref_max_extra: int = 6 # Obergrenze der Ergänzungen
|
||||
rrf_k: int = 60
|
||||
recency_boost: float = 0.005 # additiv auf RRF-Score, gewichtet nach Stand
|
||||
|
||||
# Service
|
||||
port: int = 8080
|
||||
|
||||
@classmethod
|
||||
def from_env(cls) -> "Config":
|
||||
d = cls()
|
||||
return cls(
|
||||
kb_dir=_env_str("PV_KB_DIR", d.kb_dir),
|
||||
db_path=_env_str("PV_DB_PATH", d.db_path),
|
||||
ollama_url=_env_str("OLLAMA_URL", d.ollama_url),
|
||||
embed_model=_env_str("PV_EMBED_MODEL", d.embed_model),
|
||||
answer_model=_env_str("PV_ANSWER_MODEL", d.answer_model),
|
||||
temperature=_env_float("PV_TEMPERATURE", d.temperature),
|
||||
num_ctx=_env_int("PV_NUM_CTX", d.num_ctx),
|
||||
num_predict=_env_int("PV_NUM_PREDICT", d.num_predict),
|
||||
think=_env_bool("PV_THINK", d.think),
|
||||
chat_timeout_s=_env_float("PV_CHAT_TIMEOUT_S", d.chat_timeout_s),
|
||||
embed_timeout_s=_env_float("PV_EMBED_TIMEOUT_S", d.embed_timeout_s),
|
||||
embed_off=_env_bool("PV_EMBED_OFF", d.embed_off),
|
||||
candidate_pool=_env_int("PV_CANDIDATE_POOL", d.candidate_pool),
|
||||
context_blocks=_env_int("PV_CONTEXT_BLOCKS", d.context_blocks),
|
||||
cross_ref_expand=_env_int("PV_CROSS_REF_EXPAND", d.cross_ref_expand),
|
||||
cross_ref_max_extra=_env_int("PV_CROSS_REF_MAX_EXTRA", d.cross_ref_max_extra),
|
||||
recency_boost=_env_float("PV_RECENCY_BOOST", d.recency_boost),
|
||||
port=_env_int("PV_PORT", d.port),
|
||||
)
|
||||
@@ -0,0 +1 @@
|
||||
"""Eval-Paket: Goldset + Metriken für Retrieval und belegte Antworten."""
|
||||
@@ -0,0 +1,178 @@
|
||||
"""Goldset-Evaluation.
|
||||
|
||||
Modus 1 (offline, ohne Ollama): Retrieval-Metriken — Recall@k, Hit-Rate, MRR.
|
||||
Modus 2 (--answers, benötigt Ollama): Zitier-Präzision (validated),
|
||||
Verweigerungskorrektheit, erwartete Quelle zitiert, Latenz.
|
||||
|
||||
Kriterien laut Skill: Recall@8 > 0,9; Zitier-Präzision 100 %;
|
||||
Verweigerungen korrekt.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import statistics
|
||||
from pathlib import Path
|
||||
|
||||
import yaml
|
||||
|
||||
from ..config import Config
|
||||
from ..retrieve import Retriever
|
||||
|
||||
GOLDSET_PATH = Path(__file__).parent / "goldset.yaml"
|
||||
|
||||
|
||||
def load_goldset(path: Path = GOLDSET_PATH) -> list[dict]:
|
||||
data = yaml.safe_load(Path(path).read_text(encoding="utf-8"))
|
||||
questions = data.get("questions", [])
|
||||
if not questions:
|
||||
raise SystemExit(f"goldset leer: {path}")
|
||||
return questions
|
||||
|
||||
|
||||
def evaluate_retrieval(cfg: Config, questions: list[dict], k: int = 8) -> list[dict]:
|
||||
retriever = Retriever(cfg)
|
||||
rows: list[dict] = []
|
||||
try:
|
||||
for q in questions:
|
||||
if q.get("expect_refusal"):
|
||||
continue # Verweigerungsfälle werden nur im Antwortmodus gemessen
|
||||
expected = set(q.get("expected_ids", []))
|
||||
if not expected:
|
||||
continue
|
||||
results = retriever.search(q["question"], n_entries=k)
|
||||
retrieved = [r.entry_id for r in results]
|
||||
hits = expected & set(retrieved)
|
||||
rank = next(
|
||||
(retrieved.index(e) + 1 for e in expected if e in retrieved), None
|
||||
)
|
||||
rows.append(
|
||||
{
|
||||
"id": q["id"],
|
||||
"question": q["question"],
|
||||
"expected": sorted(expected),
|
||||
"retrieved": retrieved,
|
||||
"recall": len(hits) / len(expected),
|
||||
"hit": bool(hits),
|
||||
"mrr": (1.0 / rank) if rank else 0.0,
|
||||
}
|
||||
)
|
||||
finally:
|
||||
retriever.close()
|
||||
return rows
|
||||
|
||||
|
||||
def summarize_retrieval(rows: list[dict]) -> dict:
|
||||
if not rows:
|
||||
return {"n": 0}
|
||||
return {
|
||||
"n": len(rows),
|
||||
"hit_rate": round(sum(r["hit"] for r in rows) / len(rows), 4),
|
||||
"mean_recall_at_k": round(statistics.mean(r["recall"] for r in rows), 4),
|
||||
"mrr": round(statistics.mean(r["mrr"] for r in rows), 4),
|
||||
}
|
||||
|
||||
|
||||
def evaluate_answers(cfg: Config, questions: list[dict]) -> list[dict]:
|
||||
from ..generate import answer_question # Ollama nötig — hier erst importieren
|
||||
rows: list[dict] = []
|
||||
for q in questions:
|
||||
result = answer_question(q["question"], cfg)
|
||||
expected = set(q.get("expected_ids", []))
|
||||
should_refuse = bool(q.get("expect_refusal"))
|
||||
rows.append(
|
||||
{
|
||||
"id": q["id"],
|
||||
"question": q["question"],
|
||||
"refused": result["refused"],
|
||||
"should_refuse": should_refuse,
|
||||
"refusal_correct": result["refused"] == should_refuse,
|
||||
"verified": result["verified"],
|
||||
"citations": result["citations"],
|
||||
"expected_cited": (
|
||||
any(c in expected for c in result["citations"]) if expected else None
|
||||
),
|
||||
"regenerations": result["regenerations"],
|
||||
"latency_ms": result["latency_ms"],
|
||||
}
|
||||
)
|
||||
return rows
|
||||
|
||||
|
||||
def summarize_answers(rows: list[dict]) -> dict:
|
||||
if not rows:
|
||||
return {"n": 0}
|
||||
lat = [r["latency_ms"] for r in rows]
|
||||
return {
|
||||
"n": len(rows),
|
||||
"citation_precision": round(
|
||||
sum(r["verified"] for r in rows) / len(rows), 4
|
||||
),
|
||||
"refusal_correct_rate": round(
|
||||
sum(r["refusal_correct"] for r in rows) / len(rows), 4
|
||||
),
|
||||
"expected_cited_rate": round(
|
||||
sum(bool(r["expected_cited"]) for r in rows)
|
||||
/ max(1, sum(1 for r in rows if r["expected_cited"] is not None)),
|
||||
4,
|
||||
),
|
||||
"latency_ms_mean": round(statistics.mean(lat)),
|
||||
"latency_ms_p95": round(sorted(lat)[int(len(lat) * 0.95) - 1]),
|
||||
"regenerations_total": sum(r["regenerations"] for r in rows),
|
||||
}
|
||||
|
||||
|
||||
def run_eval(cfg: Config, args) -> int:
|
||||
questions = load_goldset()
|
||||
if args.limit:
|
||||
questions = questions[: args.limit]
|
||||
mode = "retrieval + Antworten" if args.answers else "nur Retrieval (offline)"
|
||||
print(f"Goldset: {len(questions)} Fragen · Modus: {mode} · k={args.k}")
|
||||
print(f"Antwortmodell: {cfg.answer_model} · Embedding: {cfg.embed_model}")
|
||||
print("-" * 78)
|
||||
|
||||
report: dict = {"mode": mode, "k": args.k, "model": cfg.answer_model}
|
||||
|
||||
rows = evaluate_retrieval(cfg, questions, k=args.k)
|
||||
for r in rows:
|
||||
mark = "✓" if r["hit"] else "✗"
|
||||
print(
|
||||
f"{mark} {r['id']:8s} recall={r['recall']:.2f} "
|
||||
f"mrr={r['mrr']:.2f} {r['question'][:56]}"
|
||||
)
|
||||
if not r["hit"]:
|
||||
print(f" erwartet: {', '.join(r['expected'])}")
|
||||
print(f" erhalten: {', '.join(r['retrieved'][:args.k])}")
|
||||
summary = summarize_retrieval(rows)
|
||||
report["retrieval"] = {"summary": summary, "rows": rows}
|
||||
print("-" * 78)
|
||||
print(f"Retrieval: n={summary['n']} Hit-Rate={summary['hit_rate']} "
|
||||
f"Recall@{args.k}={summary['mean_recall_at_k']} MRR={summary['mrr']}")
|
||||
|
||||
if args.answers:
|
||||
arows = evaluate_answers(cfg, questions)
|
||||
for r in arows:
|
||||
mark = "✓" if (r["verified"] and r["refusal_correct"]) else "✗"
|
||||
print(
|
||||
f"{mark} {r['id']:8s} refused={r['refused']} verified={r['verified']} "
|
||||
f"zit={','.join(r['citations'][:4])} {r['latency_ms']}ms"
|
||||
)
|
||||
asummary = summarize_answers(arows)
|
||||
report["answers"] = {"summary": asummary, "rows": arows}
|
||||
print("-" * 78)
|
||||
print(
|
||||
f"Antworten: n={asummary['n']} Zitier-Präzision={asummary['citation_precision']} "
|
||||
f"Verweigerung korrekt={asummary['refusal_correct_rate']} "
|
||||
f"erwartete Quelle zitiert={asummary['expected_cited_rate']}"
|
||||
)
|
||||
print(
|
||||
f"Latenz: mean={asummary['latency_ms_mean']}ms p95={asummary['latency_ms_p95']}ms "
|
||||
f"Regenerierungen={asummary['regenerations_total']}"
|
||||
)
|
||||
|
||||
if args.json_out:
|
||||
Path(args.json_out).parent.mkdir(parents=True, exist_ok=True)
|
||||
Path(args.json_out).write_text(
|
||||
json.dumps(report, indent=2, ensure_ascii=False), encoding="utf-8"
|
||||
)
|
||||
print(f"Report geschrieben: {args.json_out}")
|
||||
return 0
|
||||
@@ -0,0 +1,116 @@
|
||||
# Goldset für die Evaluation des PV RAG Agent.
|
||||
# expected_ids: verifiziert gegen wissensbasis/kb.json (Titel-Suche).
|
||||
# expect_refusal: Frage darf/darf nicht aus der Wissensbasis beantwortet
|
||||
# werden — Antwortmodus (eval --answers) muss verweigern.
|
||||
# q-002 ist der bekannte Korpuskonflikt (ATZ-Ersatzquote 28,5 vs. 27,5 %).
|
||||
|
||||
questions:
|
||||
- id: q-001
|
||||
question: "Was ist Altersteilzeit und wie funktioniert der Lohnausgleich?"
|
||||
expected_ids: [lb-atz-07, lb-atz-04]
|
||||
- id: q-002
|
||||
question: "Wie hoch ist die AMS-Ersatzquote bei geblockter Altersteilzeit?"
|
||||
expected_ids: [lb-atz-07, lb-atz-09, lb-atz-12]
|
||||
note: "Korpuskonflikt — beide Werte mit ⚠ nennen."
|
||||
- id: q-003
|
||||
question: "Für welche Arbeitnehmer ist die Altersteilzeit förderbar?"
|
||||
expected_ids: [lb-atz-03]
|
||||
- id: q-004
|
||||
question: "Wie viele Wochen gesetzlicher Urlaub stehen einem Arbeitnehmer zu?"
|
||||
expected_ids: [lb-url-05]
|
||||
- id: q-005
|
||||
question: "Wie wird das Urlaubsentgelt beim Wechsel zwischen Teilzeit und Vollzeit berechnet?"
|
||||
expected_ids: [lb-url-13]
|
||||
- id: q-006
|
||||
question: "Wie lange dauert die Entgeltfortzahlung im Krankenstand?"
|
||||
expected_ids: [lb-krs-08]
|
||||
- id: q-007
|
||||
question: "Welche Fristen gelten für die Auflösung in der Probezeit?"
|
||||
expected_ids: [lb-bnd-37]
|
||||
- id: q-008
|
||||
question: "Welche Verfügungsmöglichkeiten gibt es bei der Abfertigung neu?"
|
||||
expected_ids: [lb-end-03]
|
||||
- id: q-009
|
||||
question: "Welche Bezugsbestandteile sind beitragsfrei nach § 49 Abs. 3 ASVG?"
|
||||
expected_ids: [lb-sva-03, lb-sva-04]
|
||||
- id: q-010
|
||||
question: "Wie hoch sind die Sozialversicherungs-Beitragssätze für Dienstnehmer?"
|
||||
expected_ids: [lb-sva-06]
|
||||
- id: q-011
|
||||
question: "Was gilt als Nachtschwerarbeit und welche Folgen hat das?"
|
||||
expected_ids: [lb-nsc-01]
|
||||
- id: q-012
|
||||
question: "Wie sind Mitarbeiterrabatte abgabenrechtlich zu behandeln?"
|
||||
expected_ids: [lb-sac-02]
|
||||
- id: q-013
|
||||
question: "Wie wird die Privatnutzung eines Dienstwagens besteuert?"
|
||||
expected_ids: [lb-sac-03]
|
||||
- id: q-014
|
||||
question: "Unter welchen Voraussetzungen gibt es Pendlerförderung?"
|
||||
expected_ids: [lb-pen-01]
|
||||
- id: q-015
|
||||
question: "Wie werden Tagesgelder bei Dienstreisen abgerechnet?"
|
||||
expected_ids: [lb-rei-09]
|
||||
- id: q-016
|
||||
question: "Wie läuft eine GPLB ab?"
|
||||
expected_ids: [lb-gpl-01]
|
||||
- id: q-017
|
||||
question: "Was ist bei der Entsendung von Arbeitnehmern ins Ausland zu beachten?"
|
||||
expected_ids: [lb-grz-06, lb-grz-07]
|
||||
- id: q-018
|
||||
question: "Wie funktioniert die betriebliche Vorsorgekasse mit Beitragszahlung?"
|
||||
expected_ids: [lb-vor-02, lb-vor-03]
|
||||
- id: q-019
|
||||
question: "Welche Pflichten gelten bei der Einstellung von Lehrlingen?"
|
||||
expected_ids: [lb-leh-03]
|
||||
- id: q-020
|
||||
question: "Welche Beschäftigungsverbote gelten für Schwangere?"
|
||||
expected_ids: [lb-sch-04]
|
||||
- id: q-021
|
||||
question: "Wann beginnt die Elternkarenz und wie lange kann sie dauern?"
|
||||
expected_ids: [lb-kar-01]
|
||||
- id: q-022
|
||||
question: "Wie ist eine Überstundenpauschale zu behandeln?"
|
||||
expected_ids: [lb-ues-02]
|
||||
- id: q-023
|
||||
question: "Was muss ein Dienstzeugnis enthalten?"
|
||||
expected_ids: [lb-bso-04]
|
||||
- id: q-024
|
||||
question: "Was ist bei Kurzarbeit arbeitsrechtlich zu beachten?"
|
||||
expected_ids: [lb-azm-04, lb-azm-05, lb-azm-06]
|
||||
- id: q-025
|
||||
question: "Was passiert mit den Ansprüchen der Arbeitnehmer beim Betriebsübergang?"
|
||||
expected_ids: [lb-ins-01]
|
||||
- id: q-026
|
||||
question: "Wie sind Nachzahlungen abzurechnen?"
|
||||
expected_ids: [lb-naz-01]
|
||||
- id: q-027
|
||||
question: "Was ist beim Einlangen einer Lohnpfändung zu beachten?"
|
||||
expected_ids: [lb-pfa-01, lb-pfa-02]
|
||||
- id: q-028
|
||||
question: "Wird die Sonderzahlung bei Eintritt oder Austritt aliquotiert?"
|
||||
expected_ids: [lb-son-01]
|
||||
- id: q-029
|
||||
question: "Welche Neuerungen behandelt WIKU Personal aktuell 2026?"
|
||||
expected_ids: [wk-akt-01]
|
||||
- id: q-030
|
||||
question: "Wer zahlt das Wochengeld und wie wird es berechnet?"
|
||||
expected_ids: [lb-msf-03]
|
||||
- id: q-031
|
||||
question: "Was regelt das Mindestlohngesetz und für wen gilt es?"
|
||||
expected_ids: [lb-ent-09]
|
||||
|
||||
# --- Verweigerungsfälle (Antwortmodus) ---
|
||||
- id: r-001
|
||||
question: "Wie hoch ist der aktuelle EUR-USD-Wechselkurs?"
|
||||
expect_refusal: true
|
||||
- id: r-002
|
||||
question: "Wer war Bundeskanzler Österreichs im Jahr 2000?"
|
||||
expect_refusal: true
|
||||
- id: r-003
|
||||
question: "Bis wann muss die Umsatzsteuervoranmeldung abgegeben werden?"
|
||||
expect_refusal: true
|
||||
note: "Schwerer Fall: 13 Chunks erwähnen 'Umsatzsteuer' beiläufig — Retrieval nicht leer, aber inhaltlich nicht gedeckt."
|
||||
- id: r-004
|
||||
question: "Wie bereite ich einen Pitch für Investoren vor?"
|
||||
expect_refusal: true
|
||||
@@ -0,0 +1,218 @@
|
||||
"""Antwort-Generierung mit verbindlichen Grounding-Regeln.
|
||||
|
||||
Kern der Pipeline: Systemprompt (nur Kontext, Zitierpflicht, Verweigerung),
|
||||
kontrollierte cross_ref-Erweiterung und Post-Validierung — jede zitierte ID
|
||||
muss im Retrieved-Set stehen, sonst eine Regenerierung, dann Verweigerung.
|
||||
Keine Antwort verlässt die Pipeline mit ungültigen Zitaten.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
import time
|
||||
|
||||
from .config import Config
|
||||
from .normalize import normalize_text
|
||||
from .ollama_client import OllamaClient
|
||||
from .retrieve import ChunkResult, Retriever
|
||||
|
||||
REFUSAL_MESSAGE = "Dazu enthält die Wissensbasis keine Aussage."
|
||||
UNCERTAIN_MESSAGE = (
|
||||
"⚠ Zu dieser Frage kann ich keine verlässlich belegte Antwort "
|
||||
"aus der Wissensbasis geben."
|
||||
)
|
||||
|
||||
SYSTEM_PROMPT = """Du bist ein präziser Assistent für österreichische Personalverrechnung.
|
||||
Du beantwortest Fragen AUSSCHLIESSLICH auf Basis der nummerierten Kontextblöcke
|
||||
aus der internen Wissensbasis.
|
||||
|
||||
Verbindliche Regeln:
|
||||
1. Jede fachliche Aussage muss durch die Kontextblöcke gedeckt sein. Verwende
|
||||
KEIN Wissen aus deinem Training und ergänze nichts aus eigenem Wissen.
|
||||
2. Belege jede fachliche Aussage mit der KB-ID in eckigen Klammern, z. B.
|
||||
[lb-atz-07]. Zitiere NUR die IDs aus den Block-Köpfen („Block N — [id] …“).
|
||||
IDs, die nur im Fließtext als Verweis genannt werden, sind Querverweise
|
||||
und KEINE Belege.
|
||||
3. Gib jeden Wert mit seinem Stand an, z. B. „28,5 % (Stand 2026-01)“.
|
||||
4. Beantworten die Kontextblöcke die Frage nicht, antworte exakt:
|
||||
„Dazu enthält die Wissensbasis keine Aussage.“ — und schlage nichts vor.
|
||||
5. Widersprechen sich Kontextblöcke, nenne beide Werte mit ihren IDs und
|
||||
kennzeichne den Widerspruch mit ⚠. Löse Widersprüche niemals stillschweigend auf.
|
||||
6. Nenne Paragraphen und Gesetze nur, wenn ein Kontextblock sie nennt.
|
||||
7. Antworte auf Deutsch, prägnant (Stichpunkte, wo sinnvoll), und füge am
|
||||
Ende eine Zeile „Quellen:“ mit den verwendeten IDs (ID — Titel, Stand) an.
|
||||
|
||||
Verletze Regel 2 oder Regel 4 niemals — im Zweifel verweigere die Antwort."""
|
||||
|
||||
CITE_RE = re.compile(r"\b(?:lb|wk)-[a-z0-9]+-\d+\b")
|
||||
_THINK_RE = re.compile(r"<think>.*?</think>", re.DOTALL)
|
||||
|
||||
|
||||
def strip_think(text: str) -> str:
|
||||
"""Entfernt <think>-Blöcke defensiv (falls Thinking nicht abschaltbar war)."""
|
||||
return _THINK_RE.sub("", text).strip()
|
||||
|
||||
|
||||
def looks_like_refusal(answer: str) -> bool:
|
||||
folded = normalize_text(answer)
|
||||
return (
|
||||
"keine aussage" in folded
|
||||
or "keine verlasslich belegte" in folded
|
||||
or "nicht in der wissensbasis" in folded
|
||||
)
|
||||
|
||||
|
||||
def build_user_content(question: str, results: list[ChunkResult]) -> str:
|
||||
blocks = []
|
||||
for i, r in enumerate(results, 1):
|
||||
header = (
|
||||
f"Block {i} — [{r.entry_id}] {r.title} · Abschnitt: {r.section} "
|
||||
f"· Stand: {r.stand} · Werk: {r.work}"
|
||||
)
|
||||
blocks.append(f"{header}\n{r.text}")
|
||||
context = "\n\n---\n\n".join(blocks)
|
||||
return f"Kontextblöcke aus der Wissensbasis:\n\n{context}\n\nFrage: {question}"
|
||||
|
||||
|
||||
def validate_answer(answer: str, allowed_ids: list[str]) -> list[str]:
|
||||
"""Regel-2/4-Prüfung: zitierte IDs ⊆ Kontext; keine unbelegte Fachantwort."""
|
||||
cited = set(CITE_RE.findall(answer))
|
||||
violations: list[str] = []
|
||||
unknown = sorted(cited - set(allowed_ids))
|
||||
if unknown:
|
||||
violations.append(f"zitierte IDs außerhalb des Kontexts: {', '.join(unknown)}")
|
||||
if not cited and not looks_like_refusal(answer):
|
||||
violations.append("keine KB-ID zitiert")
|
||||
return violations
|
||||
|
||||
|
||||
def _source_rows(results: list[ChunkResult]) -> list[dict]:
|
||||
rows = []
|
||||
seen: set[str] = set()
|
||||
for r in results:
|
||||
if r.entry_id in seen:
|
||||
continue
|
||||
seen.add(r.entry_id)
|
||||
rows.append(
|
||||
{
|
||||
"id": r.entry_id,
|
||||
"title": r.title,
|
||||
"section": r.section,
|
||||
"stand": r.stand,
|
||||
"work": r.work,
|
||||
"source": r.source,
|
||||
}
|
||||
)
|
||||
return rows
|
||||
|
||||
|
||||
def answer_question(
|
||||
question: str,
|
||||
cfg: Config,
|
||||
client: OllamaClient | None = None,
|
||||
retriever: Retriever | None = None,
|
||||
top_k: int | None = None,
|
||||
) -> dict:
|
||||
"""Vollständiger Ask-Zyklus: Retrieval -> Prompt -> LLM -> Post-Validierung."""
|
||||
t0 = time.perf_counter()
|
||||
own_retriever = retriever is None
|
||||
if retriever is None:
|
||||
retriever = Retriever(cfg)
|
||||
try:
|
||||
results = retriever.search(question, n_entries=top_k)
|
||||
finally:
|
||||
if own_retriever:
|
||||
retriever.close()
|
||||
|
||||
def finish(answer, refused, verified, citations, regenerations=0, draft=None):
|
||||
return {
|
||||
"question": question,
|
||||
"answer": answer,
|
||||
"refused": refused,
|
||||
"verified": verified,
|
||||
"citations": citations,
|
||||
"sources": _source_rows(results),
|
||||
"n_context": len(results),
|
||||
"model": cfg.answer_model,
|
||||
"regenerations": regenerations,
|
||||
"latency_ms": round((time.perf_counter() - t0) * 1000),
|
||||
"draft": draft,
|
||||
}
|
||||
|
||||
if not results:
|
||||
# Verweigerungspflicht: leeres Retrieval -> deterministische Antwort
|
||||
return finish(REFUSAL_MESSAGE, refused=True, verified=True, citations=[])
|
||||
|
||||
allowed = [r.entry_id for r in results]
|
||||
by_id = {r.entry_id: r for r in results}
|
||||
|
||||
if client is None:
|
||||
client = OllamaClient(
|
||||
cfg.ollama_url,
|
||||
embed_timeout_s=cfg.embed_timeout_s,
|
||||
chat_timeout_s=cfg.chat_timeout_s,
|
||||
)
|
||||
messages = [
|
||||
{"role": "system", "content": SYSTEM_PROMPT},
|
||||
{"role": "user", "content": build_user_content(question, results)},
|
||||
]
|
||||
|
||||
def chat(msgs):
|
||||
return strip_think(
|
||||
client.chat(
|
||||
cfg.answer_model,
|
||||
msgs,
|
||||
temperature=cfg.temperature,
|
||||
num_ctx=cfg.num_ctx,
|
||||
num_predict=cfg.num_predict,
|
||||
think=cfg.think,
|
||||
)
|
||||
)
|
||||
|
||||
final = chat(messages)
|
||||
violations = validate_answer(final, allowed)
|
||||
regenerations = 0
|
||||
if violations:
|
||||
regenerations = 1
|
||||
warn = (
|
||||
"Deine letzte Antwort verstieß gegen die Regeln: "
|
||||
+ "; ".join(violations)
|
||||
+ f". Erlaubte KB-IDs sind ausschließlich: {', '.join(sorted(set(allowed)))}. "
|
||||
"Beantworte die Frage erneut und zitiere nur diese IDs — oder verweigere "
|
||||
f"mit dem vorgesehenen Satz („{REFUSAL_MESSAGE}“)."
|
||||
)
|
||||
retry = chat(
|
||||
messages
|
||||
+ [{"role": "assistant", "content": final},
|
||||
{"role": "user", "content": warn}]
|
||||
)
|
||||
retry_violations = validate_answer(retry, allowed)
|
||||
if not retry_violations:
|
||||
final = retry
|
||||
violations = []
|
||||
else:
|
||||
return finish(
|
||||
UNCERTAIN_MESSAGE,
|
||||
refused=True,
|
||||
verified=False,
|
||||
citations=[],
|
||||
regenerations=regenerations,
|
||||
draft=retry,
|
||||
)
|
||||
|
||||
citations = sorted(set(CITE_RE.findall(final)))
|
||||
refused = looks_like_refusal(final)
|
||||
sources = [
|
||||
{
|
||||
"id": cid,
|
||||
"title": by_id[cid].title,
|
||||
"section": by_id[cid].section,
|
||||
"stand": by_id[cid].stand,
|
||||
"work": by_id[cid].work,
|
||||
}
|
||||
for cid in citations
|
||||
if cid in by_id
|
||||
]
|
||||
return finish(
|
||||
final, refused=refused, verified=not violations,
|
||||
citations=citations, regenerations=regenerations,
|
||||
)
|
||||
+202
@@ -0,0 +1,202 @@
|
||||
"""Index-Bau: Layer-2-Einträge -> SQLite (FTS5-BM25 + Dense-Vektoren).
|
||||
|
||||
Chunking: H2-Sektion je Eintrag (Parent-Child: Retrieval auf Sektion,
|
||||
Kontext = Sektion + Metadatenkopf). Die Vektoren-Tabelle ist ein Cache
|
||||
(Content-Hash) und überlebt Rebuilds — ein Reindex bettet nur Neues ein.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import sqlite3
|
||||
import time
|
||||
from dataclasses import asdict, dataclass
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
|
||||
from .config import Config
|
||||
from .kb import KBEntry, Section, load_kb
|
||||
from .normalize import normalize_text
|
||||
|
||||
SCHEMA_VERSION = 1
|
||||
EMBED_BATCH = 32
|
||||
|
||||
SCHEMA = """
|
||||
CREATE TABLE IF NOT EXISTS meta (key TEXT PRIMARY KEY, value TEXT);
|
||||
CREATE TABLE IF NOT EXISTS chunks (
|
||||
chunk_id INTEGER PRIMARY KEY,
|
||||
entry_id TEXT NOT NULL,
|
||||
section TEXT NOT NULL,
|
||||
text TEXT NOT NULL,
|
||||
norm TEXT NOT NULL,
|
||||
content_hash TEXT NOT NULL,
|
||||
title TEXT, work TEXT, chapter TEXT, topic TEXT,
|
||||
stand TEXT, batch INTEGER,
|
||||
tags TEXT, legal_bases TEXT, cross_refs TEXT,
|
||||
source_pdf TEXT, source_text TEXT
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_chunks_entry ON chunks(entry_id);
|
||||
CREATE VIRTUAL TABLE IF NOT EXISTS chunks_fts USING fts5(norm);
|
||||
CREATE TABLE IF NOT EXISTS vectors (
|
||||
content_hash TEXT NOT NULL,
|
||||
model TEXT NOT NULL,
|
||||
dim INTEGER NOT NULL,
|
||||
vec BLOB NOT NULL,
|
||||
PRIMARY KEY (content_hash, model)
|
||||
);
|
||||
"""
|
||||
|
||||
|
||||
@dataclass
|
||||
class IndexStats:
|
||||
n_entries: int = 0
|
||||
n_chunks: int = 0
|
||||
n_embedded: int = 0
|
||||
embed_error: str | None = None
|
||||
duration_s: float = 0.0
|
||||
kb_dir: str = ""
|
||||
db_path: str = ""
|
||||
schema_version: int = SCHEMA_VERSION
|
||||
|
||||
def as_dict(self) -> dict:
|
||||
return asdict(self)
|
||||
|
||||
|
||||
def embed_text(title: str, section_title: str, text: str) -> str:
|
||||
"""Einheitlicher Embedding-Input (Title + Abschnitt + Body)."""
|
||||
return f"{title}\n{section_title}\n\n{text}"
|
||||
|
||||
|
||||
def content_hash_for(title: str, section_title: str, text: str) -> str:
|
||||
return hashlib.sha256(
|
||||
embed_text(title, section_title, text).encode("utf-8")
|
||||
).hexdigest()
|
||||
|
||||
|
||||
def norm_text_for(entry: KBEntry, section: Section) -> str:
|
||||
"""FTS-Text: Titel + Abschnitt + Tags + Rechtsgrundlagen + Kapitel + Body."""
|
||||
parts = [
|
||||
entry.title,
|
||||
section.title,
|
||||
" ".join(entry.tags),
|
||||
" ".join(entry.legal_bases),
|
||||
entry.chapter,
|
||||
section.text,
|
||||
]
|
||||
return normalize_text("\n".join(p for p in parts if p))
|
||||
|
||||
|
||||
def _insert_chunk(con: sqlite3.Connection, entry: KBEntry, section: Section) -> int:
|
||||
h = content_hash_for(entry.title, section.title, section.text)
|
||||
cur = con.execute(
|
||||
"""INSERT INTO chunks (
|
||||
entry_id, section, text, norm, content_hash,
|
||||
title, work, chapter, topic, stand, batch,
|
||||
tags, legal_bases, cross_refs, source_pdf, source_text
|
||||
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
|
||||
(
|
||||
entry.id, section.title, section.text, norm_text_for(entry, section), h,
|
||||
entry.title, entry.work, entry.chapter, entry.topic, entry.stand,
|
||||
entry.batch,
|
||||
json.dumps(entry.tags, ensure_ascii=False),
|
||||
json.dumps(entry.legal_bases, ensure_ascii=False),
|
||||
json.dumps(entry.cross_refs, ensure_ascii=False),
|
||||
entry.source.get("pdf", ""), entry.source.get("text", ""),
|
||||
),
|
||||
)
|
||||
chunk_id = cur.lastrowid
|
||||
con.execute(
|
||||
"INSERT INTO chunks_fts(rowid, norm) VALUES (?, ?)",
|
||||
(chunk_id, norm_text_for(entry, section)),
|
||||
)
|
||||
return chunk_id
|
||||
|
||||
|
||||
def _embed_missing(cfg: Config, con: sqlite3.Connection, client) -> tuple[int, str | None]:
|
||||
"""Bettet alle Chunk-Hashes ein, die für cfg.embed_model fehlen."""
|
||||
rows = con.execute(
|
||||
"SELECT DISTINCT content_hash, title, section, text FROM chunks"
|
||||
).fetchall()
|
||||
unique: dict[str, str] = {}
|
||||
for h, title, section, text in rows:
|
||||
if h not in unique:
|
||||
unique[h] = embed_text(title, section, text)
|
||||
have = {
|
||||
r[0] for r in con.execute(
|
||||
"SELECT content_hash FROM vectors WHERE model = ?",
|
||||
(cfg.embed_model,),
|
||||
).fetchall()
|
||||
}
|
||||
todo = [h for h in unique if h not in have]
|
||||
if not todo:
|
||||
return 0, None
|
||||
n = 0
|
||||
for i in range(0, len(todo), EMBED_BATCH):
|
||||
batch = todo[i:i + EMBED_BATCH]
|
||||
texts = [unique[h] for h in batch]
|
||||
try:
|
||||
embs = client.embed(cfg.embed_model, texts)
|
||||
except Exception as e: # httpx/Ollama-Fehler -> BM25-only weiterlaufen
|
||||
return n, f"embedding failed at batch {i // EMBED_BATCH + 1}: {e}"
|
||||
for h, vec in zip(batch, embs):
|
||||
arr = np.asarray(vec, dtype=np.float32)
|
||||
con.execute(
|
||||
"INSERT OR REPLACE INTO vectors(content_hash, model, dim, vec) "
|
||||
"VALUES (?, ?, ?, ?)",
|
||||
(h, cfg.embed_model, int(arr.shape[0]), arr.tobytes()),
|
||||
)
|
||||
n += 1
|
||||
return n, None
|
||||
|
||||
|
||||
def build_index(cfg: Config, client=None) -> IndexStats:
|
||||
"""Vollständiger Rebuild von chunks/FTS; Vektoren-Cache bleibt erhalten."""
|
||||
t0 = time.perf_counter()
|
||||
entries = load_kb(cfg.kb_dir, verify_registry=True)
|
||||
db_path = Path(cfg.db_path)
|
||||
db_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
stats = IndexStats(kb_dir=str(cfg.kb_dir), db_path=str(db_path))
|
||||
con = sqlite3.connect(db_path)
|
||||
try:
|
||||
con.executescript(SCHEMA)
|
||||
con.execute("DELETE FROM chunks")
|
||||
con.execute("DELETE FROM chunks_fts")
|
||||
con.execute("DELETE FROM meta")
|
||||
for entry in entries:
|
||||
for section in entry.sections:
|
||||
_insert_chunk(con, entry, section)
|
||||
con.commit()
|
||||
stats.n_entries = len(entries)
|
||||
stats.n_chunks = con.execute("SELECT COUNT(*) FROM chunks").fetchone()[0]
|
||||
|
||||
if not cfg.embed_off:
|
||||
if client is None:
|
||||
from .ollama_client import OllamaClient
|
||||
client = OllamaClient(
|
||||
cfg.ollama_url,
|
||||
embed_timeout_s=cfg.embed_timeout_s,
|
||||
chat_timeout_s=cfg.chat_timeout_s,
|
||||
)
|
||||
try:
|
||||
stats.n_embedded, stats.embed_error = _embed_missing(cfg, con, client)
|
||||
except Exception as e:
|
||||
stats.embed_error = f"{type(e).__name__}: {e}"
|
||||
|
||||
con.executemany(
|
||||
"INSERT OR REPLACE INTO meta(key, value) VALUES (?, ?)",
|
||||
[
|
||||
("schema_version", str(SCHEMA_VERSION)),
|
||||
("built_at", datetime.now(timezone.utc).isoformat()),
|
||||
("kb_dir", str(cfg.kb_dir)),
|
||||
("embed_model", "" if cfg.embed_off else cfg.embed_model),
|
||||
("n_entries", str(stats.n_entries)),
|
||||
("n_chunks", str(stats.n_chunks)),
|
||||
],
|
||||
)
|
||||
con.commit()
|
||||
finally:
|
||||
con.close()
|
||||
stats.duration_s = round(time.perf_counter() - t0, 2)
|
||||
return stats
|
||||
+164
@@ -0,0 +1,164 @@
|
||||
"""Layer-2-Wissensbasis: Parsing (Frontmatter + H2-Sektionen) und kb.json-Gate.
|
||||
|
||||
Die Layer-2-Frontmatter ist die Single Source of Truth; kb.json ist deren
|
||||
generierte, validierte Projektion. Der Gate bricht den Ingest bei Abweichung
|
||||
ab (Fehlermeldung nennt die Regenerierung der Registry als nächsten Schritt).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
|
||||
import yaml
|
||||
|
||||
ID_RE = re.compile(r"^(lb|wk)-[a-z0-9]+-\d+$")
|
||||
STAND_RE = re.compile(r"^\d{4}-\d{2}$")
|
||||
REQUIRED_KEYS = (
|
||||
"id", "batch", "title", "work", "chapter", "topic", "author",
|
||||
"stand", "source", "legal_bases", "tags", "cross_refs",
|
||||
)
|
||||
|
||||
|
||||
class KbValidationError(Exception):
|
||||
"""Wissensbasis oder Registry ist inkonsistent — Ingest wird abgebrochen."""
|
||||
|
||||
|
||||
@dataclass
|
||||
class Section:
|
||||
title: str
|
||||
text: str
|
||||
|
||||
|
||||
@dataclass
|
||||
class KBEntry:
|
||||
id: str
|
||||
title: str
|
||||
work: str
|
||||
chapter: str
|
||||
topic: str
|
||||
author: str
|
||||
stand: str
|
||||
batch: int
|
||||
source: dict
|
||||
legal_bases: list
|
||||
tags: list
|
||||
cross_refs: list
|
||||
path: Path
|
||||
sections: list = field(default_factory=list)
|
||||
|
||||
|
||||
def parse_frontmatter(raw: str) -> tuple[dict, str]:
|
||||
lines = raw.splitlines()
|
||||
if not lines or lines[0].strip() != "---":
|
||||
raise KbValidationError("missing frontmatter delimiter '---'")
|
||||
for i in range(1, len(lines)):
|
||||
if lines[i].strip() == "---":
|
||||
meta = yaml.safe_load("\n".join(lines[1:i]))
|
||||
body = "\n".join(lines[i + 1:])
|
||||
break
|
||||
else:
|
||||
raise KbValidationError("unterminated frontmatter")
|
||||
if not isinstance(meta, dict):
|
||||
raise KbValidationError("frontmatter is not a mapping")
|
||||
return meta, body
|
||||
|
||||
|
||||
def split_sections(body: str) -> list[Section]:
|
||||
"""H2-Sektionen als Chunks; H1-Titel und Quellzeile fallen weg."""
|
||||
sections: list[Section] = []
|
||||
current_title: str | None = None
|
||||
current: list[str] = []
|
||||
for line in body.splitlines():
|
||||
if line.startswith("## "):
|
||||
if current_title is not None:
|
||||
sections.append(Section(current_title, "\n".join(current).strip()))
|
||||
current_title = line[3:].strip()
|
||||
current = []
|
||||
elif line.startswith("# "):
|
||||
continue
|
||||
elif current_title is not None:
|
||||
current.append(line)
|
||||
if current_title is not None:
|
||||
sections.append(Section(current_title, "\n".join(current).strip()))
|
||||
return [s for s in sections if s.text]
|
||||
|
||||
|
||||
def load_entry(path: Path) -> KBEntry:
|
||||
meta, body = parse_frontmatter(path.read_text(encoding="utf-8"))
|
||||
missing = [k for k in REQUIRED_KEYS if k not in meta]
|
||||
if missing:
|
||||
raise KbValidationError(f"{path.name}: missing frontmatter keys {missing}")
|
||||
for key in ("id", "title", "work", "chapter", "topic", "author", "stand"):
|
||||
if not isinstance(meta[key], str) or not meta[key].strip():
|
||||
raise KbValidationError(f"{path.name}: empty '{key}'")
|
||||
if not ID_RE.match(meta["id"]):
|
||||
raise KbValidationError(f"{path.name}: invalid id '{meta['id']}'")
|
||||
if not STAND_RE.match(meta["stand"]):
|
||||
raise KbValidationError(f"{path.name}: stand '{meta['stand']}' not YYYY-MM")
|
||||
if not isinstance(meta["source"], dict) or not {"pdf", "text"} <= set(meta["source"]):
|
||||
raise KbValidationError(f"{path.name}: source needs pdf+text")
|
||||
for key in ("legal_bases", "tags", "cross_refs"):
|
||||
if not isinstance(meta[key], list):
|
||||
raise KbValidationError(f"{path.name}: '{key}' must be a list")
|
||||
entry = KBEntry(
|
||||
id=meta["id"],
|
||||
title=meta["title"],
|
||||
work=meta["work"],
|
||||
chapter=meta["chapter"],
|
||||
topic=meta["topic"],
|
||||
author=meta["author"],
|
||||
stand=meta["stand"],
|
||||
batch=int(meta["batch"]),
|
||||
source=dict(meta["source"]),
|
||||
legal_bases=list(meta["legal_bases"]),
|
||||
tags=list(meta["tags"]),
|
||||
cross_refs=list(meta["cross_refs"]),
|
||||
path=path,
|
||||
)
|
||||
entry.sections = split_sections(body)
|
||||
return entry
|
||||
|
||||
|
||||
def load_kb(kb_dir: str | Path, verify_registry: bool = True) -> list[KBEntry]:
|
||||
"""Lädt alle Layer-2-Einträge, prüft Cross-Ref-Integrität und das kb.json-Gate."""
|
||||
root = Path(kb_dir)
|
||||
doc_dir = root / "dokumente"
|
||||
paths = sorted(doc_dir.glob("*.md"))
|
||||
if not paths:
|
||||
raise KbValidationError(f"no Layer-2 documents under {doc_dir}")
|
||||
entries: dict[str, KBEntry] = {}
|
||||
for p in paths:
|
||||
e = load_entry(p)
|
||||
if e.id in entries:
|
||||
raise KbValidationError(
|
||||
f"duplicate id {e.id}: {entries[e.id].path.name} and {p.name}"
|
||||
)
|
||||
entries[e.id] = e
|
||||
for e in entries.values():
|
||||
dangling = [ref for ref in e.cross_refs if ref not in entries]
|
||||
if dangling:
|
||||
raise KbValidationError(f"{e.id}: dangling cross_refs {dangling}")
|
||||
if verify_registry:
|
||||
_registry_gate(root, entries)
|
||||
return list(entries.values())
|
||||
|
||||
|
||||
def _registry_gate(root: Path, entries: dict[str, KBEntry]) -> None:
|
||||
reg_path = root / "kb.json"
|
||||
if not reg_path.is_file():
|
||||
raise KbValidationError("kb.json missing — regenerate with --registry")
|
||||
registry = json.loads(reg_path.read_text(encoding="utf-8"))
|
||||
reg_ids = [e.get("id") for e in registry.get("entries", []) if e.get("id")]
|
||||
reg_set = set(reg_ids)
|
||||
kb_set = set(entries)
|
||||
if reg_set != kb_set:
|
||||
kb_only = sorted(kb_set - reg_set)[:5]
|
||||
reg_only = sorted(reg_set - kb_set)[:5]
|
||||
raise KbValidationError(
|
||||
"kb.json out of sync with Layer 2 "
|
||||
f"(docs-only: {kb_only}, registry-only: {reg_only}; "
|
||||
f"registry n={registry.get('n_entries')}, docs n={len(entries)}) — "
|
||||
"regenerate the registry first (build_lexis_kb.py --registry)"
|
||||
)
|
||||
@@ -0,0 +1,43 @@
|
||||
"""Deutsch-affine Textnormalisierung für FTS5-Index und -Anfragen.
|
||||
|
||||
Konvention (einmalig, konsistent): lowercase, Diakritika via NFKD entfernen
|
||||
(ä→a, ü→u), ß→ss. Dies gilt für die FTS-Spalte `norm` und die Query gleich.
|
||||
Die ASCII-Slug-Konvention der Wissensbasis (Umlaute "fallen") betrifft nur
|
||||
`topic`/`tags`/Dateinamen, nicht die Volltextsuche.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
import unicodedata
|
||||
|
||||
# Stopwörter in normalisierter Form (nach Fold), klein halten.
|
||||
STOPWORDS = frozenset(
|
||||
"""der die das und oder ein eine einen einem einer eines den dem des ist im in
|
||||
von fur mit auf zu zum zur an am beim wie was wann wird werden kann muss
|
||||
sind hat nicht man als auch aus bei sein ihre ihr es sie er doch noch nur
|
||||
schon sehr mehr hier da durch fuer wird""".split()
|
||||
)
|
||||
|
||||
_TOKEN_RE = re.compile(r"[a-z0-9]+")
|
||||
|
||||
|
||||
def normalize_text(s: str) -> str:
|
||||
"""Lowercase, Diakritika entfernen (NFKD), ß→ss."""
|
||||
s = unicodedata.normalize("NFKD", s.casefold())
|
||||
s = "".join(c for c in s if unicodedata.category(c) != "Mn")
|
||||
return s.replace("ß", "ss")
|
||||
|
||||
|
||||
def tokenize(s: str) -> list[str]:
|
||||
return _TOKEN_RE.findall(normalize_text(s))
|
||||
|
||||
|
||||
def fts_query(question: str, min_len: int = 2) -> str:
|
||||
"""OR-verknüpfte FTS5-Phrasen aus normalisierten Termen; '' wenn leer."""
|
||||
terms: list[str] = []
|
||||
seen: set[str] = set()
|
||||
for t in tokenize(question):
|
||||
if len(t) >= min_len and t not in STOPWORDS and t not in seen:
|
||||
seen.add(t)
|
||||
terms.append(t)
|
||||
return " OR ".join(f'"{t}"' for t in terms)
|
||||
@@ -0,0 +1,108 @@
|
||||
"""Schmaler Ollama-HTTP-Client (Chat + Embeddings).
|
||||
|
||||
Keine Tools, kein Browsing, keine Web-Hooks — der Agent hat architektonisch
|
||||
keinen Weg aus der Wissensbasis hinaus (Grounding-Regel 1).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
|
||||
|
||||
class OllamaError(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
class OllamaClient:
|
||||
def __init__(self, base_url: str, embed_timeout_s: float = 240.0,
|
||||
chat_timeout_s: float = 300.0):
|
||||
self.base_url = base_url.rstrip("/")
|
||||
self.embed_timeout_s = embed_timeout_s
|
||||
self.chat_timeout_s = chat_timeout_s
|
||||
# Per-Request-Timeouts mit kurzem Connect-Budget — ein unerreichbarer
|
||||
# Server muss in Sekunden, nicht Minuten scheitern.
|
||||
self.embed_timeout = httpx.Timeout(embed_timeout_s, connect=10.0)
|
||||
self.chat_timeout = httpx.Timeout(chat_timeout_s, connect=10.0)
|
||||
self._client = httpx.Client(timeout=self.chat_timeout)
|
||||
|
||||
def close(self) -> None:
|
||||
self._client.close()
|
||||
|
||||
# -- Infrastruktur ------------------------------------------------------
|
||||
|
||||
def is_up(self) -> bool:
|
||||
try:
|
||||
r = self._client.get(f"{self.base_url}/api/tags", timeout=5.0)
|
||||
return r.status_code == 200
|
||||
except httpx.HTTPError:
|
||||
return False
|
||||
|
||||
def list_models(self) -> list[str]:
|
||||
r = self._client.get(f"{self.base_url}/api/tags")
|
||||
r.raise_for_status()
|
||||
return [m.get("model", "") for m in r.json().get("models", [])]
|
||||
|
||||
# -- Embeddings ---------------------------------------------------------
|
||||
|
||||
def embed(self, model: str, texts: list[str]) -> list[list[float]]:
|
||||
"""Batch-Embeddings via /api/embed; Fallback auf Legacy /api/embeddings."""
|
||||
r = self._client.post(
|
||||
f"{self.base_url}/api/embed",
|
||||
json={"model": model, "input": texts},
|
||||
timeout=self.embed_timeout,
|
||||
)
|
||||
if r.status_code == 404:
|
||||
out: list[list[float]] = []
|
||||
for t in texts:
|
||||
rr = self._client.post(
|
||||
f"{self.base_url}/api/embeddings",
|
||||
json={"model": model, "prompt": t},
|
||||
timeout=self.embed_timeout,
|
||||
)
|
||||
rr.raise_for_status()
|
||||
emb = rr.json().get("embedding")
|
||||
if not emb:
|
||||
raise OllamaError("legacy /api/embeddings returned no embedding")
|
||||
out.append(emb)
|
||||
return out
|
||||
r.raise_for_status()
|
||||
emb = r.json().get("embeddings")
|
||||
if not isinstance(emb, list) or len(emb) != len(texts):
|
||||
raise OllamaError(f"unexpected /api/embed response for {len(texts)} inputs")
|
||||
return emb
|
||||
|
||||
# -- Chat ---------------------------------------------------------------
|
||||
|
||||
def chat(self, model: str, messages: list[dict],
|
||||
temperature: float = 0.1, num_ctx: int = 16384,
|
||||
num_predict: int = 1024, think: bool = False) -> str:
|
||||
"""POST /api/chat, stream=False; `think`-Flag mit 404/400-Fallback."""
|
||||
body: dict[str, Any] = {
|
||||
"model": model,
|
||||
"messages": messages,
|
||||
"stream": False,
|
||||
"options": {
|
||||
"temperature": temperature,
|
||||
"num_ctx": num_ctx,
|
||||
"num_predict": num_predict,
|
||||
},
|
||||
}
|
||||
if think is not None:
|
||||
body["think"] = bool(think)
|
||||
r = self._client.post(
|
||||
f"{self.base_url}/api/chat", json=body, timeout=self.chat_timeout
|
||||
)
|
||||
if r.status_code in (400, 404) and "think" in body:
|
||||
# Ältere Ollama-Versionen kennen das think-Flag nicht -> Retry ohne.
|
||||
body.pop("think")
|
||||
r = self._client.post(
|
||||
f"{self.base_url}/api/chat", json=body, timeout=self.chat_timeout
|
||||
)
|
||||
r.raise_for_status()
|
||||
data = r.json()
|
||||
msg = data.get("message") or {}
|
||||
content = msg.get("content") or ""
|
||||
if not content.strip():
|
||||
raise OllamaError(f"empty response from {model} (keys: {list(data.keys())})")
|
||||
return content
|
||||
@@ -0,0 +1,252 @@
|
||||
"""Hybrid-Retrieval: BM25 (FTS5) + Dense (bge-m3) -> RRF-Fusion,
|
||||
milde Stand-Aktualitätsgewichtung und kontrollierte cross_ref-Erweiterung.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import sqlite3
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
|
||||
from .config import Config
|
||||
from .normalize import fts_query
|
||||
from .ollama_client import OllamaClient
|
||||
|
||||
|
||||
@dataclass
|
||||
class ChunkResult:
|
||||
chunk_id: int
|
||||
entry_id: str
|
||||
section: str
|
||||
text: str
|
||||
title: str
|
||||
stand: str
|
||||
work: str
|
||||
chapter: str
|
||||
topic: str
|
||||
tags: list = field(default_factory=list)
|
||||
legal_bases: list = field(default_factory=list)
|
||||
cross_refs: list = field(default_factory=list)
|
||||
batch: int = 0
|
||||
score: float = 0.0
|
||||
source: str = "fused" # bm25 | dense | fused | cross_ref
|
||||
|
||||
|
||||
class Retriever:
|
||||
def __init__(self, cfg: Config, db_path: str | None = None, client=None):
|
||||
self.cfg = cfg
|
||||
self.db_path = str(db_path or cfg.db_path)
|
||||
if not Path(self.db_path).is_file():
|
||||
raise RuntimeError(
|
||||
f"index fehlt ({self.db_path}) — zuerst 'python -m agent.cli ingest' ausführen"
|
||||
)
|
||||
self._con = sqlite3.connect(self.db_path)
|
||||
self._con.row_factory = sqlite3.Row
|
||||
self._client = client
|
||||
self._owns_client = client is None
|
||||
self._mat: np.ndarray | None = None
|
||||
self._mat_chunk_ids: list[int] | None = None
|
||||
row = self._con.execute("SELECT MIN(stand), MAX(stand) FROM chunks").fetchone()
|
||||
self._stand_min = int((row[0] or "2026-01").replace("-", ""))
|
||||
self._stand_max = int((row[1] or "2026-01").replace("-", ""))
|
||||
|
||||
def close(self) -> None:
|
||||
self._con.close()
|
||||
if self._owns_client and self._client is not None:
|
||||
self._client.close()
|
||||
|
||||
# -- Index-Kennzahlen ---------------------------------------------------
|
||||
|
||||
def stats(self) -> dict:
|
||||
n_chunks = self._con.execute("SELECT COUNT(*) FROM chunks").fetchone()[0]
|
||||
n_entries = self._con.execute(
|
||||
"SELECT COUNT(DISTINCT entry_id) FROM chunks"
|
||||
).fetchone()[0]
|
||||
n_vec = self._con.execute(
|
||||
"SELECT COUNT(*) FROM vectors WHERE model = ?",
|
||||
(self.cfg.embed_model,),
|
||||
).fetchone()[0]
|
||||
meta = dict(self._con.execute("SELECT key, value FROM meta").fetchall())
|
||||
return {
|
||||
"n_entries": n_entries,
|
||||
"n_chunks": n_chunks,
|
||||
"n_vectors": n_vec,
|
||||
"dense_available": n_vec > 0 and not self.cfg.embed_off,
|
||||
"stand_min": str(self._stand_min),
|
||||
"stand_max": str(self._stand_max),
|
||||
"built_at": meta.get("built_at"),
|
||||
}
|
||||
|
||||
# -- Einzelverfahren ----------------------------------------------------
|
||||
|
||||
def _bm25(self, question: str, limit: int) -> dict[int, float]:
|
||||
q = fts_query(question)
|
||||
if not q:
|
||||
return {}
|
||||
rows = self._con.execute(
|
||||
"SELECT rowid, bm25(chunks_fts) AS rank FROM chunks_fts "
|
||||
"WHERE chunks_fts MATCH ? ORDER BY rank LIMIT ?",
|
||||
(q, limit),
|
||||
).fetchall()
|
||||
# bm25(): kleinere Werte = besser -> negieren für "größer = besser"
|
||||
return {r["rowid"]: -float(r["rank"]) for r in rows}
|
||||
|
||||
def _dense(self, question: str, limit: int) -> dict[int, float]:
|
||||
if self.cfg.embed_off:
|
||||
return {}
|
||||
self._ensure_matrix()
|
||||
if self._mat is None or len(self._mat) == 0:
|
||||
return {}
|
||||
if self._client is None:
|
||||
self._client = OllamaClient(
|
||||
self.cfg.ollama_url,
|
||||
embed_timeout_s=self.cfg.embed_timeout_s,
|
||||
chat_timeout_s=self.cfg.chat_timeout_s,
|
||||
)
|
||||
try:
|
||||
qvec = np.asarray(
|
||||
self._client.embed(self.cfg.embed_model, [question])[0],
|
||||
dtype=np.float32,
|
||||
)
|
||||
except Exception:
|
||||
return {} # Ollama nicht erreichbar -> BM25-only weiter
|
||||
qn = np.linalg.norm(qvec)
|
||||
if qn == 0:
|
||||
return {}
|
||||
sims = self._mat_norm @ (qvec / qn)
|
||||
order = np.argsort(-sims)[:limit]
|
||||
return {self._mat_chunk_ids[i]: float(sims[i]) for i in order}
|
||||
|
||||
def _ensure_matrix(self) -> None:
|
||||
if self._mat is not None:
|
||||
return
|
||||
rows = self._con.execute(
|
||||
"SELECT c.chunk_id, v.vec, v.dim FROM chunks c "
|
||||
"JOIN vectors v ON v.content_hash = c.content_hash AND v.model = ?",
|
||||
(self.cfg.embed_model,),
|
||||
).fetchall()
|
||||
if not rows:
|
||||
self._mat = np.zeros((0, 1), dtype=np.float32)
|
||||
self._mat_chunk_ids = []
|
||||
return
|
||||
ids = [r[0] for r in rows]
|
||||
mat = np.vstack(
|
||||
[np.frombuffer(r[1], dtype=np.float32) for r in rows]
|
||||
)
|
||||
norms = np.linalg.norm(mat, axis=1, keepdims=True)
|
||||
self._mat = mat
|
||||
self._mat_norm = mat / np.where(norms == 0, 1.0, norms)
|
||||
self._mat_chunk_ids = ids
|
||||
|
||||
# -- Metadaten & Fusion -------------------------------------------------
|
||||
|
||||
def _stand_factor(self, stand: str) -> float:
|
||||
if self._stand_max <= self._stand_min:
|
||||
return 0.0
|
||||
try:
|
||||
s = int(stand.replace("-", ""))
|
||||
except (ValueError, AttributeError):
|
||||
return 0.0
|
||||
f = (s - self._stand_min) / (self._stand_max - self._stand_min)
|
||||
return min(1.0, max(0.0, f))
|
||||
|
||||
def _fetch_chunks(self, chunk_ids: list[int]) -> dict[int, sqlite3.Row]:
|
||||
out: dict[int, sqlite3.Row] = {}
|
||||
for i in range(0, len(chunk_ids), 500):
|
||||
part = chunk_ids[i:i + 500]
|
||||
qm = ",".join("?" * len(part))
|
||||
for r in self._con.execute(
|
||||
f"SELECT * FROM chunks WHERE chunk_id IN ({qm})", part
|
||||
).fetchall():
|
||||
out[r["chunk_id"]] = r
|
||||
return out
|
||||
|
||||
def _row_to_result(self, row: sqlite3.Row, score: float, source: str) -> ChunkResult:
|
||||
return ChunkResult(
|
||||
chunk_id=row["chunk_id"],
|
||||
entry_id=row["entry_id"],
|
||||
section=row["section"],
|
||||
text=row["text"],
|
||||
title=row["title"],
|
||||
stand=row["stand"],
|
||||
work=row["work"],
|
||||
chapter=row["chapter"],
|
||||
topic=row["topic"],
|
||||
tags=json.loads(row["tags"]),
|
||||
legal_bases=json.loads(row["legal_bases"]),
|
||||
cross_refs=json.loads(row["cross_refs"]),
|
||||
batch=row["batch"],
|
||||
score=score,
|
||||
source=source,
|
||||
)
|
||||
|
||||
def _best_chunk_of_entry(self, entry_id: str) -> ChunkResult | None:
|
||||
rows = self._con.execute(
|
||||
"SELECT * FROM chunks WHERE entry_id = ? "
|
||||
"ORDER BY CASE WHEN section LIKE 'Zusammenfassung%' THEN 0 ELSE 1 END, "
|
||||
"chunk_id LIMIT 1",
|
||||
(entry_id,),
|
||||
).fetchall()
|
||||
if not rows:
|
||||
return None
|
||||
return self._row_to_result(rows[0], 0.0, "cross_ref")
|
||||
|
||||
# -- öffentliche Suche --------------------------------------------------
|
||||
|
||||
def search(self, question: str, n_entries: int | None = None) -> list[ChunkResult]:
|
||||
"""Liefert die Top-Kontextblöcke (Hauptretrieval + cross_ref-Erweiterung)."""
|
||||
n = n_entries or self.cfg.context_blocks
|
||||
pool = self.cfg.candidate_pool
|
||||
bm = self._bm25(question, pool)
|
||||
try:
|
||||
dn = self._dense(question, pool)
|
||||
except Exception:
|
||||
dn = {}
|
||||
fused: dict[int, float] = {}
|
||||
for ranking in (bm, dn):
|
||||
ordered = sorted(ranking.items(), key=lambda kv: -kv[1])
|
||||
for rank, (cid, _) in enumerate(ordered):
|
||||
fused[cid] = fused.get(cid, 0.0) + 1.0 / (self.cfg.rrf_k + rank)
|
||||
if not fused:
|
||||
return []
|
||||
|
||||
rows = self._fetch_chunks(list(fused))
|
||||
results: list[ChunkResult] = []
|
||||
for cid, score in fused.items():
|
||||
if cid not in rows:
|
||||
continue
|
||||
source = "fused" if (cid in bm and cid in dn) else (
|
||||
"bm25" if cid in bm else "dense"
|
||||
)
|
||||
score += self.cfg.recency_boost * self._stand_factor(rows[cid]["stand"])
|
||||
results.append(self._row_to_result(rows[cid], score, source))
|
||||
results.sort(key=lambda r: -r.score)
|
||||
|
||||
# Bester Chunk je Eintrag -> Kontext (Entry-Level-Dedup)
|
||||
main: list[ChunkResult] = []
|
||||
seen: set[str] = set()
|
||||
for r in results:
|
||||
if r.entry_id in seen:
|
||||
continue
|
||||
seen.add(r.entry_id)
|
||||
main.append(r)
|
||||
if len(main) >= n:
|
||||
break
|
||||
|
||||
# cross_ref-Erweiterung (kontrolliert, markiert, begrenzt)
|
||||
extra: list[ChunkResult] = []
|
||||
budget = self.cfg.cross_ref_max_extra
|
||||
for r in main[: self.cfg.cross_ref_expand]:
|
||||
for ref in r.cross_refs:
|
||||
if budget <= 0:
|
||||
break
|
||||
if ref in seen:
|
||||
continue
|
||||
er = self._best_chunk_of_entry(ref)
|
||||
if er is not None:
|
||||
extra.append(er)
|
||||
seen.add(ref)
|
||||
budget -= 1
|
||||
return main + extra
|
||||
Reference in New Issue
Block a user