odoo_ocr/.agents/skills/local-vlm-client/SKILL.md

name, description

name	description
local-vlm-client	Use this skill when implementing or modifying local vision-language model clients, prompts, or inference calls for the odoo_ocr project.

Local VLM Client Skill

Use this skill whenever you are writing model clients, prompts, or retry logic for local VLMs (Ollama or llama.cpp).

Supported Runtimes

The project must support two local inference backends through a common interface:

1. Ollama (src/odoo_ocr/clients/ollama_client.py)

   • Default endpoint: http://localhost:11434.
   • Uses the Ollama Chat API: POST /api/chat.
   • Multimodal models receive images as base64 data URLs.
   • Model names follow Ollama convention, e.g., qwen2.5-vl:7b.

2. llama.cpp server (src/odoo_ocr/clients/llama_cpp_client.py)

   • Default endpoint: http://localhost:8080.
   • Uses OpenAI-compatible POST /v1/chat/completions.
   • Image data as base64 inside the image_url object.
   • Model name is often local or empty for single-model servers.

Unified Interface

Both clients implement BaseVLMClient from src/odoo_ocr/models/base_vlm.py:

class BaseVLMClient(ABC):
    @abstractmethod
    async def complete(
        self,
        system_prompt: str,
        user_prompt: str,
        images: list[Image.Image],
        response_format: type[BaseModel] | None = None,
        temperature: float = 0.1,
        max_tokens: int = 4096,
    ) -> VLMResponse:
        ...

class VLMResponse(BaseModel):
    content: str
    model: str
    prompt_tokens: int | None
    completion_tokens: int | None
    duration_ms: int

Prompt Engineering Rules

1. Keep system prompts in prompts/ as plain text files and load them at runtime.
2. For extraction tasks, the system prompt must:
   • State that the output must be valid JSON only.
   • Provide the target Pydantic schema.
   • Include 1–2 examples of valid output.
   • Forbid markdown code fences and explanatory text.
3. The user prompt should include the task and any context (e.g., "Extract invoice fields from this image. Currency is EUR unless otherwise stated.").
4. For review tasks, the prompt must pass both the image and the JSON to compare.

JSON Enforcement

1. Set temperature low (0.0–0.2) for extraction/review.
2. Prefer structured output when the runtime supports it:
   • Ollama: use format="json" or format=<json_schema> if available.
   • llama.cpp server: use response_format={"type": "json_object"} or json_schema if the server build supports it.
3. Always wrap the model output with a JSON parser; if parsing fails, retry once with a stronger "JSON only" reminder.
4. Validate parsed JSON against the target Pydantic model and surface validation errors.

Retry & Caching

1. Retry transient HTTP errors with exponential backoff (max 3 retries, base delay 1s).
2. Cache successful responses keyed by SHA256 of (prompt + image bytes + model + temperature) to ~/.cache/odoo_ocr/llm_cache/.
3. Do not cache errors.
4. Log every request: model, endpoint, prompt hash, duration, token counts.

Image Encoding

1. Accept PIL.Image.Image inputs.
2. Convert to RGB before encoding.
3. Encode as JPEG or PNG base64 depending on content; prefer PNG if the image is already low-color or binarized.
4. Resize images before sending if dimensions exceed model limits, preserving aspect ratio.

Default Models

Use these defaults unless the task explicitly says otherwise:

Task	Default model
Classification	qwen2.5-vl:3b
OCR / extraction	qwen2.5-vl:7b
Handwriting OCR	glm-ocr or project-configured handwriting model
Review	qwen2.5-vl:7b

Testing

• Mock the HTTP client in unit tests; never call real models during CI.
• Provide a FakeVLMClient in tests that returns canned responses.
• Test prompt assembly and JSON parsing separately from model calls.