Initial scaffold: local invoice OCR pipeline with Ollama, classifier branches, structured extraction, review, and Odoo XML export
- Add AGENTS.md and project-specific Zed skills (odoo-ocr-pipeline, odoo-xml-import, local-vlm-client). - Implement Pydantic schemas for documents, invoices, review results, and VLM responses. - Add unified BaseVLMClient with Ollama implementation and llama.cpp stub. - Build pipeline stages: loader, classifier, digital_pdf/scanned_print/handwritten/mixed_unknown branches, extractor, reviewer, xml_builder. - Add CLI entry point with sidecar JSON and confidence-gated XML output. - Include prompts for classifier, OCR, extraction, and review models. - Add tests with FakeVLMClient; pytest, ruff, and mypy all pass.
This commit is contained in:
@@ -0,0 +1,101 @@
|
||||
---
|
||||
name: local-vlm-client
|
||||
description: Use this skill when implementing or modifying local vision-language model clients, prompts, or inference calls for the odoo_ocr project.
|
||||
---
|
||||
|
||||
# Local VLM Client Skill
|
||||
|
||||
Use this skill whenever you are writing model clients, prompts, or retry logic for local VLMs (Ollama or llama.cpp).
|
||||
|
||||
## Supported Runtimes
|
||||
|
||||
The project must support two local inference backends through a common interface:
|
||||
|
||||
1. **Ollama** (`src/odoo_ocr/clients/ollama_client.py`)
|
||||
- Default endpoint: `http://localhost:11434`.
|
||||
- Uses the Ollama Chat API: `POST /api/chat`.
|
||||
- Multimodal models receive images as base64 data URLs.
|
||||
- Model names follow Ollama convention, e.g., `qwen2.5-vl:7b`.
|
||||
|
||||
2. **llama.cpp server** (`src/odoo_ocr/clients/llama_cpp_client.py`)
|
||||
- Default endpoint: `http://localhost:8080`.
|
||||
- Uses OpenAI-compatible `POST /v1/chat/completions`.
|
||||
- Image data as base64 inside the `image_url` object.
|
||||
- Model name is often `local` or empty for single-model servers.
|
||||
|
||||
## Unified Interface
|
||||
|
||||
Both clients implement `BaseVLMClient` from `src/odoo_ocr/models/base_vlm.py`:
|
||||
|
||||
```python
|
||||
class BaseVLMClient(ABC):
|
||||
@abstractmethod
|
||||
async def complete(
|
||||
self,
|
||||
system_prompt: str,
|
||||
user_prompt: str,
|
||||
images: list[Image.Image],
|
||||
response_format: type[BaseModel] | None = None,
|
||||
temperature: float = 0.1,
|
||||
max_tokens: int = 4096,
|
||||
) -> VLMResponse:
|
||||
...
|
||||
|
||||
class VLMResponse(BaseModel):
|
||||
content: str
|
||||
model: str
|
||||
prompt_tokens: int | None
|
||||
completion_tokens: int | None
|
||||
duration_ms: int
|
||||
```
|
||||
|
||||
## Prompt Engineering Rules
|
||||
|
||||
1. Keep system prompts in `prompts/` as plain text files and load them at runtime.
|
||||
2. For extraction tasks, the system prompt must:
|
||||
- State that the output must be valid JSON only.
|
||||
- Provide the target Pydantic schema.
|
||||
- Include 1–2 examples of valid output.
|
||||
- Forbid markdown code fences and explanatory text.
|
||||
3. The user prompt should include the task and any context (e.g., "Extract invoice fields from this image. Currency is EUR unless otherwise stated.").
|
||||
4. For review tasks, the prompt must pass both the image and the JSON to compare.
|
||||
|
||||
## JSON Enforcement
|
||||
|
||||
1. Set `temperature` low (0.0–0.2) for extraction/review.
|
||||
2. Prefer structured output when the runtime supports it:
|
||||
- Ollama: use `format="json"` or `format=<json_schema>` if available.
|
||||
- llama.cpp server: use `response_format={"type": "json_object"}` or `json_schema` if the server build supports it.
|
||||
3. Always wrap the model output with a JSON parser; if parsing fails, retry once with a stronger "JSON only" reminder.
|
||||
4. Validate parsed JSON against the target Pydantic model and surface validation errors.
|
||||
|
||||
## Retry & Caching
|
||||
|
||||
1. Retry transient HTTP errors with exponential backoff (max 3 retries, base delay 1s).
|
||||
2. Cache successful responses keyed by SHA256 of (prompt + image bytes + model + temperature) to `~/.cache/odoo_ocr/llm_cache/`.
|
||||
3. Do not cache errors.
|
||||
4. Log every request: model, endpoint, prompt hash, duration, token counts.
|
||||
|
||||
## Image Encoding
|
||||
|
||||
1. Accept `PIL.Image.Image` inputs.
|
||||
2. Convert to RGB before encoding.
|
||||
3. Encode as JPEG or PNG base64 depending on content; prefer PNG if the image is already low-color or binarized.
|
||||
4. Resize images before sending if dimensions exceed model limits, preserving aspect ratio.
|
||||
|
||||
## Default Models
|
||||
|
||||
Use these defaults unless the task explicitly says otherwise:
|
||||
|
||||
| Task | Default model |
|
||||
|------|---------------|
|
||||
| Classification | `qwen2.5-vl:3b` |
|
||||
| OCR / extraction | `qwen2.5-vl:7b` |
|
||||
| Handwriting OCR | `glm-ocr` or project-configured handwriting model |
|
||||
| Review | `qwen2.5-vl:7b` |
|
||||
|
||||
## Testing
|
||||
|
||||
- Mock the HTTP client in unit tests; never call real models during CI.
|
||||
- Provide a `FakeVLMClient` in tests that returns canned responses.
|
||||
- Test prompt assembly and JSON parsing separately from model calls.
|
||||
@@ -0,0 +1,36 @@
|
||||
You are an invoice data extraction assistant. Given an invoice image and its OCR text, produce a structured JSON representation of the invoice.
|
||||
|
||||
Output must match this Pydantic schema exactly and contain no markdown, no commentary:
|
||||
|
||||
{
|
||||
"vendor_name": "string",
|
||||
"vendor_address": "string or null",
|
||||
"vendor_vat": "string or null",
|
||||
"invoice_number": "string",
|
||||
"invoice_date": "YYYY-MM-DD",
|
||||
"due_date": "YYYY-MM-DD or null",
|
||||
"currency": "ISO 4217 code, e.g. EUR",
|
||||
"payment_terms": "string or null",
|
||||
"line_items": [
|
||||
{
|
||||
"description": "string",
|
||||
"quantity": 1.0,
|
||||
"unit_price": 0.00,
|
||||
"total_price": 0.00,
|
||||
"tax_rate": 0.00
|
||||
}
|
||||
],
|
||||
"subtotal": 0.00,
|
||||
"tax_total": 0.00,
|
||||
"total": 0.00,
|
||||
"iban": "string or null",
|
||||
"raw_ocr_text": "string"
|
||||
}
|
||||
|
||||
Rules:
|
||||
- Use null for missing optional fields.
|
||||
- Dates must be ISO 8601.
|
||||
- All monetary values are decimal numbers (do not use strings).
|
||||
- line_items total_price should equal quantity * unit_price (within rounding).
|
||||
- subtotal + tax_total should equal total (within rounding).
|
||||
- If the image and OCR disagree, trust the image.
|
||||
@@ -0,0 +1,5 @@
|
||||
You are an OCR engine. Read all text from the provided invoice image accurately.
|
||||
Preserve line breaks and table structure as much as possible.
|
||||
If a value is unclear, mark it with [UNCLEAR].
|
||||
Respond with a JSON object containing a single field "text" with the full OCR output.
|
||||
Do not add markdown formatting or explanations.
|
||||
@@ -0,0 +1,25 @@
|
||||
You are a meticulous invoice review assistant. You are given the original invoice image and a JSON object representing the extracted invoice data.
|
||||
|
||||
Your task:
|
||||
1. Verify that every field in the JSON is supported by the image.
|
||||
2. Check arithmetic: sum of line item totals plus tax should equal the invoice total.
|
||||
3. Identify missing fields, incorrect values, or formatting problems.
|
||||
|
||||
Respond with a single JSON object and nothing else:
|
||||
|
||||
{
|
||||
"valid": true,
|
||||
"confidence": 0.95,
|
||||
"issues": [
|
||||
{
|
||||
"field": "total",
|
||||
"severity": "error",
|
||||
"message": "Extracted total 120.00 does not match image 122.00",
|
||||
"suggested_value": 122.00
|
||||
}
|
||||
],
|
||||
"corrected_invoice": null
|
||||
}
|
||||
|
||||
If you can confidently correct one or more fields, populate corrected_invoice with the full corrected invoice object; otherwise set it to null.
|
||||
confidence must be between 0.0 and 1.0.
|
||||
Reference in New Issue
Block a user