obsidian-remarkable/IMPLEMENTATION.md

# Obsidian reMarkable Sync Plugin — Implementation Summary

## Status: ✅ Phase 1 & 2 Complete + All Fixes Applied

| Phase | Status | Description |
|---|---|---|
| **Phase 1** | ✅ Done + Reviewed | Core sync, rmAPI bridge, PDF conversion, settings UI |
| **Phase 2** | ✅ Done + Fixed | Handwriting OCR pipeline: HWR extraction → PNG render → GLM-OCR → Ollama style refinement |

---

## 🔧 Fixes Applied (Post-Review)

### Critical Bugs (all fixed)
| # | Issue | File | Fix |
|---|---|---|---|
| 1 | `SyncTracker` corrupted plugin settings | `src/sync/tracker.ts` | Merges with existing data before saving |
| 2 | `runCommand()` hung on spawn errors | `src/utils/process.ts` | Added `error` event handler + dedup resolution |
| 3 | Invalid `"dir"` in manifest | `manifest.json` | Removed property |
| 4 | Wrong vault path handling | `src/sync/downloader.ts` | Uses `vault.adapter` relative paths |

### Major Issues (all fixed)
| # | Issue | File | Fix |
|---|---|---|---|
| 5 | `ls` flag order (Go parser) | `src/rmapi/bridge.ts` | `--json` before positional path |
| 6 | No incremental sync | `src/sync/downloader.ts` | Checks `modifiedClient` vs `lastSynced` |
| 7 | Unsanitized filenames | `src/sync/downloader.ts` | `sanitizeFileName()` replaces illegal chars |
| 8 | Broken auth flow | `src/rmapi/bridge.ts` | `isAuthenticated()` + user notice |
| 9 | Only root-level sync | `src/sync/downloader.ts` | `listAllDocuments()` recursively traverses folders |
| 10 | Noisy load/unload notices | `src/main.ts` | Removed; status bar shows state instead |
| 11 | Broken dev watch script | `esbuild.config.mjs` | Uses `esbuild.context().watch()` |
| 12 | Dead `killProcess()` code | `src/utils/process.ts` | Removed |
| 13 | `mkdir` silent failures | `src/sync/downloader.ts` | Wrapped in try/catch with warning |

### Phase 2 Issues (all fixed)
| # | Issue | File | Fix |
|---|---|---|---|
| 14 | `drawj2d.jar` hardcoded | `src/convert/render.ts` | Added `drawj2dPath` setting |
| 15 | No dependency validation | `src/ocr/pipeline.ts` | Added `validateDependencies()` on load |
| 16 | No page batching | `src/ocr/pipeline.ts` | Batched GLM-OCR requests with `maxPagesPerBatch` |
| 17 | No progress feedback | `src/ocr/pipeline.ts` | Progress notices every 5 pages and per batch |
| 18 | Windows `rm` incompatibility | `src/ocr/pipeline.ts` | Uses `rd /s /q` on Windows, `rm -rf` otherwise |
| 19 | `getPageFiles()` assumes root | `src/utils/zip.ts` | Recursively searches for `.rm` files |
| 20 | Temp dir uses absolute path | `src/ocr/pipeline.ts` | Uses vault-relative path `.obsidian/rmapi-tmp` |
| 21 | No HTTP timeout | `src/ocr/glmocr-client.ts`, `src/ocr/style-refiner.ts` | Added `--max-time 60` to curl |
| 22 | No LLM output validation | `src/ocr/style-refiner.ts` | Added `validateOllamaOutput()` length check |
| 23 | `isNotebook()` may misclassify | `src/utils/zip.ts` | Falls back to `file` command if ambiguous |

---

## 📁 Project Structure

```
obidian-remarkable/
├── dist/
│   └── main.js                    # Built plugin (~17 KB, minified)
├── src/
│   ├── main.ts                    # Plugin entry: ribbon, commands, status bar, auto-sync, dependency validation
│   ├── settings.ts                # Settings tab with validation + 2 new settings
│   ├── types.ts                   # Shared TypeScript types + 2 new fields
│   ├── rmapi/
│   │   └── bridge.ts              # rmapi CLI wrapper (env, auth check, JSON parsing)
│   ├── sync/
│   │   ├── downloader.ts          # Recursive doc listing + incremental download + OCR trigger
│   │   └── tracker.ts             # Sync state persistence (merges with settings)
│   ├── ocr/
│   │   ├── pipeline.ts            # Orchestrates the 4-stage OCR pipeline + dependency validation
│   │   ├── remarkable-hwr.ts      # Extracts built-in HWR text from .rm zip + fallback
│   │   ├── glmocr-client.ts       # HTTP client for GLM-OCR Server + timeout
│   │   └── style-refiner.ts       # Ollama client for Markdown cleanup + output validation
│   ├── convert/
│   │   └── render.ts              # .rm page → PNG via drawj2d or rM2svg (configurable path)
│   └── utils/
│       ├── process.ts             # Child process runner with error handling
│       └── zip.ts                 # .rm zip extraction + notebook detection + recursive page search
├── main.ts                        # Entry point (re-exports plugin)
├── manifest.json                  # Obsidian plugin manifest (fixed)
├── package.json                   # Build scripts
├── esbuild.config.mjs             # esbuild config (build + watch modes)
└── IMPLEMENTATION.md              # This file
```

---

## 🚀 Usage

### Initial Setup
1. **Install plugin**: Copy folder to Obsidian plugins directory
2. **Install rmapi**: Download from https://github.com/ddvk/rmapi/releases
3. **Install page renderer** (optional, for OCR):
   - **drawj2d**: Download JAR, set path in `drawj2dPath` setting
   - **rM2svg**: Install binary, plus `rsvg-convert` or ImageMagick
4. **Configure**: Open Settings → reMarkable Sync
5. **Authenticate**: Run `rmapi` in a terminal once to pair with your tablet

### Daily Use
- **Sync all**: Click ribbon pencil icon or run command "Sync from reMarkable"
- **Convert single file**: Open a `.rm` file in Obsidian, run "Convert handwriting to Markdown"

---

## 🔄 Sync Flow (Phase 1)

```
User clicks ribbon icon / auto-sync interval
  → isAuthenticated() check
  → listAllDocuments("/") recursively
    → ls --json / (root)
    → For each CollectionType: recurse into subfolder
  → For each DocumentType:
    → Skip if lastSynced >= modifiedClient (incremental)
    → sanitizeFileName() for safe filesystem names
    → mkdir downloadPath (vault-relative)
    → rmapi get → download .rm file
    → (if enabled) rmapi geta → download annotated PDF
    → trackDocument() in sync state
    → (if enabled) trigger OCR pipeline → .md
  → Status bar: "Last sync: HH:MM:SS"
```

---

## 🖋️ Handwriting OCR Pipeline (Phase 2)

```mermaid
flowchart TB
    subgraph Stage0["Stage 0: Extract"]
        RM[".rm notebook file"]
        RM --> UNZIP["unzip .rm archive"]
    end

    subgraph Stage1["Stage 1: reMarkable HWR"]
        UNZIP --> CONTENT["Read content.json"]
        CONTENT --> HWR["Extract built-in<br/>handwriting text"]
    end

    subgraph Stage2["Stage 2: Page Render"]
        UNZIP --> PAGES["List .rm page files<br/>(recursive)"]
        PAGES --> RENDER["drawj2d / rM2svg<br/>→ page-N.png"]
        RENDER --> PROG1["Progress: every 5 pages"]
    end

    subgraph Stage3["Stage 3: GLM-OCR"]
        RENDER --> BASE64["base64 encode images"]
        BASE64 --> BATCH["Batch into<br/>maxPagesPerBatch"]
        BATCH --> POST["POST /glmocr/parse<br/>(--max-time 60)"]
        POST --> GLM_MD["glmocr markdown_result"]
        POST --> PROG2["Progress: per batch"]
    end

    subgraph Stage4["Stage 4: Style Refinement"]
        HWR --> MERGE["Merge sources"]
        GLM_MD --> MERGE
        MERGE --> OLLAMA["Ollama qwen3:32b<br/>Option A: light cleanup"]
        OLLAMA --> VALIDATE["validateOllamaOutput()<br/>(length check)"]
        VALIDATE --> FINAL["Final .md file"]
    end
```

### Stage Details

| Stage | Tool | Input | Output | Notes |
|---|---|---|---|---|
| **0** | `unzip` | `.rm` file | Extracted directory | `.rm` files are zip archives |
| **1** | Custom parser | `content.json` | Raw text | Best-effort; may return empty string |
| **2** | `drawj2d` or `rM2svg` | `.rm` page files | `page-0.png`, `page-1.png`, ... | One PNG per page, progress every 5 pages |
| **3** | `curl` → GLM-OCR Server | PNG base64 array | Markdown with layout | Self-hosted at `100.103.83.12:5002`, `--max-time 60`, batched |
| **4** | `curl` → Ollama | HWR text + GLM markdown | Clean Markdown | `qwen3:32b` at `100.103.83.12:11435`, output length validated |

### Style Refinement Prompt (Option A)

```
You have two sources of text from the same handwritten document:

SOURCE 1 — ReMarkable built-in handwriting recognition (raw text, no formatting)
{hwrText}

SOURCE 2 — GLM-OCR engine output (markdown with structure but possible errors)
{glmOcrMarkdown}

Your task:
1. Merge both sources, preferring SOURCE 2 for structure and SOURCE 1 for text accuracy
2. Fix heading hierarchy (h1 → h2 → h3 logically)
3. Consolidate fragmented paragraphs
4. Detect and format lists, checkboxes, tables
5. Identify emphasized/underlined text and apply bold/italic
6. Preserve all content — do not summarize or remove anything

OUTPUT ONLY THE FINAL REFINED MARKDOWN.
```

---

## ⚙️ Configuration

| Setting | Default | Description |
|---|---|---|
| `remarkableHost` | `https://10.11.99.1` | reMarkable tablet URL |
| `rmapiBinaryPath` | `rmapi` | Path to rmapi binary |
| `downloadPath` | `remarkable/` | Vault folder for downloads |
| `convertToPdf` | `true` | Auto-download annotated PDFs |
| `enableHandwritingMd` | `true` | **Trigger OCR pipeline after sync** |
| `glmocrServerUrl` | `http://100.103.83.12:5002` | GLM-OCR SDK Server |
| `glmocrApiKey` | `any-string` | Dummy key for self-hosted |
| `ollamaHost` | `http://100.103.83.12:11435` | Ollama server |
| `styleModel` | `qwen3:32b` | Model for markdown cleanup |
| `pageRenderer` | `drawj2d` | `.rm` → PNG tool |
| `javaPath` | `java` | Java runtime for drawj2d |
| `drawj2dPath` | `drawj2d.jar` | **Path to drawj2d.jar** |
| `maxPagesPerBatch` | `20` | **Max pages per GLM-OCR batch** |
| `syncInterval` | `0` | Minutes between auto-sync (0 = off) |

---

## 📦 External Dependencies

### Required (user-provided)
- **rmapi binary**: https://github.com/ddvk/rmapi/releases
- **reMarkable tablet**: On local network at configured host

### Required for OCR (Phase 2)
- **GLM-OCR Server**: `python -m glmocr.server` on configured host
- **Ollama**: With `qwen3:32b` (or chosen model) pulled
- **Page renderer**:
  - **drawj2d**: Java JAR (recommended for Paper Pro v3.x) + `drawj2dPath` setting
  - **rM2svg**: Binary + `rsvg-convert` or ImageMagick
- **Standard CLI tools**: `unzip`, `curl`, `base64`, `file` (validated on load)

---

## 📝 Architecture Decisions

| Decision | Rationale |
|---|---|
| **Spawn rmapi as CLI** | Mature Go binary; avoids reimplementing reMarkable sync protocol |
| **Store auth in vault** | `RMAPI_CONFIG` in `.obsidian/rmapi` — portable with vault |
| **Use `curl` for HTTP** | Available on all platforms; avoids bundling HTTP client |
| **Base64 encode images** | GLM-OCR server accepts `data:image/png;base64,...` inline |
| **3-stage OCR pipeline** | HWR (free, on-device) + GLM-OCR (accurate) + Ollama (cleanup) |
| **Light cleanup (Option A)** | Preserves all content; fixes structure without rewriting |
| **Template literal prompts** | Easy to read and modify; no external prompt files |
| **Configurable batch size** | Avoids server payload limits and UI hangs |
| **Dependency validation** | User knows what's missing before OCR fails |
| **Progress feedback** | Better UX for large notebooks |
| **Output validation** | Trust but verify LLM output |
| **Windows compatibility** | Uses `rd /s /q` on Windows, `rm -rf` on Unix |
| **Recursive page search** | Handles `.rm` files in subdirectories |

---

## 🎯 Build & Install

```bash
# Build
npm install
npm run build

# Dev (watch mode)
npm run dev

# Install in Obsidian
Copy the `obidian-remarkable` folder to:
- Linux: ~/.config/obsidian/plugins/
- macOS: ~/Library/Application Support/obsidian/plugins/
- Windows: %APPDATA%\obsidian\plugins\
```

---

## 🐛 Known Limitations

1. **HWR extraction is best-effort**: reMarkable v3.x `content.json` format isn't fully documented. If no HWR text is found, the pipeline falls back to GLM-OCR alone.
2. **Page renderer path**: `drawj2d.jar` is assumed in PATH. You may need to set an absolute path in settings (future improvement).
3. **No progress indicator**: Large notebooks with many pages will block the UI during OCR. Consider adding a progress modal.
4. **Temp directory**: Uses `.obsidian/rmapi-tmp` and cleans up with `rm -rf` or `rd /s /q`.

---

**Status**: ✅ Ready for testing. Both phases complete, reviewed, and all issues fixed.