obsidian-remarkable/IMPLEMENTATION.md

Obsidian reMarkable Sync Plugin — Implementation Summary

Status: ✅ Phase 1 & 2 Complete

Phase	Status	Description
Phase 1	✅ Done + Reviewed	Core sync, rmAPI bridge, PDF conversion, settings UI
Phase 2	✅ Done	Handwriting OCR pipeline: HWR extraction → PNG render → GLM-OCR → Ollama style refinement

---

🔧 Fixes Applied (Post-Review)

Critical Bugs (all fixed)

#	Issue	File	Fix
1	SyncTracker corrupted plugin settings	src/sync/tracker.ts	Merges with existing data before saving
2	runCommand() hung on spawn errors	src/utils/process.ts	Added error event handler + dedup resolution
3	Invalid "dir" in manifest	manifest.json	Removed property
4	Wrong vault path handling	src/sync/downloader.ts	Uses vault.adapter relative paths

Major Issues (all fixed)

#	Issue	File	Fix
5	ls flag order (Go parser)	src/rmapi/bridge.ts	--json before positional path
6	No incremental sync	src/sync/downloader.ts	Checks modifiedClient vs lastSynced
7	Unsanitized filenames	src/sync/downloader.ts	sanitizeFileName() replaces illegal chars
8	Broken auth flow	src/rmapi/bridge.ts	isAuthenticated() + user notice
9	Only root-level sync	src/sync/downloader.ts	listAllDocuments() recursively traverses folders
10	Noisy load/unload notices	src/main.ts	Removed; status bar shows state instead
11	Broken dev watch script	esbuild.config.mjs	Uses esbuild.context().watch()
12	Dead killProcess() code	src/utils/process.ts	Removed
13	mkdir silent failures	src/sync/downloader.ts	Wrapped in try/catch with warning

---

📁 Project Structure

obidian-remarkable/
├── dist/
│   └── main.js                    # Built plugin (~15 KB, minified)
├── src/
│   ├── main.ts                    # Plugin entry: ribbon, commands, status bar, auto-sync
│   ├── settings.ts                # Settings tab with validation
│   ├── types.ts                   # Shared TypeScript types
│   ├── rmapi/
│   │   └── bridge.ts              # rmapi CLI wrapper (env, auth check, JSON parsing)
│   ├── sync/
│   │   ├── downloader.ts          # Recursive doc listing + incremental download + OCR trigger
│   │   └── tracker.ts             # Sync state persistence (merges with settings)
│   ├── ocr/
│   │   ├── pipeline.ts            # Orchestrates the 4-stage OCR pipeline
│   │   ├── remarkable-hwr.ts      # Extracts built-in HWR text from .rm zip
│   │   ├── glmocr-client.ts       # HTTP client for GLM-OCR Server
│   │   └── style-refiner.ts       # Ollama client for Markdown cleanup (Option A)
│   ├── convert/
│   │   └── render.ts              # .rm page → PNG via drawj2d or rM2svg
│   └── utils/
│       ├── process.ts             # Child process runner with error handling
│       └── zip.ts                 # .rm zip extraction + notebook detection
├── main.ts                        # Entry point (re-exports plugin)
├── manifest.json                  # Obsidian plugin manifest
├── package.json                   # Build scripts
├── esbuild.config.mjs             # esbuild config (build + watch modes)
└── IMPLEMENTATION.md              # This file

---

🚀 Usage

Initial Setup

1. Install plugin: Copy folder to Obsidian plugins directory
2. Install rmapi: Download from https://github.com/ddvk/rmapi/releases
3. Install page renderer (optional, for OCR):
   • drawj2d: Download JAR, place in PATH or configure javaPath
   • rM2svg: Install binary, plus rsvg-convert or ImageMagick
4. Configure: Open Settings → reMarkable Sync
5. Authenticate: Run rmapi in a terminal once to pair with your tablet

Daily Use

• Sync all: Click ribbon pencil icon or run command "Sync from reMarkable"
• Convert single file: Open a .rm file in Obsidian, run "Convert handwriting to Markdown"

---

🔄 Sync Flow (Phase 1)

User clicks ribbon icon / auto-sync interval
  → isAuthenticated() check
  → listAllDocuments("/") recursively
    → ls --json / (root)
    → For each CollectionType: recurse into subfolder
  → For each DocumentType:
    → Skip if lastSynced >= modifiedClient (incremental)
    → sanitizeFileName() for safe filesystem names
    → mkdir downloadPath (vault-relative)
    → rmapi get → download .rm file
    → (if enabled) rmapi geta → download annotated PDF
    → trackDocument() in sync state
    → (if enabled) trigger OCR pipeline → .md
  → Status bar: "Last sync: HH:MM:SS"

---

🖋️ Handwriting OCR Pipeline (Phase 2)

flowchart TB
    subgraph Stage0["Stage 0: Extract"]
        RM[".rm notebook file"]
        RM --> UNZIP["unzip .rm archive"]
    end

    subgraph Stage1["Stage 1: reMarkable HWR"]
        UNZIP --> CONTENT["Read content.json"]
        CONTENT --> HWR["Extract built-in<br/>handwriting text"]
    end

    subgraph Stage2["Stage 2: Page Render"]
        UNZIP --> PAGES["List .rm page files"]
        PAGES --> RENDER["drawj2d / rM2svg<br/>→ page-N.png"]
    end

    subgraph Stage3["Stage 3: GLM-OCR"]
        RENDER --> BASE64["base64 encode images"]
        BASE64 --> POST["POST /glmocr/parse"]
        POST --> GLM_MD["glmocr markdown_result"]
    end

    subgraph Stage4["Stage 4: Style Refinement"]
        HWR --> MERGE["Merge sources"]
        GLM_MD --> MERGE
        MERGE --> OLLAMA["Ollama qwen3:32b<br/>Option A: light cleanup"]
        OLLAMA --> FINAL["Final .md file"]
    end

Stage Details

Stage	Tool	Input	Output	Notes
0	unzip	.rm file	Extracted directory	.rm files are zip archives
1	Custom parser	content.json	Raw text	Best-effort; may return empty string
2	drawj2d or rM2svg	.rm page files	page-0.png, page-1.png, ...	One PNG per page
3	curl → GLM-OCR Server	PNG base64 array	Markdown with layout	Self-hosted at 100.103.83.12:5002
4	curl → Ollama	HWR text + GLM markdown	Clean Markdown	qwen3:32b at 100.103.83.12:11435

Style Refinement Prompt (Option A)

You have two sources of text from the same handwritten document:

SOURCE 1 — ReMarkable built-in handwriting recognition (raw text, no formatting)
{hwrText}

SOURCE 2 — GLM-OCR engine output (markdown with structure but possible errors)
{glmOcrMarkdown}

Your task:
1. Merge both sources, preferring SOURCE 2 for structure and SOURCE 1 for text accuracy
2. Fix heading hierarchy (h1 → h2 → h3 logically)
3. Consolidate fragmented paragraphs
4. Detect and format lists, checkboxes, tables
5. Identify emphasized/underlined text and apply bold/italic
6. Preserve all content — do not summarize or remove anything

OUTPUT ONLY THE FINAL REFINED MARKDOWN.

---

⚙️ Configuration

Setting	Default	Description
remarkableHost	https://10.11.99.1	reMarkable tablet URL
rmapiBinaryPath	rmapi	Path to rmapi binary
downloadPath	remarkable/	Vault folder for downloads
convertToPdf	true	Auto-download annotated PDFs
enableHandwritingMd	true	Trigger OCR pipeline after sync
glmocrServerUrl	http://100.103.83.12:5002	GLM-OCR SDK Server
glmocrApiKey	any-string	Dummy key for self-hosted
ollamaHost	http://100.103.83.12:11435	Ollama server
styleModel	qwen3:32b	Model for markdown cleanup
pageRenderer	drawj2d	.rm → PNG tool
javaPath	java	Java runtime for drawj2d
syncInterval	0	Minutes between auto-sync (0 = off)

---

📦 External Dependencies

Required (user-provided)

• rmapi binary: https://github.com/ddvk/rmapi/releases
• reMarkable tablet: On local network at configured host

Required for OCR (Phase 2)

• GLM-OCR Server: python -m glmocr.server on configured host
• Ollama: With qwen3:32b (or chosen model) pulled
• Page renderer:
  • drawj2d: Java JAR (recommended for Paper Pro v3.x)
  • rM2svg: Binary + rsvg-convert or ImageMagick

---

📝 Architecture Decisions

Decision	Rationale
Spawn rmapi as CLI	Mature Go binary; avoids reimplementing reMarkable sync protocol
Store auth in vault	RMAPI_CONFIG in .obsidian/rmapi — portable with vault
Use curl for HTTP	Available on all platforms; avoids bundling HTTP client
Base64 encode images	GLM-OCR server accepts data:image/png;base64,... inline
3-stage OCR pipeline	HWR (free, on-device) + GLM-OCR (accurate) + Ollama (cleanup)
Light cleanup (Option A)	Preserves all content; fixes structure without rewriting
Template literal prompts	Easy to read and modify; no external prompt files

---

🎯 Build & Install

# Build
npm install
npm run build

# Dev (watch mode)
npm run dev

# Install in Obsidian
Copy the `obidian-remarkable` folder to:
- Linux: ~/.config/obsidian/plugins/
- macOS: ~/Library/Application Support/obsidian/plugins/
- Windows: %APPDATA%\obsidian\plugins\

---

🐛 Known Limitations

1. HWR extraction is best-effort: reMarkable v3.x content.json format isn't fully documented. If no HWR text is found, the pipeline falls back to GLM-OCR alone.
2. Page renderer path: drawj2d.jar is assumed in PATH. You may need to set an absolute path in settings (future improvement).
3. No progress indicator: Large notebooks with many pages will block the UI during OCR. Consider adding a progress modal.
4. Temp directory: Uses .obsidian/rmapi-tmp and cleans up with rm -rf. On Windows, rm may not exist (needs rd /s /q fallback).

---

Status: ✅ Ready for testing. Both phases complete and reviewed.