obsidian-remarkable/IMPLEMENTATION.md

Obsidian reMarkable Sync Plugin — Implementation Summary

Status: ✅ Phase 1 & 2 Complete + All Fixes Applied

Phase	Status	Description
Phase 1	✅ Done + Reviewed	Core sync, rmAPI bridge, PDF conversion, settings UI
Phase 2	✅ Done + Fixed	Handwriting OCR pipeline: HWR extraction → PNG render → GLM-OCR → Ollama style refinement

---

🔧 Fixes Applied (Post-Review)

Critical Bugs (all fixed)

#	Issue	File	Fix
1	SyncTracker corrupted plugin settings	src/sync/tracker.ts	Merges with existing data before saving
2	runCommand() hung on spawn errors	src/utils/process.ts	Added error event handler + dedup resolution
3	Invalid "dir" in manifest	manifest.json	Removed property
4	Wrong vault path handling	src/sync/downloader.ts	Uses vault.adapter relative paths

Major Issues (all fixed)

#	Issue	File	Fix
5	ls flag order (Go parser)	src/rmapi/bridge.ts	--json before positional path
6	No incremental sync	src/sync/downloader.ts	Checks modifiedClient vs lastSynced
7	Unsanitized filenames	src/sync/downloader.ts	sanitizeFileName() replaces illegal chars
8	Broken auth flow	src/rmapi/bridge.ts	isAuthenticated() + user notice
9	Only root-level sync	src/sync/downloader.ts	listAllDocuments() recursively traverses folders
10	Noisy load/unload notices	src/main.ts	Removed; status bar shows state instead
11	Broken dev watch script	esbuild.config.mjs	Uses esbuild.context().watch()
12	Dead killProcess() code	src/utils/process.ts	Removed
13	mkdir silent failures	src/sync/downloader.ts	Wrapped in try/catch with warning

Phase 2 Issues (all fixed)

#	Issue	File	Fix
14	drawj2d.jar hardcoded	src/convert/render.ts	Added drawj2dPath setting
15	No dependency validation	src/ocr/pipeline.ts	Added validateDependencies() on load
16	No page batching	src/ocr/pipeline.ts	Batched GLM-OCR requests with maxPagesPerBatch
17	No progress feedback	src/ocr/pipeline.ts	Progress notices every 5 pages and per batch
18	Windows rm incompatibility	src/ocr/pipeline.ts	Uses Node filesystem APIs for temp cleanup
19	getPageFiles() assumes root	src/utils/zip.ts	Recursively searches for .rm files
20	Temp dir path handling	src/ocr/pipeline.ts	Uses an absolute filesystem path under .obsidian/rmapi-tmp for shell tools
21	No HTTP timeout	src/ocr/glmocr-client.ts, src/ocr/style-refiner.ts	Added --max-time 60 to curl
22	No LLM output validation	src/ocr/style-refiner.ts	Added validateOllamaOutput() length check
23	Unix-only recursive listing	src/utils/zip.ts	Uses Node recursive directory traversal instead of find

---

📁 Project Structure

obidian-remarkable/
├── main.js                        # Built plugin entry loaded by Obsidian
├── dist/
│   └── main.js                    # Secondary built artifact
├── src/
│   ├── main.ts                    # Plugin entry: ribbon, commands, status bar, auto-sync, dependency validation
│   ├── settings.ts                # Settings tab with validation + 2 new settings
│   ├── types.ts                   # Shared TypeScript types + 2 new fields
│   ├── rmapi/
│   │   └── bridge.ts              # rmapi CLI wrapper (env, auth check, JSON parsing)
│   ├── sync/
│   │   ├── downloader.ts          # Recursive doc listing + incremental download + OCR trigger
│   │   └── tracker.ts             # Sync state persistence (merges with settings)
│   ├── ocr/
│   │   ├── pipeline.ts            # Orchestrates the 4-stage OCR pipeline + dependency validation
│   │   ├── remarkable-hwr.ts      # Extracts built-in HWR text from .rm zip + fallback
│   │   ├── glmocr-client.ts       # HTTP client for GLM-OCR Server + timeout
│   │   └── style-refiner.ts       # Ollama client for Markdown cleanup + output validation
│   ├── convert/
│   │   └── render.ts              # .rm page → PNG via drawj2d or rM2svg (configurable path)
│   └── utils/
│       ├── process.ts             # Child process runner with error handling
│       ├── paths.ts               # Vault-relative and absolute path helpers
│       └── zip.ts                 # .rm zip extraction + notebook detection + recursive page search
├── main.ts                        # Entry point (re-exports plugin)
├── manifest.json                  # Obsidian plugin manifest (fixed)
├── package.json                   # Build scripts
├── esbuild.config.mjs             # esbuild config (build + watch modes)
└── IMPLEMENTATION.md              # This file

---

🚀 Usage

Initial Setup

1. Install plugin: Copy folder to Obsidian plugins directory
2. Install rmapi: Download from https://github.com/ddvk/rmapi/releases
3. Install page renderer (optional, for OCR):
   • drawj2d: Download JAR, set path in drawj2dPath setting
   • rM2svg: Install binary, plus rsvg-convert or ImageMagick
4. Configure: Open Settings → reMarkable Sync
5. Authenticate: Run rmapi in a terminal once to pair with your tablet

Daily Use

• Sync all: Click ribbon pencil icon or run command "Sync from reMarkable"
• Browse files: Click ribbon folder icon or run command "Browse reMarkable" to open the side-panel browser
• Per-file actions: In the browser, click "Import" or "Handwriting to Markdown" for each document
• Convert single file: Open a .rm file in Obsidian, run "Convert handwriting to Markdown"

---

🔄 Sync Flow (Phase 1)

User clicks ribbon icon / auto-sync interval
  → isAuthenticated() check
  → listAllDocuments("/") recursively
    → ls --json / (root)
    → For each CollectionType: recurse into subfolder
  → For each DocumentType:
    → Skip if tracked remoteVersion and remoteModified match (incremental)
    → sanitizeFileName() for safe filesystem names
    → mkdir downloadPath (vault-relative)
    → rmapi get → download .rm file
    → (if enabled) rmapi geta → download annotated PDF
    → trackDocument() in sync state
    → (if enabled) trigger OCR pipeline → .md
  → Status bar: "Last sync: HH:MM:SS"

---

🖋️ Handwriting OCR Pipeline (Phase 2)

flowchart TB
    subgraph Stage0["Stage 0: Extract"]
        RM[".rm notebook file"]
        RM --> UNZIP["unzip .rm archive"]
    end

    subgraph Stage1["Stage 1: reMarkable HWR"]
        UNZIP --> CONTENT["Read content.json"]
        CONTENT --> HWR["Extract built-in<br/>handwriting text"]
    end

    subgraph Stage2["Stage 2: Page Render"]
        UNZIP --> PAGES["List .rm page files<br/>(recursive)"]
        PAGES --> RENDER["drawj2d / rM2svg<br/>→ page-N.png"]
        RENDER --> PROG1["Progress: every 5 pages"]
    end

    subgraph Stage3["Stage 3: GLM-OCR"]
        RENDER --> BASE64["base64 encode images"]
        BASE64 --> BATCH["Batch into<br/>maxPagesPerBatch"]
        BATCH --> POST["POST /glmocr/parse<br/>(--max-time 60)"]
        POST --> GLM_MD["glmocr markdown_result"]
        POST --> PROG2["Progress: per batch"]
    end

    subgraph Stage4["Stage 4: Style Refinement"]
        HWR --> MERGE["Merge sources"]
        GLM_MD --> MERGE
        MERGE --> OLLAMA["Ollama qwen3:32b<br/>Option A: light cleanup"]
        OLLAMA --> VALIDATE["validateOllamaOutput()<br/>(length check)"]
        VALIDATE --> FINAL["Final .md file"]
    end

Stage Details

Stage	Tool	Input	Output	Notes
0	unzip	.rm file	Extracted directory	.rm files are zip archives
1	Custom parser	content.json	Raw text	Best-effort; may return empty string
2	drawj2d or rM2svg	.rm page files	page-0.png, page-1.png, ...	One PNG per page, progress every 5 pages
3	curl → GLM-OCR Server	PNG base64 array	Markdown with layout	Self-hosted at configured server URL, --max-time 60, batched
4	curl → Ollama	HWR text + GLM markdown	Clean Markdown	Configured Ollama host/model, output length validated

Style Refinement Prompt (Option A)

You have two sources of text from the same handwritten document:

SOURCE 1 — ReMarkable built-in handwriting recognition (raw text, no formatting)
{hwrText}

SOURCE 2 — GLM-OCR engine output (markdown with structure but possible errors)
{glmOcrMarkdown}

Your task:
1. Merge both sources, preferring SOURCE 2 for structure and SOURCE 1 for text accuracy
2. Fix heading hierarchy (h1 → h2 → h3 logically)
3. Consolidate fragmented paragraphs
4. Detect and format lists, checkboxes, tables
5. Identify emphasized/underlined text and apply bold/italic
6. Preserve all content — do not summarize or remove anything

OUTPUT ONLY THE FINAL REFINED MARKDOWN.

---

⚙️ Configuration

Setting	Default	Description
remarkableHost	https://10.11.99.1	reMarkable tablet URL
rmapiBinaryPath	rmapi	Path to rmapi binary
downloadPath	remarkable/	Vault folder for downloads
convertToPdf	true	Auto-download annotated PDFs
enableHandwritingMd	true	Trigger OCR pipeline after sync
glmocrServerUrl	http://localhost:5002	GLM-OCR SDK Server
glmocrApiKey	empty	API key for GLM-OCR Server
ollamaHost	http://localhost:11435	Ollama server
styleModel	qwen3:32b	Model for markdown cleanup
pageRenderer	drawj2d	.rm → PNG tool
javaPath	java	Java runtime for drawj2d
drawj2dPath	drawj2d.jar	Path to drawj2d.jar
maxPagesPerBatch	20	Max pages per GLM-OCR batch
syncInterval	0	Minutes between auto-sync (0 = off)

---

📦 External Dependencies

Required (user-provided)

• rmapi binary: https://github.com/ddvk/rmapi/releases
• reMarkable tablet: On local network at configured host

Required for OCR (Phase 2)

• GLM-OCR Server: python -m glmocr.server on configured host
• Ollama: With qwen3:32b (or chosen model) pulled
• Page renderer:
  • drawj2d: Java JAR (recommended for Paper Pro v3.x) + drawj2dPath setting
  • rM2svg: Binary + rsvg-convert or ImageMagick
• Standard CLI tools: unzip, curl (validated on load)

---

📝 Architecture Decisions

Decision	Rationale
Spawn rmapi as CLI	Mature Go binary; avoids reimplementing reMarkable sync protocol
Store auth in vault	RMAPI_CONFIG in .obsidian/rmapi — portable with vault
Use curl for HTTP	Available on all platforms; avoids bundling HTTP client
Base64 encode images	GLM-OCR server accepts data:image/png;base64,... inline
3-stage OCR pipeline	HWR (free, on-device) + GLM-OCR (accurate) + Ollama (cleanup)
Light cleanup (Option A)	Preserves all content; fixes structure without rewriting
Template literal prompts	Easy to read and modify; no external prompt files
Configurable batch size	Avoids server payload limits and UI hangs
Dependency validation	User knows what's missing before OCR fails
Progress feedback	Better UX for large notebooks
Output validation	Trust but verify LLM output
Windows compatibility	Uses Node filesystem APIs for temp cleanup
Recursive page search	Handles .rm files in subdirectories

---

🎯 Build & Install

# Build
npm install
npm run build

# Dev (watch mode)
npm run dev

# Install in Obsidian
Run `npm run build`, then copy the plugin folder containing `manifest.json` and root `main.js` to:
- Linux: ~/.config/obsidian/plugins/
- macOS: ~/Library/Application Support/obsidian/plugins/
- Windows: %APPDATA%\obsidian\plugins\

# Or install directly into a vault
npm run install-plugin -- --vault /path/to/vault

# Exact target directory also works
npm run install-plugin -- --plugin-dir /path/to/vault/.obsidian/plugins/obsidian-remarkable

# OCR dependency setup
npm run install-plugin -- --vault /path/to/vault --drawj2d-jar /path/to/drawj2d.jar
npm run install-plugin -- --vault /path/to/vault --install-deps

---

🐛 Known Limitations

1. HWR extraction is best-effort: reMarkable v3.x content.json format isn't fully documented. If no HWR text is found, the pipeline falls back to GLM-OCR alone.
2. OCR runtime dependencies: GLM-OCR, Ollama, and a page renderer must be running/installed outside Obsidian.
3. Progress UX: Large notebooks use notices for progress, but there is no cancellable progress modal yet.
4. Temp directory: Uses an absolute filesystem path at <vault>/.obsidian/rmapi-tmp and cleans it up with Node filesystem APIs.

---

Status: ✅ Ready for testing. Both phases complete, reviewed, and all issues fixed.