22 Commits

Author SHA1 Message Date
fegger 378642152e fix: plugin cannot be installed — manifest path, deps, and install script
- manifest.json: fix 'main' from 'src/main.js' to 'dist/main.js'
  (TypeScript compiles to dist/, not src/ — Obsidian could not find the entry point)
- manifest.json: set isDesktopOnly to true (plugin requires a local Ollama server)
- package.json: remove unused node-fetch dependency (ESM-only, conflicts with CommonJS build)
- src/semantic-cache.ts: use dynamic import for chromadb instead of top-level import
  (prevents plugin load crash when chromadb is not installed; cache defaults to disabled)
- install.sh: new script that installs deps, builds, and copies the plugin into a vault
- README.md: add quick install, manual install, and expanded troubleshooting sections
2026-05-19 17:39:10 +02:00
fegger 26d86d01db Update main entry point to dist/main.js 2026-05-19 17:35:47 +02:00
fegger 2d78882594 chore: move compiled output to dist
Update the plugin entrypoint and TypeScript output directory to use dist instead of writing generated JavaScript into src.

Remove previously checked-in compiled source files and add coverage for the Ollama client and tool executor.
2026-05-19 17:30:05 +02:00
fegger 2e26c72c0c Add workflow engine for multi-step chained reasoning 2026-05-08 19:44:16 +02:00
fegger 52b4a6c021 Add tests for conversation state, graph view, tool executor, and vectorization 2026-05-08 18:21:20 +02:00
fegger 4a66b46844 Add dependency graph view and refine chat message type 2026-05-08 12:33:45 +02:00
fegger 771db09d24 Refactor chat view and add conversation state management
Introduce ConversationStateManager to handle short, medium, and long-term
context for improved conversation flow. Update ChatView to use this manager
and refactor input handling to accept values directly for better testability.

Update OllamaClient with non-streaming chat support and improved error
handling for malformed chunks. Enhance vault indexer with caching, better
scoring, and stop word filtering. Refactor main plugin entry point and
semantic cache initialization for robustness.
2026-05-08 11:52:31 +02:00
fegger b37ca2c178 Add cache interface and semantic cache tests 2026-05-08 02:20:06 +02:00
fegger 6bc1133a81 Refactor type annotations and error handling in core modules
Standardize error message extraction to handle non-Error objects.
Add explicit return types and interface definitions to VaultIndexer.
Update SemanticCacheService to use any types where necessary.
Rename SemanticCache to SemanticCacheService and adjust imports.
2026-05-08 02:19:44 +02:00
fegger fff98d1a2e Merge branch 'modularize-indexing-pipeline' 2026-05-07 23:45:31 +02:00
fegger 6423e2aa0d Clean up settings display and remove validation logic
Remove unused imports and simplify Ollama client stream handling

Add configurable ChromaDB host and UUID fallback

Refactor vault indexer to use tokenized content and improve caching
2026-05-07 23:23:32 +02:00
fegger 1985f849f4 Add semantic caching support with ChromaDB integration
Implement semantic caching for Ollama chat responses using ChromaDB to store and retrieve embeddings. The cache can be
enabled/disabled in settings and includes configurable similarity threshold, embedding model, and ChromaDB URL. Added
cache clear functionality and error handling for cache operations.
2026-05-07 23:03:58 +02:00
fegger 951c3bbc92 Fix variable naming consistency for chroma URL configuration
Update chromaUrl to chromaURL throughout the codebase to ensure consistent naming convention for the Chroma database URL
configuration parameter. This change affects the semantic cache service implementation and related tests.

The change updates the configuration property name from `chromaUrl` to `chromaURL` in:
- SemanticCacheService class
- Test files (chat-view.test.ts, ollama-client-cache.test.ts, semantic-cache.test.ts)

This maintains consistency with other URL configuration parameters in the codebase and improves code readability.
2026-05-07 22:49:07 +02:00
fegger d4a0919764 Fix merge conflicts and apply all semantic cache fixes 2026-05-07 22:49:07 +02:00
fegger e6d791a655 Update README.md 2026-05-07 22:49:01 +02:00
fegger ae16396a7a Integrate semantic cache with ChromaDB URL 2026-05-07 22:47:32 +02:00
fegger 179a58b95b Add semantic cache support using ChromaDB 2026-05-07 22:47:01 +02:00
fegger d37b9f23bd Add semantic caching and indexing pipeline
This commit adds semantic caching functionality to speed up repeated queries and implements a complete indexing pipeline
for processing vault files. The changes include:

- Added semantic cache service using ChromaDB for storing and retrieving cached responses
- Implemented indexing pipeline with extraction, normalization, and vectorization steps
- Added cache configuration settings to the plugin
- Updated Ollama client to support cache integration
- Added tests for all new indexing components
- Extended vault indexer with indexing pipeline support
2026-05-07 22:23:35 +02:00
fegger 47ecd3f803 feat: Implement modular indexing pipeline with extraction, normalization, and vectorization components 2026-05-07 22:13:43 +02:00
fegger f3a10a4b01 Merge pull request 'Refactor retry logic and extract scoring weights' (#1) from agent/review into main
Reviewed-on: http://100.103.83.12:3003/fegger/obsidian_ollama/pulls/1
2026-05-07 20:56:27 +02:00
fegger 79db888f9e Refactor README with comprehensive documentation 2026-05-07 19:10:15 +02:00
fegger 3ab326ffc2 Refactor retry logic and extract scoring weights 2026-05-07 19:02:36 +02:00
46 changed files with 9571 additions and 3219 deletions
+159 -40
View File
@@ -1,65 +1,184 @@
# Ollama Chat Plugin for Obsidian
# Obsidian Ollama Plugin
A plugin that integrates Ollama with Obsidian to create a chat interface that can access your vault content.
A plugin that integrates [Ollama](https://ollama.ai) with Obsidian, allowing you to chat with local AI models, search your vault context, and use AI tools like creating files.
## Features
- Chat with Ollama models directly in Obsidian
- Vault context search - the assistant can reference your notes
- Tool integration - create files based on chat responses
- Vault context search the assistant can reference your notes
- Tool integration create files based on chat responses
- Streaming responses
- Customizable model and URL settings
- Semantic response cache — repeated or similar queries are answered instantly without hitting the model (requires ChromaDB)
- Customisable model, URL, and cache settings
## Prerequisites
1. **Install Ollama**: Follow the instructions at [ollama.ai](https://ollama.ai)
2. **Start Ollama**: `ollama serve`
3. **Pull a chat model**: `ollama pull llama3` (or any other model you prefer)
### Optional — Semantic Cache
The semantic cache stores responses in a local [ChromaDB](https://www.trychroma.com) vector database. When you ask a question that is semantically similar to one already cached, the stored answer is returned immediately instead of calling the model.
1. **Install ChromaDB**:
```bash
pip install chromadb
```
2. **Start ChromaDB**:
```bash
chroma run --host localhost --port 8000
```
3. **Pull an embedding model** (used to generate vectors for cache lookups):
```bash
ollama pull nomic-embed-text
```
4. Enable the cache in the plugin settings and configure the ChromaDB URL.
## Installation
1. Install the plugin via Obsidian's community plugins
2. Make sure you have Ollama installed and running
### Quick install (recommended)
## Setup
Use the included install script. It handles dependency installation, building, and copying the plugin into your vault:
1. **Install Ollama**: Follow the instructions at [ollama.ai](https://ollama.ai) to install Ollama
2. **Start Ollama service**: `ollama serve`
3. **Pull a model**: `ollama pull llama3` (or any other model you prefer)
```bash
# Clone or download this repository, then run:
./install.sh /path/to/your/obsidian/vault
```
## Configuration
The script will:
- Install npm dependencies (excluding Ollama — you install that separately)
- Compile the TypeScript plugin
- Copy the built plugin into `<vault>/.obsidian/plugins/ollama-plugin/`
1. Open the plugin settings via Obsidian's settings panel
2. Configure the Ollama URL (default: `http://localhost:11434`)
3. Configure the model name (default: `llama3`)
4. Restart the plugin if needed
### Manual install
## Usage
1. Click the ribbon icon to open the chat view
2. Type your message in the input box
3. Press Enter or click Send to send your message
4. Click the "New Chat" button to start a fresh conversation
## Supported Models
Any model supported by Ollama should work, including:
- llama3
- llama2
- mistral
- codellama
- etc.
## Development
To build from source:
If you prefer to install manually:
```bash
npm install
npm run build
```
Then copy the plugin into your vault:
```bash
mkdir -p /path/to/vault/.obsidian/plugins/ollama-plugin
cp manifest.json /path/to/vault/.obsidian/plugins/ollama-plugin/
cp -r dist /path/to/vault/.obsidian/plugins/ollama-plugin/
cp -r node_modules/chromadb /path/to/vault/.obsidian/plugins/ollama-plugin/node_modules/ # optional: only needed for semantic cache
```
> **Note:** The `obsidian` npm package is a dev-only type stub — Obsidian provides its own API at runtime. The `chromadb` package is only needed if you enable the semantic cache feature.
### After installation
1. Restart Obsidian (or reload: `Ctrl+Shift+P` → "Reload app without saving")
2. Go to **Settings → Community plugins** → enable **Ollama Plugin**
3. Configure the plugin at **Settings → Ollama Settings**
## Configuration
Open **Settings → Ollama Settings** to configure the plugin.
| Setting | Default | Description |
|---------|---------|-------------|
| Ollama URL | `http://localhost:11434` | Base URL of your Ollama instance |
| Model | `llama3` | Model used for chat responses |
| Vault Search Limit | `3` | Maximum number of vault entries to include in context |
| Max Message History | `50` | Maximum number of messages kept in conversation history |
| Enable Semantic Cache | Off | Cache responses for fast repeated queries |
| ChromaDB URL | `http://localhost:8000` | URL of your running ChromaDB instance |
| Cache Embedding Model | `nomic-embed-text` | Ollama model used to generate cache embeddings |
| Cache Similarity Threshold | `0.85` | Minimum cosine similarity (01) for a cache hit — higher values require closer matches |
| Clear Semantic Cache | — | Button to wipe all cached responses from ChromaDB |
## Usage
1. Open the chat view via the command palette (`Ctrl+P` → "Open Ollama Chat") or the ribbon icon
2. Type your message in the input box
3. Press **Enter** or click **Send** to send your message
4. Press **Shift+Enter** to insert a line break
5. Click **New Chat** to start a fresh conversation
## Semantic Cache Behaviour
- The cache is **bypassed** when tool calls are involved (e.g. file creation), since those requests have side effects.
- Responses are stored against the last user message in the conversation. If a new query is sufficiently similar (above the configured threshold), the cached response is returned.
- Re-asking the same question updates the existing cache entry rather than creating a duplicate.
- Use the **Clear Semantic Cache** button in settings to remove all stored responses (for example after switching embedding models).
## Vault Context
When you send a message, the plugin automatically searches your vault for relevant notes and includes them as context. The search uses weighted scoring:
- **Headings** — 5x weight
- **Frontmatter title** — 3x weight
- **Frontmatter tags** — 2.5x weight
- **First paragraph** — 1.5x weight
- **General content** — 1x weight
## Tools
The plugin exposes a `create_file` tool that allows the AI to create new markdown files in your vault. Paths are validated for safety (no `.obsidian`/`.git` access, no path traversal).
## Supported Models
Any Ollama-supported model works. Popular choices:
- `llama3`
- `llama2`
- `mistral`
- `codellama`
- and many more — see [ollama.com/library](https://ollama.com/library)
## Development
```bash
npm install
npm run build
npm test
```
## Troubleshooting
- **Connection issues**: Ensure Ollama is running and accessible at the configured URL
- **Model not found**: Make sure you've pulled the model (`ollama pull <modelname>`)
- **Permission issues**: Check that your Obsidian vault has proper write permissions
| Symptom | Likely cause | Fix |
|---------|-------------|-----|
| Plugin doesn't appear in Obsidian | Install script was not run or failed | Run `./install.sh /path/to/vault` and reload Obsidian |
| Cannot connect to Ollama | Ollama is not running | Run `ollama serve` |
| Model not found | Model not pulled | Run `ollama pull <model>` |
| Semantic cache unavailable (notice shown) | ChromaDB is not running, or the ChromaDB URL is wrong | Start ChromaDB (`chroma run`) and verify the URL in settings |
| Cache always misses | Similarity threshold is too high, or the embedding model was changed | Lower the threshold or click **Clear Semantic Cache** and let the cache rebuild |
| Slow first response after enabling cache | Embedding model not yet pulled | Run `ollama pull nomic-embed-text` (or the model you configured) |
| Permission issues | Vault write permissions | Check that your Obsidian vault has proper write permissions |
## Security
- File paths are validated to prevent access to `.obsidian/` and `.git/` directories
- Path traversal attempts (`..`) are blocked
- Absolute paths and Windows drive letters are rejected
- Maximum path length is enforced (200 characters)
## License
MIT
MIT License
Copyright (c) 2024 Flo Egger
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
Executable
+133
View File
@@ -0,0 +1,133 @@
#!/usr/bin/env bash
#
# install.sh — Install the Ollama Plugin into an Obsidian vault
#
# Usage:
# ./install.sh /path/to/your/vault
#
# The script will:
# 1. Install npm dependencies (excluding Ollama itself — you need that separately)
# 2. Compile the TypeScript plugin
# 3. Copy the built plugin into <vault>/.obsidian/plugins/ollama-plugin/
#
# Prerequisites:
# - Node.js and npm
# - Ollama installed and running (https://ollama.ai)
#
set -euo pipefail
# ── Helpers ──────────────────────────────────────────────────────────────────
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BOLD='\033[1m'
NC='\033[0m' # No Color
info() { printf "${GREEN}${NC} %s\n" "$1"; }
warn() { printf "${YELLOW}${NC} %s\n" "$1"; }
error() { printf "${RED}${NC} %s\n" "$1"; }
step() { printf "\n${BOLD}══ %s ══${NC}\n" "$1"; }
# ── Argument validation ──────────────────────────────────────────────────────
VAULT_PATH="${1:-}"
if [ -z "$VAULT_PATH" ]; then
echo "Usage: ./install.sh /path/to/your/vault"
echo ""
echo " Provide the path to your Obsidian vault directory."
echo " The plugin will be installed into <vault>/.obsidian/plugins/ollama-plugin/"
exit 1
fi
if [ ! -d "$VAULT_PATH" ]; then
error "Vault directory does not exist: $VAULT_PATH"
exit 1
fi
if [ ! -d "$VAULT_PATH/.obsidian" ]; then
warn "No .obsidian folder found in $VAULT_PATH — this may not be a valid Obsidian vault."
warn "Continuing anyway, but the plugin may not load."
fi
PLUGIN_DIR="$VAULT_PATH/.obsidian/plugins/ollama-plugin"
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
# ── Check prerequisites ──────────────────────────────────────────────────────
step "Checking prerequisites"
if ! command -v node &> /dev/null; then
error "Node.js is required but not found. Install it from https://nodejs.org"
exit 1
fi
info "Node.js $(node --version)"
if ! command -v npm &> /dev/null; then
error "npm is required but not found."
exit 1
fi
info "npm $(npm --version)"
# ── Install dependencies ─────────────────────────────────────────────────────
step "Installing npm dependencies"
cd "$SCRIPT_DIR"
npm install --production
info "Dependencies installed"
# ── Build ────────────────────────────────────────────────────────────────────
step "Building the plugin"
npm run build
info "Build complete"
# ── Install into vault ───────────────────────────────────────────────────────
step "Installing into vault"
mkdir -p "$PLUGIN_DIR"
# Copy manifest and built output
cp "$SCRIPT_DIR/manifest.json" "$PLUGIN_DIR/"
cp -r "$SCRIPT_DIR/dist" "$PLUGIN_DIR/"
# Copy only necessary runtime dependencies (obsidian and chromadb stubs)
# The obsidian package is only used for type definitions at build time —
# at runtime Obsidian provides its own API, so we don't need it in node_modules.
# chromadb is loaded dynamically and only needed when the cache is enabled.
mkdir -p "$PLUGIN_DIR/node_modules"
if [ -d "$SCRIPT_DIR/node_modules/chromadb" ]; then
cp -r "$SCRIPT_DIR/node_modules/chromadb" "$PLUGIN_DIR/node_modules/"
fi
info "Plugin installed to $PLUGIN_DIR"
# ── Verify ────────────────────────────────────────────────────────────────────
step "Verifying installation"
if [ -f "$PLUGIN_DIR/manifest.json" ] && [ -f "$PLUGIN_DIR/dist/main.js" ]; then
info "manifest.json ✓"
info "dist/main.js ✓"
else
error "Installation verification failed — missing files in $PLUGIN_DIR"
exit 1
fi
echo ""
echo " ${GREEN}${BOLD}Done!${NC} The plugin has been installed."
echo ""
echo " Next steps:"
echo " 1. Restart Obsidian (or reload plugins: Ctrl+Shift+P → 'Reload app without saving')"
echo " 2. Go to Settings → Community plugins → enable 'Ollama Plugin'"
echo " 3. Configure the plugin: Settings → Ollama Settings"
echo ""
echo " Make sure Ollama is running:"
echo " ollama serve"
echo " ollama pull llama3 (or your preferred model)"
echo ""
+2 -2
View File
@@ -6,8 +6,8 @@
"description": "Ollama integration plugin for Obsidian",
"author": "Anonymous",
"authorUrl": "",
"isDesktopOnly": false,
"main": "src/main.js",
"isDesktopOnly": true,
"main": "dist/main.js",
"authorization": [],
"permissions": [],
"defaultEnabled": true,
+1 -2
View File
@@ -2,7 +2,7 @@
"name": "ollama-plugin",
"version": "1.0.0",
"description": "Ollama integration plugin for Obsidian",
"main": "main.ts",
"main": "dist/main.js",
"scripts": {
"test": "jest",
"build": "tsc",
@@ -33,7 +33,6 @@
},
"dependencies": {
"chromadb": "^1.5.3",
"node-fetch": "^3.3.2",
"obsidian": "^1.4.11"
}
}
+4
View File
@@ -0,0 +1,4 @@
export interface Cache {
get(key: string): Promise<string | null>;
put(key: string, value: string): Promise<void>;
}
-412
View File
@@ -1,412 +0,0 @@
"use strict";
Object.defineProperty(exports, "__esModule", { value: true });
exports.ChatView = void 0;
const obsidian_1 = require("obsidian");
const MAX_STREAM_CHUNKS = 1000;
const ollama_client_1 = require("./ollama-client");
const vault_indexer_1 = require("./vault-indexer");
const tool_executor_1 = require("./tool-executor");
const error_handler_1 = require("./error-handler");
class ChatView extends obsidian_1.ItemView {
// Getters for testing
getSendButtonClickHandler() {
return this.sendButtonClickHandler;
}
getInputKeyDownHandler() {
return this.inputKeyDownHandler;
}
getNewChatButtonClickHandler() {
return this.newChatButtonClickHandler;
}
constructor(leaf, settings) {
super(leaf);
this.messages = [];
this.lastMessageEl = null;
this.newChatButton = null;
this.sendButton = null;
this.inputEl = null;
this.chatContainer = null;
this.sendButtonClickHandler = null;
this.inputKeyDownHandler = null;
this.newChatButtonClickHandler = null;
this.sendButtonClickWrapper = null;
this.inputKeyDownWrapper = null;
this.newChatButtonClickWrapper = null;
this.listenersAttached = false;
this.settings = settings;
this.ollamaClient = new ollama_client_1.OllamaClient(settings.ollamaUrl, settings.model);
this.vaultIndexer = new vault_indexer_1.VaultIndexer(this.app.vault);
this.toolExecutor = new tool_executor_1.ToolExecutor(this.app.vault, this.app);
}
updateSettings(newSettings) {
this.settings = newSettings;
this.ollamaClient = new ollama_client_1.OllamaClient(newSettings.ollamaUrl, newSettings.model);
}
getViewType() {
return 'ollama-chat-view';
}
getDisplayText() {
return 'Ollama Chat';
}
onOpen() {
this.render();
this.removeEventListeners(); // Clean up any existing listeners before reattaching
this.setupEventListeners();
return Promise.resolve();
}
onSettingsChange(newSettings) {
this.updateSettings(newSettings);
}
onClose() {
this.ollamaClient.cancelStream();
this.removeEventListeners();
this.cleanupStreamingResources();
this.lastMessageEl = null;
this.sendButton = null;
this.inputEl = null;
this.chatContainer = null;
return Promise.resolve();
}
cleanupStreamingResources() {
// Only cleanup if there's still an active streaming message
const streamingMessage = this.messages.find((msg) => msg.isStreaming);
if (streamingMessage && this.lastMessageEl && this.lastMessageEl.parentElement) {
this.lastMessageEl.parentElement.removeChild(this.lastMessageEl);
this.lastMessageEl = null;
}
}
render() {
const container = this.chatContainer || this.contentEl.createEl('div', { cls: 'ollama-chat-container' });
this.chatContainer = container;
const inputContainer = this.contentEl.querySelector('.ollama-input-container') ||
this.contentEl.createEl('div', { cls: 'ollama-input-container' });
if (!this.inputEl) {
this.inputEl = inputContainer.createEl('textarea', { cls: 'ollama-input' });
}
if (!this.sendButton) {
this.sendButton = inputContainer.createEl('button', {
cls: 'ollama-send-button',
});
this.sendButton.textContent = 'Send';
}
if (!this.newChatButton) {
const newChatContainer = this.contentEl.querySelector('.ollama-new-chat') ||
this.contentEl.createEl('div', { cls: 'ollama-new-chat' });
this.newChatButton = newChatContainer.createEl('button', {
cls: 'ollama-new-chat-button',
});
this.newChatButton.textContent = '🔄 New Chat';
this.newChatButton.title = 'Start a new conversation';
}
// Create immutable snapshot for rendering
const messagesSnapshot = [...this.messages];
// Only render messages that are not currently streaming
const nonStreamingMessages = messagesSnapshot.filter((msg) => !msg.isStreaming);
// Differential update: only update messages that have changed
const existingMessages = container.querySelectorAll('.ollama-message');
for (const msg of nonStreamingMessages) {
const existingEl = container.querySelector(`.ollama-message[data-msg-id="${msg.id}"]`);
if (existingEl) {
existingEl.textContent = msg.content;
}
else {
const messageEl = container.createEl('div', {
cls: `ollama-message ${msg.role}`,
});
messageEl.setAttribute('data-msg-id', msg.id);
messageEl.textContent = msg.content;
}
}
// Remove messages that are no longer in the array
for (const el of Array.from(existingMessages)) {
const id = el.getAttribute('data-msg-id');
if (!id || !nonStreamingMessages.some((m) => m.id === id)) {
el.remove();
}
}
// Re-attach streaming message if it exists
const streamingMessage = messagesSnapshot.find((msg) => msg.isStreaming);
if (streamingMessage && this.lastMessageEl) {
const existingStreamingEl = container.querySelector(`.ollama-message[data-msg-id="${streamingMessage.id}"]`);
if (!existingStreamingEl) {
container.appendChild(this.lastMessageEl);
}
}
}
setupEventListeners() {
if (!this.sendButton || !this.inputEl || this.listenersAttached)
return;
// Create handlers if they don't exist
if (!this.sendButtonClickHandler) {
this.sendButtonClickHandler = async () => {
if (!this.inputEl)
return;
await this.handleUserInput(this.inputEl.value);
this.inputEl.value = '';
};
}
if (!this.inputKeyDownHandler) {
this.inputKeyDownHandler = async (e) => {
if (!this.inputEl || e.key !== 'Enter' || e.shiftKey)
return;
e.preventDefault();
await this.handleUserInput(this.inputEl.value);
this.inputEl.value = '';
};
}
// Create wrapper functions for event listeners
this.sendButtonClickWrapper = () => {
void this.sendButtonClickHandler?.();
};
this.inputKeyDownWrapper = (e) => {
void this.inputKeyDownHandler?.(e);
};
this.newChatButtonClickWrapper = () => {
void this.newChatButtonClickHandler?.();
};
// Add event listeners using wrappers
this.sendButton.addEventListener('click', this.sendButtonClickWrapper);
this.inputEl.addEventListener('keydown', this.inputKeyDownWrapper);
if (this.newChatButton) {
if (!this.newChatButtonClickHandler) {
this.newChatButtonClickHandler = () => this.clearConversation();
}
this.newChatButton.addEventListener('click', this.newChatButtonClickWrapper);
}
this.listenersAttached = true;
}
removeEventListeners() {
if (this.sendButton && this.sendButtonClickWrapper) {
this.sendButton.removeEventListener('click', this.sendButtonClickWrapper);
}
if (this.inputEl && this.inputKeyDownWrapper) {
this.inputEl.removeEventListener('keydown', this.inputKeyDownWrapper);
}
if (this.newChatButton && this.newChatButtonClickWrapper) {
this.newChatButton.removeEventListener('click', this.newChatButtonClickWrapper);
}
this.sendButtonClickWrapper = null;
this.inputKeyDownWrapper = null;
this.newChatButtonClickWrapper = null;
this.listenersAttached = false;
}
clearConversation() {
// Create new array to ensure immutability
this.messages = [];
this.lastMessageEl = null;
this.render();
new obsidian_1.Notice('Conversation cleared');
}
updateMessageById(id, partial) {
const index = this.messages.findIndex((m) => m.id === id);
if (index < 0)
return false;
this.messages = [
...this.messages.slice(0, index),
{ ...this.messages[index], ...partial },
...this.messages.slice(index + 1),
];
return true;
}
updateLastMessage(content) {
const streamingMessage = this.messages.find((msg) => msg.isStreaming);
if (streamingMessage && !this.lastMessageEl) {
this.lastMessageEl = (this.chatContainer ?? this.contentEl).createEl('div', {
cls: `ollama-message assistant`,
});
this.lastMessageEl.setAttribute('data-msg-id', streamingMessage.id);
}
if (this.lastMessageEl) {
this.lastMessageEl.textContent = content;
}
}
getTools() {
return [
{
type: 'function',
function: {
name: 'create_file',
description: 'Create a new file in the vault',
parameters: {
type: 'object',
properties: {
path: {
type: 'string',
description: "Relative path within the vault, e.g. 'Notes/todo.md'",
},
content: { type: 'string', description: 'Content of the file to create' },
},
required: ['path', 'content'],
},
},
},
];
}
buildMessages(userMessage, context) {
const systemContent = context
? `You are a helpful assistant.\n\nRelevant vault context:\n${context}`
: 'You are a helpful assistant.';
const systemMessage = {
role: 'system',
content: systemContent,
};
const userMessageWithContext = {
role: 'user',
content: userMessage,
};
return [
systemMessage,
...this.messages.map((m) => ({
role: m.role,
content: m.content,
tool_calls: m.tool_calls,
})),
userMessageWithContext,
];
}
async processToolCalls(toolCalls, messages, tools, fullResponse, assistantMessageId) {
// Validate tool calls before processing
const MAX_TOOL_CALLS = 10;
if (toolCalls.length > MAX_TOOL_CALLS) {
throw new Error(`Too many tool calls (max ${MAX_TOOL_CALLS})`);
}
// Collect all tool results using allSettled to support partial results
const settledResults = await Promise.allSettled(toolCalls.map((call) => this.toolExecutor.handleToolCall(call)));
const toolResults = [];
for (const result of settledResults) {
if (result.status === 'fulfilled') {
toolResults.push(result.value);
}
else {
// Use centralized error handler for tool errors
error_handler_1.ErrorHandler.handleError(result.reason, 'ChatView.handleUserInput');
}
}
// Only create follow-up when we have tool results
if (toolResults.length > 0) {
// Create follow-up messages including the assistant's tool calls and results
const followUpMessages = [
...messages,
{ role: 'assistant', content: fullResponse, tool_calls: toolCalls },
...toolResults.map((result) => ({
role: 'tool',
content: JSON.stringify(result),
})),
];
const followUp = await this.ollamaClient.chat(followUpMessages, tools);
fullResponse += followUp.content;
this.updateLastMessage(fullResponse);
// Update the assistant message with the final response immutably
this.updateMessageById(assistantMessageId, {
content: fullResponse,
isStreaming: false,
});
}
else {
// Even if no tool results were successful, mark streaming as complete
// to prevent the assistant message from disappearing
this.updateMessageById(assistantMessageId, {
content: fullResponse,
isStreaming: false,
});
}
}
async handleUserInput(content) {
if (!this.sendButton || !this.inputEl)
return;
this.sendButton.disabled = true;
try {
// Guard against empty messages
const userMessage = content.trim();
if (!userMessage)
return;
// Search vault using user message as query
const entries = await this.vaultIndexer.searchVault(userMessage, this.settings.vaultSearchLimit);
let context = entries.map((entry) => `### ${entry.title}\n${entry.content}`).join('\n\n');
// Cap context size to prevent prompt bloat with large vaults
const MAX_CONTEXT_LENGTH = 4000;
if (context.length > MAX_CONTEXT_LENGTH) {
context = context.substring(0, MAX_CONTEXT_LENGTH) + '\n\n... (truncated)';
}
const messages = this.buildMessages(userMessage, context);
const tools = this.getTools();
const messageId = `${Date.now()}-${Math.random().toString(36).slice(2, 8)}`;
const userMessageId = messageId;
const assistantMessageId = `${messageId}-assistant`;
// Store user message in conversation history
const userChatMessage = {
id: userMessageId,
role: 'user',
content: userMessage,
timestamp: Date.now(),
};
const assistantMessage = {
id: assistantMessageId,
role: 'assistant',
content: '',
timestamp: Date.now(),
isStreaming: true,
};
// Update messages immutably
this.messages = [...this.messages, userChatMessage, assistantMessage];
try {
this.render();
const stream = this.ollamaClient.streamChat(messages, tools);
let fullResponse = '';
let toolCalls = [];
let chunkCount = 0;
for await (const chunk of stream) {
chunkCount++;
if (chunkCount > MAX_STREAM_CHUNKS) {
throw new Error('Response too long, stopped streaming');
}
if (chunk.content) {
fullResponse += chunk.content;
}
if (chunk.tool_calls) {
toolCalls = toolCalls.concat(chunk.tool_calls);
}
this.updateLastMessage(fullResponse);
}
// Update the assistant message with the full response immutably
this.updateMessageById(assistantMessageId, {
content: fullResponse,
tool_calls: toolCalls,
});
// Process tool calls with proper follow-up context
if (toolCalls.length > 0) {
await this.processToolCalls(toolCalls, messages, tools, fullResponse, assistantMessageId);
}
// Update assistant message immutably — only if no tool calls were processed
if (toolCalls.length === 0) {
this.updateMessageById(assistantMessageId, {
isStreaming: false,
});
}
// Limit conversation history to prevent memory issues
if (this.messages.length > this.settings.maxMessageHistory) {
this.messages = this.messages.slice(-this.settings.maxMessageHistory);
}
this.render();
}
finally {
// Clean up streaming resources regardless of outcome
this.cleanupStreamingResources();
}
}
catch (error) {
// Use centralized error handler
error_handler_1.ErrorHandler.handleError(error, 'ChatView.handleUserInput');
// Update any streaming messages to non-streaming state to prevent stale messages
this.messages = this.messages.map((msg) => msg.isStreaming ? { ...msg, isStreaming: false } : msg);
this.cleanupStreamingResources();
this.render();
}
finally {
if (this.sendButton) {
this.sendButton.disabled = false;
}
}
}
}
exports.ChatView = ChatView;
+370 -318
View File
@@ -1,58 +1,42 @@
import { ItemView, WorkspaceLeaf, Notice } from 'obsidian';
/// <reference lib="dom" />
// Use global types from JSDOM setup
type KeyboardEvent = globalThis.KeyboardEvent;
type HTMLTextAreaElement = globalThis.HTMLTextAreaElement;
type HTMLButtonElement = globalThis.HTMLButtonElement;
const MAX_STREAM_CHUNKS = 1000;
import {
PluginSettings,
OllamaMessage,
ChatMessage,
OllamaTool,
ToolCall,
ToolResult,
} from './types';
import { ItemView, Notice, WorkspaceLeaf } from 'obsidian';
import { OllamaClient } from './ollama-client';
import { VaultIndexer } from './vault-indexer';
import { ToolExecutor } from './tool-executor';
import { PluginSettings, OllamaMessage, OllamaTool, OllamaToolCall, ChatMessage } from './types';
import { ConversationStateManager } from './conversation-state';
import { ErrorHandler } from './error-handler';
export class ChatView extends ItemView {
private settings: PluginSettings;
private messages: ChatMessage[] = [];
private ollamaClient: OllamaClient;
private vaultIndexer: VaultIndexer;
private toolExecutor: ToolExecutor;
private lastMessageEl: HTMLElement | null = null;
private newChatButton: HTMLButtonElement | null = null;
private sendButton: HTMLButtonElement | null = null;
private inputEl: HTMLTextAreaElement | null = null;
private chatContainer: HTMLElement | null = null;
private sendButtonClickHandler: (() => Promise<void>) | null = null;
private inputKeyDownHandler: ((e: KeyboardEvent) => Promise<void>) | null = null;
private newChatButtonClickHandler: (() => void) | null = null;
private sendButtonClickWrapper: (() => void) | null = null;
private inputKeyDownWrapper: ((e: KeyboardEvent) => void) | null = null;
private newChatButtonClickWrapper: (() => void) | null = null;
private listenersAttached = false;
export const VIEW_TYPE_OLLAMA_CHAT = 'ollama-chat-view';
export class ChatView extends ItemView {
// Getters for testing
public getSendButtonClickHandler(): (() => Promise<void>) | null {
getSendButtonClickHandler() {
return this.sendButtonClickHandler;
}
public getInputKeyDownHandler(): ((e: KeyboardEvent) => Promise<void>) | null {
getInputKeyDownHandler() {
return this.inputKeyDownHandler;
}
public getNewChatButtonClickHandler(): (() => void) | null {
getNewChatButtonClickHandler() {
return this.newChatButtonClickHandler;
}
constructor(leaf: WorkspaceLeaf, settings: PluginSettings) {
super(leaf);
this.messages = [];
this.lastMessageEl = null;
this.newChatButton = null;
this.sendButton = null;
this.inputEl = null;
this.chatContainer = null;
this.sendButtonClickHandler = null;
this.inputKeyDownHandler = null;
this.newChatButtonClickHandler = null;
this.sendButtonClickWrapper = null;
this.inputKeyDownWrapper = null;
this.newChatButtonClickWrapper = null;
this.listenersAttached = false;
this.settings = settings;
this.ollamaClient = new OllamaClient(
settings.ollamaUrl,
@@ -62,9 +46,10 @@ export class ChatView extends ItemView {
);
this.vaultIndexer = new VaultIndexer(this.app.vault);
this.toolExecutor = new ToolExecutor(this.app.vault, this.app);
this.conversationStateManager = new ConversationStateManager();
}
public updateSettings(newSettings: PluginSettings): void {
updateSettings(newSettings: PluginSettings) {
this.settings = newSettings;
this.ollamaClient = new OllamaClient(
newSettings.ollamaUrl,
@@ -72,6 +57,15 @@ export class ChatView extends ItemView {
undefined,
newSettings.cacheConfig
);
void this.ollamaClient.initializeCache().catch(() => {
new Notice(
'Semantic cache unavailable: could not connect to ChromaDB. Check the ChromaDB URL in settings.'
);
});
}
async clearCache(): Promise<void> {
await this.ollamaClient.clearCache();
}
getViewType(): string {
@@ -83,13 +77,19 @@ export class ChatView extends ItemView {
}
async onOpen(): Promise<void> {
await this.ollamaClient.initializeCache();
try {
await this.ollamaClient.initializeCache();
} catch {
new Notice(
'Semantic cache unavailable: could not connect to ChromaDB. Check the ChromaDB URL in settings.'
);
}
this.render();
this.removeEventListeners(); // Clean up any existing listeners before reattaching
this.setupEventListeners();
}
public onSettingsChange(newSettings: PluginSettings): void {
onSettingsChange(newSettings: PluginSettings): void {
this.updateSettings(newSettings);
}
@@ -104,7 +104,7 @@ export class ChatView extends ItemView {
return Promise.resolve();
}
private cleanupStreamingResources(): void {
cleanupStreamingResources(): void {
// Only cleanup if there's still an active streaming message
const streamingMessage = this.messages.find((msg) => msg.isStreaming);
if (streamingMessage && this.lastMessageEl && this.lastMessageEl.parentElement) {
@@ -113,57 +113,21 @@ export class ChatView extends ItemView {
}
}
render() {
render(): void {
const container =
this.chatContainer || this.contentEl.createEl('div', { cls: 'ollama-chat-container' });
this.chatContainer = container;
const inputContainer =
this.contentEl.querySelector('.ollama-input-container') ||
this.contentEl.createEl('div', { cls: 'ollama-input-container' });
const newChatContainer =
this.contentEl.querySelector('.ollama-new-chat-container') ||
this.contentEl.createEl('div', { cls: 'ollama-new-chat-container' });
if (!this.inputEl) {
this.inputEl = inputContainer.createEl('textarea', { cls: 'ollama-input' });
}
if (!this.sendButton) {
this.sendButton = inputContainer.createEl('button', {
cls: 'ollama-send-button',
});
this.sendButton.textContent = 'Send';
}
if (!this.newChatButton) {
const newChatContainer =
this.contentEl.querySelector('.ollama-new-chat') ||
this.contentEl.createEl('div', { cls: 'ollama-new-chat' });
this.newChatButton = newChatContainer.createEl('button', {
cls: 'ollama-new-chat-button',
});
this.newChatButton.textContent = '🔄 New Chat';
this.newChatButton.title = 'Start a new conversation';
}
// Create immutable snapshot for rendering
const messagesSnapshot = [...this.messages];
// Only render messages that are not currently streaming
const nonStreamingMessages = messagesSnapshot.filter((msg) => !msg.isStreaming);
// Differential update: only update messages that have changed
const existingMessages = container.querySelectorAll('.ollama-message');
for (const msg of nonStreamingMessages) {
const existingEl = container.querySelector(`.ollama-message[data-msg-id="${msg.id}"]`);
if (existingEl) {
existingEl.textContent = msg.content;
} else {
const messageEl = container.createEl('div', {
cls: `ollama-message ${msg.role}`,
}) as HTMLElement;
messageEl.setAttribute('data-msg-id', msg.id);
messageEl.textContent = msg.content;
}
}
// Remove messages that are no longer in the array
for (const el of Array.from(existingMessages)) {
const id = el.getAttribute('data-msg-id');
@@ -172,6 +136,23 @@ export class ChatView extends ItemView {
}
}
// Render non-streaming messages
for (const msg of nonStreamingMessages) {
const existingEl = container.querySelector(`.ollama-message[data-msg-id="${msg.id}"]`);
if (existingEl) {
const contentEl = existingEl.querySelector('.ollama-message-content');
if (contentEl) {
contentEl.textContent = msg.content;
}
} else {
const messageEl = container.createEl('div', { cls: 'ollama-message' });
messageEl.setAttribute('data-msg-id', msg.id);
messageEl.createEl('div', { cls: 'ollama-message-role', text: msg.role });
const contentEl = messageEl.createEl('div', { cls: 'ollama-message-content' });
contentEl.textContent = msg.content;
}
}
// Re-attach streaming message if it exists
const streamingMessage = messagesSnapshot.find((msg) => msg.isStreaming);
if (streamingMessage && this.lastMessageEl) {
@@ -182,322 +163,393 @@ export class ChatView extends ItemView {
container.appendChild(this.lastMessageEl);
}
}
// Setup new chat button
if (!this.newChatButton) {
this.newChatButton = newChatContainer.createEl('button', {
cls: 'ollama-new-chat-button',
text: 'New Chat',
});
} else {
newChatContainer.appendChild(this.newChatButton);
}
// Setup input area
if (!this.inputEl) {
this.inputEl = inputContainer.createEl('textarea', {
cls: 'ollama-input',
attr: { placeholder: 'Type your message...' },
});
} else {
inputContainer.appendChild(this.inputEl);
}
// Setup send button
if (!this.sendButton) {
this.sendButton = inputContainer.createEl('button', {
cls: 'ollama-send-button',
text: 'Send',
});
} else {
inputContainer.appendChild(this.sendButton);
}
// Append containers to contentEl
this.contentEl.appendChild(newChatContainer);
this.contentEl.appendChild(inputContainer);
this.contentEl.appendChild(container);
// Focus input on open
this.inputEl.focus();
}
private setupEventListeners(): void {
if (!this.sendButton || !this.inputEl || this.listenersAttached) return;
// Create handlers if they don't exist
if (!this.sendButtonClickHandler) {
this.sendButtonClickHandler = async () => {
if (!this.inputEl) return;
await this.handleUserInput(this.inputEl.value);
this.inputEl.value = '';
};
setupEventListeners(): void {
if (this.listenersAttached) {
return;
}
if (!this.inputKeyDownHandler) {
this.inputKeyDownHandler = async (e: KeyboardEvent) => {
if (!this.inputEl || e.key !== 'Enter' || e.shiftKey) return;
e.preventDefault();
await this.handleUserInput(this.inputEl.value);
this.inputEl.value = '';
};
}
// Create wrapper functions for event listeners
this.sendButtonClickWrapper = () => {
void this.sendButtonClickHandler?.();
};
this.inputKeyDownWrapper = (e: KeyboardEvent) => {
void this.inputKeyDownHandler?.(e);
};
this.newChatButtonClickWrapper = () => {
void this.newChatButtonClickHandler?.();
this.sendButtonClickHandler = () => {
void this.handleUserInput(this.inputEl?.value);
};
// Add event listeners using wrappers
this.sendButton.addEventListener('click', this.sendButtonClickWrapper);
this.inputEl.addEventListener('keydown', this.inputKeyDownWrapper);
if (this.newChatButton) {
if (!this.newChatButtonClickHandler) {
this.newChatButtonClickHandler = () => this.clearConversation();
this.inputKeyDownHandler = (event: KeyboardEvent) => {
if (event.key === 'Enter' && !event.shiftKey) {
event.preventDefault();
void this.handleUserInput(this.inputEl?.value);
}
this.newChatButton.addEventListener('click', this.newChatButtonClickWrapper);
};
this.newChatButtonClickHandler = () => {
this.clearConversation();
};
if (this.sendButton && this.sendButtonClickHandler) {
this.sendButton.addEventListener('click', this.sendButtonClickHandler);
}
if (this.inputEl && this.inputKeyDownHandler) {
this.inputEl.addEventListener('keydown', this.inputKeyDownHandler);
}
if (this.newChatButton && this.newChatButtonClickHandler) {
this.newChatButton.addEventListener('click', this.newChatButtonClickHandler);
}
this.listenersAttached = true;
}
private removeEventListeners(): void {
if (this.sendButton && this.sendButtonClickWrapper) {
this.sendButton.removeEventListener('click', this.sendButtonClickWrapper);
removeEventListeners(): void {
if (!this.listenersAttached) {
return;
}
if (this.inputEl && this.inputKeyDownWrapper) {
this.inputEl.removeEventListener('keydown', this.inputKeyDownWrapper);
if (this.sendButton && this.sendButtonClickHandler) {
this.sendButton.removeEventListener('click', this.sendButtonClickHandler);
}
if (this.newChatButton && this.newChatButtonClickWrapper) {
this.newChatButton.removeEventListener('click', this.newChatButtonClickWrapper);
if (this.inputEl && this.inputKeyDownHandler) {
this.inputEl.removeEventListener('keydown', this.inputKeyDownHandler);
}
this.sendButtonClickWrapper = null;
this.inputKeyDownWrapper = null;
this.newChatButtonClickWrapper = null;
if (this.newChatButton && this.newChatButtonClickHandler) {
this.newChatButton.removeEventListener('click', this.newChatButtonClickHandler);
}
this.listenersAttached = false;
}
private clearConversation(): void {
// Create new array to ensure immutability
clearConversation(): void {
this.messages = [];
this.lastMessageEl = null;
this.conversationStateManager.clear();
this.render();
new Notice('Conversation cleared');
}
private updateMessageById(id: string, partial: Partial<ChatMessage>): boolean {
updateMessageById(id: string, updates: Partial<ChatMessage>): void {
const index = this.messages.findIndex((m) => m.id === id);
if (index < 0) return false;
this.messages = [
...this.messages.slice(0, index),
{ ...this.messages[index], ...partial },
...this.messages.slice(index + 1),
];
return true;
if (index !== -1) {
this.messages[index] = { ...this.messages[index], ...updates };
this.render();
}
}
private updateLastMessage(content: string) {
updateLastMessage(updates: Partial<ChatMessage>): void {
const streamingMessage = this.messages.find((msg) => msg.isStreaming);
if (streamingMessage && !this.lastMessageEl) {
this.lastMessageEl = (this.chatContainer ?? this.contentEl).createEl('div', {
cls: `ollama-message assistant`,
}) as HTMLElement;
this.lastMessageEl.setAttribute('data-msg-id', streamingMessage.id);
}
if (this.lastMessageEl) {
this.lastMessageEl.textContent = content;
if (streamingMessage) {
const index = this.messages.findIndex((msg) => msg.id === streamingMessage.id);
if (index !== -1) {
this.messages[index] = { ...this.messages[index], ...updates };
this.render();
}
}
}
private getTools(): OllamaTool[] {
getTools(): OllamaTool[] {
return [
{
type: 'function',
function: {
name: 'create_file',
description: 'Create a new file in the vault',
name: 'read_vault_file',
description: 'Reads the content of a file from the vault',
parameters: {
type: 'object' as const,
type: 'object',
properties: {
path: {
type: 'string' as const,
description: "Relative path within the vault, e.g. 'Notes/todo.md'",
type: 'string',
description: 'The path to the file to read',
},
content: {
type: 'string',
description: 'The content of the file to read',
},
content: { type: 'string' as const, description: 'Content of the file to create' },
},
required: ['path', 'content'],
required: ['path'],
},
},
},
{
type: 'function',
function: {
name: 'search_vault_files',
description: 'Searches for files in the vault that match a given query',
parameters: {
type: 'object',
properties: {
query: {
type: 'string',
description: 'The search query to use',
},
limit: {
type: 'number',
description: 'The maximum number of results to return',
},
},
required: ['query'],
},
},
},
];
}
private buildMessages(userMessage: string, context: string): OllamaMessage[] {
const systemContent = context
? `You are a helpful assistant.\n\nRelevant vault context:\n${context}`
: 'You are a helpful assistant.';
buildMessages(userMessageContent: string, tools?: OllamaTool[]): OllamaMessage[] {
const systemContent = `You are an assistant that can help answer questions using the contents of a vault.
The user can ask questions about their vault contents, and you should provide helpful responses based on the files.
When a user asks for information, try to find relevant files using the search_vault_files tool and read their contents with the read_vault_file tool.
Only use the tools if you need to access vault content that is not already in the context.`;
const systemMessage: OllamaMessage = {
role: 'system',
content: systemContent,
};
const userMessageWithContext: OllamaMessage = {
const userMessage: OllamaMessage = {
role: 'user',
content: userMessage,
content: userMessageContent,
};
return [
systemMessage,
...this.messages.map(
(m) =>
({
role: m.role,
content: m.content,
tool_calls: m.tool_calls,
}) as OllamaMessage
),
userMessageWithContext,
];
const messages: OllamaMessage[] = [systemMessage, userMessage];
if (tools && tools.length > 0) {
messages.push({
role: 'assistant',
content: 'I have access to the following tools to help answer your questions:',
});
}
return messages;
}
private async processToolCalls(
toolCalls: ToolCall[],
async processToolCalls(
toolCalls: OllamaToolCall[],
messages: OllamaMessage[],
tools: OllamaTool[],
fullResponse: string,
assistantMessageId: string
): Promise<void> {
// Validate tool calls before processing
const MAX_TOOL_CALLS = 10;
if (toolCalls.length > MAX_TOOL_CALLS) {
throw new Error(`Too many tool calls (max ${MAX_TOOL_CALLS})`);
}
const toolResults = (
await Promise.all(
toolCalls.slice(0, MAX_TOOL_CALLS).map(async (toolCall) => {
try {
const toolResult = await this.toolExecutor.handleToolCall(toolCall);
return { ...toolResult, id: toolCall.id };
} catch (error) {
ErrorHandler.handleError(error, 'ChatView.handleUserInput');
return null;
}
})
)
).filter((result): result is NonNullable<typeof result> => result !== null);
// Collect all tool results using allSettled to support partial results
const settledResults = await Promise.allSettled(
toolCalls.map((call) => this.toolExecutor.handleToolCall(call))
);
const followUpMessages: OllamaMessage[] = toolResults.map((result) => {
return {
role: 'tool',
content: JSON.stringify(result),
tool_call_id: result.id ?? '',
};
});
const toolResults: ToolResult[] = [];
for (const result of settledResults) {
if (result.status === 'fulfilled') {
toolResults.push(result.value);
} else {
// Use centralized error handler for tool errors
ErrorHandler.handleError(result.reason, 'ChatView.handleUserInput');
}
}
const followUp: OllamaMessage = {
role: 'assistant',
content: 'I have processed your request using the following tools. Here are the results:',
tool_calls: toolCalls,
};
// Only create follow-up when we have tool results
if (toolResults.length > 0) {
// Create follow-up messages including the assistant's tool calls and results
const followUpMessages: OllamaMessage[] = [
...messages,
{ role: 'assistant' as const, content: fullResponse, tool_calls: toolCalls },
...toolResults.map((result) => ({
role: 'tool' as const,
content: JSON.stringify(result),
})),
];
const followUp = await this.ollamaClient.chat(followUpMessages, tools);
fullResponse += followUp.content;
this.updateLastMessage(fullResponse);
// Update the assistant message with the final response immutably
if (followUpMessages.length > 0) {
const finalMessages = [...messages, followUp, ...followUpMessages];
const response = await this.ollamaClient.chat(finalMessages, tools);
const finalResponse = response.content || fullResponse;
this.updateMessageById(assistantMessageId, {
content: fullResponse,
isStreaming: false,
});
} else {
// Even if no tool results were successful, mark streaming as complete
// to prevent the assistant message from disappearing
this.updateMessageById(assistantMessageId, {
content: fullResponse,
content: finalResponse,
isStreaming: false,
});
}
}
private async handleUserInput(content: string) {
if (!this.sendButton || !this.inputEl) return;
this.sendButton.disabled = true;
async handleUserInput(inputValue?: string): Promise<void> {
const userMessage = (inputValue ?? this.inputEl?.value ?? '').trim();
if (!userMessage) {
return;
}
const MAX_CONTEXT_LENGTH = 2000;
const tools = this.getTools();
const messageId = crypto.randomUUID();
const userMessageId = `${messageId}-user`;
const assistantMessageId = `${messageId}-assistant`;
const userChatMessage: ChatMessage = {
id: userMessageId,
role: 'user',
content: userMessage,
timestamp: Date.now(),
};
const assistantMessage: ChatMessage = {
id: assistantMessageId,
role: 'assistant',
content: '',
timestamp: Date.now(),
isStreaming: true,
};
const previousStreamingEl = this.lastMessageEl;
this.messages = [...this.messages, userChatMessage, assistantMessage];
this.render();
if (this.inputEl) {
this.inputEl.value = '';
}
// Add the assistant message to the DOM to enable streaming
this.lastMessageEl =
this.chatContainer?.querySelector(`.ollama-message[data-msg-id="${assistantMessageId}"]`) ??
null;
if (!this.lastMessageEl && previousStreamingEl) {
previousStreamingEl.classList.add('ollama-message');
previousStreamingEl.setAttribute('data-msg-id', assistantMessageId);
this.contentEl.appendChild(previousStreamingEl);
this.lastMessageEl = previousStreamingEl;
}
try {
// Guard against empty messages
const userMessage = content.trim();
if (!userMessage) return;
const entries = await this.vaultIndexer.searchVault(userMessage, this.settings.vaultSearchLimit);
const context = entries
.map((entry) => `${entry.title}\n${entry.content}`)
.join('\n\n')
.slice(0, MAX_CONTEXT_LENGTH);
const userMessageWithContext = context
? `Relevant vault context:\n${context}\n\nUser question:\n${userMessage}`
: userMessage;
// Search vault using user message as query
const entries = await this.vaultIndexer.searchVault(
userMessage,
this.settings.vaultSearchLimit
);
let context = entries.map((entry) => `### ${entry.title}\n${entry.content}`).join('\n\n');
// Get the complete messages array for the LLM with all context layers
const completeMessages = this.conversationStateManager.getCompleteMessages(userMessageWithContext);
// Cap context size to prevent prompt bloat with large vaults
const MAX_CONTEXT_LENGTH = 4000;
if (context.length > MAX_CONTEXT_LENGTH) {
context = context.substring(0, MAX_CONTEXT_LENGTH) + '\n\n... (truncated)';
}
const stream = this.ollamaClient.streamChat(completeMessages, tools);
const messages = this.buildMessages(userMessage, context);
const tools = this.getTools();
let fullResponse = '';
let toolCalls: OllamaToolCall[] = [];
let chunkCount = 0;
const messageId = `${Date.now()}-${Math.random().toString(36).slice(2, 8)}`;
const userMessageId = messageId;
const assistantMessageId = `${messageId}-assistant`;
// Store user message in conversation history
const userChatMessage: ChatMessage = {
id: userMessageId,
role: 'user' as const,
content: userMessage,
timestamp: Date.now(),
};
const assistantMessage: ChatMessage = {
id: assistantMessageId,
role: 'assistant' as const,
content: '',
timestamp: Date.now(),
isStreaming: true,
};
// Update messages immutably
this.messages = [...this.messages, userChatMessage, assistantMessage];
try {
this.render();
const stream = this.ollamaClient.streamChat(messages, tools);
let fullResponse = '';
let toolCalls: ToolCall[] = [];
let chunkCount = 0;
for await (const chunk of stream) {
chunkCount++;
if (chunkCount > MAX_STREAM_CHUNKS) {
throw new Error('Response too long, stopped streaming');
}
if (chunk.content) {
fullResponse += chunk.content;
}
if (chunk.tool_calls) {
toolCalls = toolCalls.concat(chunk.tool_calls);
}
this.updateLastMessage(fullResponse);
}
// Update the assistant message with the full response immutably
this.updateMessageById(assistantMessageId, {
content: fullResponse,
tool_calls: toolCalls,
});
// Process tool calls with proper follow-up context
if (toolCalls.length > 0) {
await this.processToolCalls(toolCalls, messages, tools, fullResponse, assistantMessageId);
}
// Update assistant message immutably — only if no tool calls were processed
if (toolCalls.length === 0) {
this.updateMessageById(assistantMessageId, {
isStreaming: false,
for await (const chunk of stream) {
if (chunk.content) {
fullResponse += chunk.content;
this.updateLastMessage({
content: fullResponse,
isStreaming: true,
});
}
// Limit conversation history to prevent memory issues
if (this.messages.length > this.settings.maxMessageHistory) {
this.messages = this.messages.slice(-this.settings.maxMessageHistory);
if (chunk.tool_calls) {
toolCalls = [...toolCalls, ...chunk.tool_calls];
}
chunkCount++;
if (chunkCount > MAX_STREAM_CHUNKS) {
break;
}
this.render();
} finally {
// Clean up streaming resources regardless of outcome
this.cleanupStreamingResources();
}
} catch (error) {
// Use centralized error handler
ErrorHandler.handleError(error, 'ChatView.handleUserInput');
// Update any streaming messages to non-streaming state to prevent stale messages
this.messages = this.messages.map((msg) =>
msg.isStreaming ? { ...msg, isStreaming: false } : msg
);
this.cleanupStreamingResources();
// Process tool calls if any
if (toolCalls.length > 0) {
await this.processToolCalls(
toolCalls,
completeMessages,
tools,
fullResponse,
assistantMessageId
);
}
// Update assistant message immutably — only if no tool calls were processed
if (toolCalls.length === 0) {
this.updateMessageById(assistantMessageId, {
content: fullResponse,
isStreaming: false,
});
}
// Update conversation state with the assistant's response
this.conversationStateManager.updateShortTermContext({ role: 'user', content: userMessage });
this.conversationStateManager.updateShortTermContext({
role: 'assistant',
content: fullResponse,
});
// Limit conversation history to prevent memory issues
if (this.messages.length > this.settings.maxMessageHistory) {
this.messages = this.messages.slice(-this.settings.maxMessageHistory);
}
this.render();
} catch (error) {
ErrorHandler.handleError(error, 'ChatView.handleUserInput');
this.updateMessageById(assistantMessageId, {
content: 'An error occurred while processing your request.',
isStreaming: false,
});
} finally {
if (this.sendButton) {
this.sendButton.disabled = false;
}
// Clean up streaming resources regardless of outcome
this.cleanupStreamingResources();
}
}
// State
private messages: ChatMessage[] = [];
private lastMessageEl: HTMLElement | null = null;
private newChatButton: HTMLElement | null = null;
private sendButton: HTMLElement | null = null;
private inputEl: HTMLTextAreaElement | null = null;
private chatContainer: HTMLElement | null = null;
private sendButtonClickHandler: (() => void) | null = null;
private inputKeyDownHandler: ((event: KeyboardEvent) => void) | null = null;
private newChatButtonClickHandler: (() => void) | null = null;
private sendButtonClickWrapper: (() => void) | null = null;
private inputKeyDownWrapper: ((event: KeyboardEvent) => void) | null = null;
private newChatButtonClickWrapper: (() => void) | null = null;
private listenersAttached: boolean = false;
private settings: PluginSettings;
private ollamaClient: OllamaClient;
private vaultIndexer: VaultIndexer;
private toolExecutor: ToolExecutor;
private conversationStateManager: ConversationStateManager;
}
const MAX_STREAM_CHUNKS = 1000;
const MAX_TOOL_CALLS = 5;
-11
View File
@@ -1,11 +0,0 @@
"use strict";
// Default plugin settings
Object.defineProperty(exports, "__esModule", { value: true });
exports.DEFAULT_SETTINGS = void 0;
exports.DEFAULT_SETTINGS = {
ollamaUrl: 'http://localhost:11434',
model: 'llama3',
vaultSearchLimit: 3,
maxMessageHistory: 50,
lastIndexTime: 0,
};
+1 -2
View File
@@ -1,5 +1,3 @@
// Default plugin settings
export const DEFAULT_SETTINGS = {
ollamaUrl: 'http://localhost:11434',
model: 'llama3',
@@ -11,5 +9,6 @@ export const DEFAULT_SETTINGS = {
similarityThreshold: 0.85,
collectionName: 'ollama_semantic_cache',
embeddingModel: 'nomic-embed-text',
chromaURL: 'http://localhost:8000',
},
};
+171
View File
@@ -0,0 +1,171 @@
// src/conversation-state.ts
import type { OllamaMessage } from './types';
export interface ConversationState {
shortTermContext: OllamaMessage[];
mediumTermContext: OllamaMessage[];
longTermContext: OllamaMessage[];
}
export class ConversationStateManager {
private shortTermContext: OllamaMessage[] = [];
private mediumTermContext: OllamaMessage[] = [];
private longTermContext: OllamaMessage[] = [];
private maxShortTermTurns: number = 10;
private maxMediumTermMessages: number = 20;
constructor() {
// Initialize with default system context
this.longTermContext = [
{
role: 'system',
content: `You are an assistant that can help answer questions using the contents of a vault.
The user can ask questions about their vault contents, and you should provide helpful responses based on the files.
When a user asks for information, try to find relevant files using the search_vault_files tool and read their contents with the read_vault_file tool.
Only use the tools if you need to access vault content that is not already in the context.`,
},
];
}
/**
* Updates the short-term context with a new message
* @param message The message to add to short-term context
*/
updateShortTermContext(message: OllamaMessage): void {
// Add new message
this.shortTermContext.push(message);
// Limit to max turns
if (this.shortTermContext.length > this.maxShortTermTurns) {
this.shortTermContext = this.shortTermContext.slice(-this.maxShortTermTurns);
}
}
/**
* Updates the medium-term context with a new message
* @param message The message to add to medium-term context
*/
updateMediumTermContext(message: OllamaMessage): void {
// Add new message
this.mediumTermContext.push(message);
// Limit to max messages
if (this.mediumTermContext.length > this.maxMediumTermMessages) {
this.mediumTermContext = this.mediumTermContext.slice(-this.maxMediumTermMessages);
}
}
/**
* Sets the user's persona or core knowledge as long-term context
* @param personaContent The persona or core knowledge content
*/
setPersona(personaContent: string): void {
// Remove any existing persona messages
this.longTermContext = this.longTermContext.filter(
(msg) =>
msg.role !== 'system' ||
!msg.content.includes(
'You are an assistant that can help answer questions using the contents of a vault'
)
);
// Add the new persona
this.longTermContext.push({
role: 'system',
content: personaContent,
});
}
/**
* Gets the combined conversation context for the current turn
* @param userMessage The user's current message
* @returns Complete conversation context with all three layers
*/
getConversationContext(_userMessage: string): ConversationState {
return {
shortTermContext: this.shortTermContext,
mediumTermContext: this.mediumTermContext,
longTermContext: this.longTermContext,
};
}
/**
* Gets the complete messages array for sending to the LLM
* @param userMessage The user's current message
* @returns Complete message array for the LLM
*/
getCompleteMessages(userMessage: string): OllamaMessage[] {
const userMessageWithContext: OllamaMessage = {
role: 'user',
content: userMessage,
};
// Build messages in the proper order:
// 1. Long-term context (user persona, system instructions)
// 2. Medium-term context (session knowledge base query results)
// 3. Short-term context (last N turns)
// 4. Current user message
return [
...this.longTermContext,
...this.mediumTermContext,
...this.shortTermContext,
userMessageWithContext,
];
}
/**
* Clears all conversation context
*/
clear(): void {
this.shortTermContext = [];
this.mediumTermContext = [];
this.longTermContext = [
{
role: 'system',
content: `You are an assistant that can help answer questions using the contents of a vault.
The user can ask questions about their vault contents, and you should provide helpful responses based on the files.
When a user asks for information, try to find relevant files using the search_vault_files tool and read their contents with the read_vault_file tool.
Only use the tools if you need to access vault content that is not already in the context.`,
},
];
}
/**
* Sets the medium-term context from a knowledge base query result
* @param queryResult The result from a knowledge base query
*/
setMediumTermContextFromQuery(queryResult: string): void {
// Clear previous medium-term context
this.mediumTermContext = [];
// Add the query result as context
if (queryResult.trim()) {
this.mediumTermContext.push({
role: 'system',
content: `Knowledge base results for current query:\n${queryResult}`,
});
}
}
/**
* Gets the current short-term context
*/
getShortTermContext(): OllamaMessage[] {
return [...this.shortTermContext];
}
/**
* Gets the current medium-term context
*/
getMediumTermContext(): OllamaMessage[] {
return [...this.mediumTermContext];
}
/**
* Gets the current long-term context
*/
getLongTermContext(): OllamaMessage[] {
return [...this.longTermContext];
}
}
-107
View File
@@ -1,107 +0,0 @@
"use strict";
// src/error-handler.ts
Object.defineProperty(exports, "__esModule", { value: true });
exports.ErrorHandler = void 0;
const obsidian_1 = require("obsidian");
const types_1 = require("./types");
class ErrorHandler {
static handleError(error, context) {
const message = this.getUserFriendlyMessage(error);
new obsidian_1.Notice(message);
if (error instanceof Error) {
const ctx = context ? ` [${context}]` : '';
// Use console.error instead of ErrorHandler.error for fatal errors
console.error(`Ollama Plugin Error${ctx}: ${error.message}`);
if (error.stack) {
console.error(error.stack);
}
}
else {
const ctx = context ? ` [${context}]` : '';
console.error(`Ollama Plugin Error${ctx}:`, error);
}
}
static getUserFriendlyMessage(error) {
if (error instanceof types_1.OllamaError) {
return this.getUserFriendlyMessageFromOllamaError(error);
}
if (error instanceof Error) {
return this.getUserFriendlyMessageFromError(error);
}
return 'An unexpected error occurred';
}
static getUserFriendlyMessageFromOllamaError(error) {
switch (error.type) {
case types_1.ErrorType.NETWORK_ERROR:
return 'Connection error. Please check if Ollama is running.';
case types_1.ErrorType.API_ERROR:
return `API error: ${error.message}`;
case types_1.ErrorType.VALIDATION_ERROR:
return this.getUserFriendlyValidationMessage(error);
case types_1.ErrorType.STREAMING_ERROR:
return 'Response too long. Please try a shorter request.';
case types_1.ErrorType.TOOL_EXECUTION_ERROR:
return `Tool error for ${error.toolName}. ${error.message}`;
case types_1.ErrorType.PATH_VALIDATION_ERROR:
return `Invalid file path: ${error.path}`;
case types_1.ErrorType.UNKNOWN_ERROR:
return 'An unexpected error occurred';
default:
return 'An unexpected error occurred';
}
}
static getUserFriendlyValidationMessage(error) {
if (error instanceof types_1.ValidationError && error.details?.field) {
const fieldMsg = error.details.field.charAt(0).toUpperCase() + error.details.field.slice(1);
return `Invalid ${fieldMsg.toLowerCase()}. ${error.details.message ?? error.message}`;
}
return 'Input validation error. Please correct your input.';
}
static getUserFriendlyMessageFromError(error) {
const msg = error.message.toLowerCase();
// Check timeout BEFORE network (more specific matches first)
if (msg.includes('timeout') || msg.includes('timed out') || msg.includes('time out')) {
return 'Request timed out. Please check your Ollama connection.';
}
if (msg.includes('network') || msg.includes('connection') || msg.includes('fetch')) {
return 'Connection error. Please check if Ollama is running.';
}
if (msg.includes('validation') || msg.includes('invalid')) {
return 'Invalid input. Please correct your input.';
}
if (msg.includes('stream') || msg.includes('chunk')) {
return 'Response too long. Please try a shorter request.';
}
if (msg.includes('tool') || msg.includes('function')) {
return 'Tool error. Please try again.';
}
if (msg.includes('path') || msg.includes('file')) {
return 'Invalid file path. Please check the path and try again.';
}
return 'An unexpected error occurred';
}
// -- Factory methods --
static createNetworkError(message, statusCode) {
return new types_1.NetworkError(message, statusCode);
}
static createApiError(message, statusCode) {
return new types_1.ApiError(message, statusCode);
}
static createValidationError(message, field) {
const details = field ? { field, message } : undefined;
return new types_1.ValidationError(message, details);
}
static createStreamingError(message) {
return new types_1.StreamingError(message);
}
static createToolExecutionError(message, toolName) {
return new types_1.ToolExecutionError(message, toolName ?? 'unknown');
}
static createPathValidationError(message, path) {
return new types_1.PathValidationError(message, path ?? '');
}
static createUnknownError(message) {
return new types_1.OllamaError(message, types_1.ErrorType.UNKNOWN_ERROR);
}
}
exports.ErrorHandler = ErrorHandler;
+1 -1
View File
@@ -109,7 +109,7 @@ export class ErrorHandler {
}
static createApiError(message: string, statusCode?: number): ApiError {
return new ApiError(message, statusCode);
return new ApiError(message, statusCode ?? 500);
}
static createValidationError(message: string, field?: string): ValidationError {
+261
View File
@@ -0,0 +1,261 @@
// src/graph-view.ts
import { VaultIndexEntry } from './types';
/**
* Represents a node in the dependency graph
*/
export interface GraphNode {
id: string;
label: string;
file: VaultIndexEntry;
type: 'file' | 'concept';
properties: Record<string, unknown>;
}
/**
* Represents an edge in the dependency graph
*/
export interface GraphEdge {
id: string;
source: string;
target: string;
label: string;
relationship: string;
properties: Record<string, unknown>;
}
/**
* Represents a dependency graph structure
*/
export interface DependencyGraph {
nodes: GraphNode[];
edges: GraphEdge[];
}
/**
* Graph visualization formats
*/
export type GraphFormat = 'dot' | 'json' | 'cytoscape';
/**
* Extracts concepts from file content
*/
export function extractConcepts(content: string, _filePath: string): string[] {
// Extract concepts from headings, bold text, and mentions
const concepts: string[] = [];
// Extract all headings as potential concepts
const headingMatches = content.match(/^#{1,6} (.*?)$/gm);
if (headingMatches) {
headingMatches.forEach((heading) => {
const concept = heading.replace(/^#{1,6} /, '').trim();
if (concept && !concepts.includes(concept)) {
concepts.push(concept);
}
});
}
// Extract bold text as potential concepts
const boldMatches = content.match(/\*\*(.*?)\*\*/g);
if (boldMatches) {
boldMatches.forEach((match) => {
const concept = match.replace(/\*\*/g, '').trim();
if (concept && !concepts.includes(concept)) {
concepts.push(concept);
}
});
}
// Extract italic text as potential concepts
const italicMatches = content.match(/\*(.*?)\*/g);
if (italicMatches) {
italicMatches.forEach((match) => {
const concept = match.replace(/\*/g, '').trim();
if (concept && !concepts.includes(concept)) {
concepts.push(concept);
}
});
}
return concepts;
}
/**
* Finds relationships between files based on concept mentions
*/
export function findRelationships(
files: VaultIndexEntry[],
conceptIndex: Record<string, string[]>
): { source: string; target: string; relationship: string }[] {
const relationships: { source: string; target: string; relationship: string }[] = [];
// For each file, look for concepts that are defined in other files
files.forEach((file) => {
const fileContent = file.content;
const fileConcepts = extractConcepts(fileContent, file.path);
fileConcepts.forEach((concept) => {
// Check if this concept is defined in another file
if (conceptIndex[concept]) {
conceptIndex[concept].forEach((definedIn) => {
if (definedIn !== file.path) {
relationships.push({
source: file.path,
target: definedIn,
relationship: `mentions "${concept}" which is defined in`,
});
}
});
}
});
});
return relationships;
}
/**
* Builds a dependency graph from vault entries
*/
export function buildDependencyGraph(files: VaultIndexEntry[]): DependencyGraph {
// Create a concept index: concept -> files where it's defined
const conceptIndex: Record<string, string[]> = {};
files.forEach((file) => {
const concepts = extractConcepts(file.content, file.path);
concepts.forEach((concept) => {
if (!conceptIndex[concept]) {
conceptIndex[concept] = [];
}
if (!conceptIndex[concept].includes(file.path)) {
conceptIndex[concept].push(file.path);
}
});
});
// Build graph nodes
const nodes: GraphNode[] = [];
const edges: GraphEdge[] = [];
// Add file nodes
files.forEach((file) => {
nodes.push({
id: file.path,
label: file.title || file.path,
file: file,
type: 'file' as const,
properties: {
path: file.path,
title: file.title,
contentPreview: file.content.substring(0, 100) + '.',
},
});
});
// Find relationships and add edges
const fileRelationships = findRelationships(files, conceptIndex);
fileRelationships.forEach((rel) => {
// Only add edge if both source and target files exist
if (files.some((f) => f.path === rel.source) && files.some((f) => f.path === rel.target)) {
edges.push({
id: `${rel.source}--${rel.target}`,
source: rel.source,
target: rel.target,
label: rel.relationship,
relationship: rel.relationship,
properties: {
relationship: rel.relationship,
},
});
}
});
return {
nodes,
edges,
};
}
/**
* Converts dependency graph to DOT format
*/
export function toDotFormat(graph: DependencyGraph): string {
let dot = 'digraph G {\n';
dot += ' rankdir=LR;\n';
dot += ' node [shape=box, style=filled, fillcolor=lightblue];\n';
dot += ' edge [arrowhead=vee];\n\n';
// Add nodes
graph.nodes.forEach((node) => {
const label = node.label.replace(/"/g, '\\"');
dot += ` "${node.id}" [label="${label}"];\n`;
});
// Add edges
graph.edges.forEach((edge) => {
const label = edge.label.replace(/"/g, '\\"');
dot += ` "${edge.source}" -> "${edge.target}" [label="${label}"];\n`;
});
dot += '}\n';
return dot;
}
/**
* Converts dependency graph to JSON format
*/
export function toJsonFormat(graph: DependencyGraph): string {
return JSON.stringify(graph, null, 2);
}
/**
* Converts dependency graph to Cytoscape format
*/
export function toCytoscapeFormat(graph: DependencyGraph): string {
const cytoscapeFormat = {
elements: {
nodes: graph.nodes.map((node) => ({
data: {
id: node.id,
label: node.label,
type: node.type,
...node.properties,
},
})),
edges: graph.edges.map((edge) => ({
data: {
id: edge.id,
source: edge.source,
target: edge.target,
label: edge.label,
relationship: edge.relationship,
...edge.properties,
},
})),
},
};
return JSON.stringify(cytoscapeFormat, null, 2);
}
/**
* Generates graph visualization in specified format
*/
export function generateGraphVisualization(
files: VaultIndexEntry[],
format: GraphFormat = 'dot'
): string {
const graph = buildDependencyGraph(files);
switch (format) {
case 'dot':
return toDotFormat(graph);
case 'json':
return toJsonFormat(graph);
case 'cytoscape':
return toCytoscapeFormat(graph);
default:
return toDotFormat(graph);
}
}
+110
View File
@@ -0,0 +1,110 @@
// src/indexing-pipeline/extraction.ts
export interface VaultFile {
basename: string;
path: string;
}
export interface Frontmatter {
title?: string;
tags?: string;
[key: string]: unknown;
}
export interface ExtractedContent {
basename: string;
path: string;
content: string;
frontmatter: Frontmatter;
headings: string[];
embeddedCodeBlocks: string[];
firstParagraph?: string;
}
/**
* Extracts raw content from a vault file including:
* - Markdown content
* - YAML frontmatter
* - Headings
* - Embedded code blocks
* - First paragraph
*/
export class ContentExtractor {
extractFromFile(file: VaultFile, content: string): ExtractedContent {
const frontmatter: Frontmatter = {};
const headings: string[] = [];
const embeddedCodeBlocks: string[] = [];
let firstParagraph: string | undefined;
// Extract frontmatter
const frontmatterMatch = content.match(/^---(.*?)---/s);
if (frontmatterMatch) {
try {
const frontmatterContent = frontmatterMatch[1];
const lines = frontmatterContent.trim().split('\n');
for (const line of lines) {
const [key, ...valueParts] = line.split(':');
if (!key) continue;
const value = valueParts.join(':').trim();
if (key.trim() === 'title') {
if (value) {
frontmatter.title = value;
}
} else if (key.trim() === 'tags') {
if (value) {
frontmatter.tags = value;
}
} else {
// Store other frontmatter fields as-is
frontmatter[key.trim()] = value;
}
}
} catch {
// If frontmatter parsing fails, continue with empty frontmatter
}
}
// Extract headings
const headingMatches = content.match(/^#{1,6} (.*?)$/gm);
if (headingMatches) {
headings.push(...headingMatches.map((h: string) => h.replace(/^#{1,6} /, '')));
}
// Extract embedded code blocks
const codeBlockMatches = content.match(/```([\s\S]*?)```/g);
if (codeBlockMatches) {
embeddedCodeBlocks.push(...codeBlockMatches);
}
// Extract first paragraph
const paragraphMatch = content.match(/^([^#]*?)(?=#|\n\n|$)/s);
if (paragraphMatch) {
firstParagraph = paragraphMatch[1].trim();
}
return {
basename: file.basename,
path: file.path,
content,
frontmatter,
headings,
embeddedCodeBlocks,
firstParagraph,
};
}
/**
* Extracts just the raw text content without headers, frontmatter, etc.
*/
extractRawText(content: string): string {
return content
.replace(/^---.*?---/s, '')
.replace(/^#.*?$/gm, '')
.replace(/```.*?```/gs, '')
.replace(/`.*?`/g, '')
.replace(/\[(.*?)\]\(.*?\)/g, '$1')
.replace(/\*\*(.*?)\*\*/g, '$1')
.replace(/\*(.*?)\*/g, '$1')
.trim();
}
}
+6
View File
@@ -0,0 +1,6 @@
// src/indexing-pipeline/index.ts
export { ContentExtractor } from './extraction';
export { ContentNormalizer } from './normalization';
export { ContentVectorizer } from './vectorization';
export { IndexingPipeline } from './pipeline';
+200
View File
@@ -0,0 +1,200 @@
// src/indexing-pipeline/normalization.ts
// Import ExtractedContent interface from extraction module
import { ExtractedContent } from './extraction';
interface NormalizedContent {
path: string;
title: string;
content: string;
tokens: string[];
headings: string[];
frontmatter: Record<string, unknown>;
firstParagraph?: string;
// Additional enrichment fields
wordCount: number;
createdAt?: string;
updatedAt?: string;
}
/**
* Normalizes and enriches extracted content
*/
export class ContentNormalizer {
/**
* Normalizes content by:
* - Standardizing dates to ISO 8601
* - Converting to lowercase for tokenization
* - Extracting tokens
* - Adding metadata
*/
normalize(extractedContent: ExtractedContent): NormalizedContent {
const { basename, path, content, frontmatter, headings, firstParagraph } = extractedContent;
// Standardize title (remove .md extension)
const title = basename.replace(/\.md$/, '');
// Extract tokens (lowercase, remove stop words, etc.)
const tokens = this.tokenize(content);
// Normalize dates (if present in frontmatter)
const normalizedFrontmatter = this.normalizeFrontmatter(frontmatter);
// Calculate word count
const wordCount = content.split(/\s+/).filter(Boolean).length;
return {
path,
title,
content,
tokens,
headings,
frontmatter: normalizedFrontmatter,
firstParagraph,
wordCount,
// Add timestamps if available in frontmatter
createdAt: this.extractDate(frontmatter, 'created') || this.extractDate(frontmatter, 'date'),
updatedAt: this.extractDate(frontmatter, 'updated'),
};
}
/**
* Tokenizes text content by splitting on whitespace and removing stop words
*/
private tokenize(text: string): string[] {
const stopWords = new Set([
'the',
'a',
'an',
'and',
'or',
'but',
'is',
'are',
'was',
'were',
'in',
'on',
'at',
'to',
'of',
'for',
'with',
'as',
'by',
'it',
'its',
'that',
'this',
'these',
'those',
'from',
'up',
'out',
'off',
'over',
'under',
'again',
'further',
'then',
'once',
'here',
'there',
'when',
'where',
'why',
'how',
'all',
'any',
'both',
'each',
'few',
'more',
'most',
'other',
'some',
'such',
'no',
'nor',
'not',
'only',
'own',
'same',
'so',
'than',
'too',
'very',
'just',
'now',
]);
return text
.toLowerCase()
.split(/\W+/)
.filter((token) => token.length > 1 && !stopWords.has(token));
}
/**
* Normalizes frontmatter by standardizing data types and formats
*/
private normalizeFrontmatter(frontmatter: Record<string, unknown>): Record<string, unknown> {
const normalized: Record<string, unknown> = {};
for (const [key, value] of Object.entries(frontmatter)) {
if (key === 'tags' && typeof value === 'string') {
// Convert tag string to array if needed
normalized.tags = value.split(',').map((tag) => tag.trim());
} else if (key === 'date' || key === 'created' || key === 'updated') {
// Try to parse and standardize date formats
if (typeof value === 'string') {
const date = new Date(value);
if (!isNaN(date.getTime())) {
normalized[key] = date.toISOString();
} else {
normalized[key] = value; // Keep original if invalid date
}
} else {
normalized[key] = value;
}
} else {
normalized[key] = value;
}
}
return normalized;
}
/**
* Extracts a date from frontmatter
*/
private extractDate(frontmatter: Record<string, unknown>, key: string): string | undefined {
const value = frontmatter[key];
if (typeof value === 'string') {
const date = new Date(value);
if (!isNaN(date.getTime())) {
return date.toISOString();
}
}
return undefined;
}
}
/**
* Interface for a normalized content chunk
*/
export interface ContentChunk {
id: string;
path: string;
title: string;
content: string;
tokens: string[];
headings: string[];
frontmatter: Record<string, unknown>;
firstParagraph?: string;
wordCount: number;
createdAt?: string;
updatedAt?: string;
// Additional enrichment fields for vectorization
chunkIndex: number;
chunkSize: number;
}
+89
View File
@@ -0,0 +1,89 @@
// src/indexing-pipeline/pipeline.ts
import { VaultIndexEntry } from '../types';
import { ContentExtractor, VaultFile } from './extraction';
import { ContentNormalizer } from './normalization';
import { ContentVectorizer } from './vectorization';
interface PipelineConfig {
ollamaUrl: string;
embeddingModel: string;
}
export class IndexingPipeline {
private extractor: ContentExtractor;
private normalizer: ContentNormalizer;
private vectorizer: ContentVectorizer;
constructor(config: PipelineConfig) {
this.extractor = new ContentExtractor();
this.normalizer = new ContentNormalizer();
this.vectorizer = new ContentVectorizer({
model: config.embeddingModel,
ollamaUrl: config.ollamaUrl,
});
}
/**
* Processes a vault file through the entire pipeline
*/
processFile(file: VaultFile, content: string): VaultIndexEntry | null {
try {
if (!content.trim()) {
return null;
}
// Extraction step
const extracted = this.extractor.extractFromFile(file, content);
// Normalization/Enrichment step
const normalized = this.normalizer.normalize(extracted);
// Return the normalized content as an index entry
return {
path: normalized.path,
title: normalized.title,
content: this.extractor.extractRawText(content).substring(0, 500),
score: 0, // Score will be calculated during search
};
} catch {
return null;
}
}
/**
* Processes multiple files in batches
*/
processFilesInBatches(
files: VaultFile[],
fileContents: Record<string, string>,
batchSize: number = 10
): VaultIndexEntry[] {
const results: VaultIndexEntry[] = [];
const seenPaths = new Set<string>();
for (let i = 0; i < files.length; i += batchSize) {
const batch = files.slice(i, i + batchSize);
const batchResults = batch.map((file) => {
const content = fileContents[file.path];
if (!content) {
return null;
}
const entry = this.processFile(file, content);
if (entry && !seenPaths.has(entry.path)) {
seenPaths.add(entry.path);
return entry;
}
return null;
});
const validResults = batchResults.filter(
(result): result is NonNullable<typeof result> => result !== null
);
results.push(...validResults);
}
return results;
}
}
+82
View File
@@ -0,0 +1,82 @@
// src/indexing-pipeline/vectorization.ts
import { ContentChunk } from './normalization';
import { Logger } from '../utils';
interface VectorizationConfig {
model: string;
ollamaUrl: string;
}
/**
* Vectorizes content chunks using Ollama embeddings
*/
export class ContentVectorizer {
private model: string;
private ollamaUrl: string;
private fetchFn: typeof fetch;
constructor(config: VectorizationConfig, fetchFn?: typeof fetch) {
this.model = config.model;
this.ollamaUrl = config.ollamaUrl;
this.fetchFn = fetchFn ?? fetch;
}
/**
* Generates embeddings for a content chunk
*/
async vectorize(chunk: ContentChunk): Promise<number[]> {
try {
const prompt = this.createPrompt(chunk);
const response = await this.fetchFn(`${this.ollamaUrl}/api/embeddings`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
model: this.model,
prompt: prompt,
}),
});
if (!response.ok) {
throw new Error(`Embedding failed with status ${response.status}`);
}
const data: unknown = await response.json();
if (!this.isEmbeddingResponse(data)) {
throw new Error('Invalid embedding response');
}
return data.embedding;
} catch (error) {
// Return empty array on failure to maintain compatibility
Logger.warn(`Failed to generate embedding: ${String(error)}`, 'indexing-pipeline');
return [];
}
}
private isEmbeddingResponse(data: unknown): data is { embedding: number[] } {
return (
typeof data === 'object' &&
data !== null &&
Array.isArray((data as { embedding?: unknown }).embedding) &&
(data as { embedding: unknown[] }).embedding.every((value) => typeof value === 'number')
);
}
/**
* Creates a prompt from content chunk for embedding
*/
private createPrompt(chunk: ContentChunk): string {
// Combine important elements for embedding
const parts = [
chunk.title,
chunk.firstParagraph,
chunk.content.substring(0, 1000), // Limit content to avoid long prompts
chunk.headings.join(' '),
JSON.stringify(chunk.frontmatter),
].filter(Boolean);
return parts.join('\n\n');
}
}
-133
View File
@@ -1,133 +0,0 @@
"use strict";
Object.defineProperty(exports, "__esModule", { value: true });
const obsidian_1 = require("obsidian");
const chat_view_1 = require("./chat-view");
const utils_1 = require("./utils");
const constants_1 = require("./constants");
const error_handler_1 = require("./error-handler");
class OllamaPlugin extends obsidian_1.Plugin {
constructor() {
super(...arguments);
this.settings = constants_1.DEFAULT_SETTINGS;
}
async onload() {
// Initialize logging
utils_1.Logger.info('Ollama Plugin loading...', 'plugin');
await this.loadSettings();
utils_1.Logger.info('Plugin loaded successfully', 'plugin');
try {
this.registerView('ollama-chat-view', (leaf) => new chat_view_1.ChatView(leaf, this.settings));
}
catch (error) {
utils_1.Logger.error('Failed to register view: ' + error.message, 'plugin');
new obsidian_1.Notice('Failed to register Ollama chat view');
// Don't throw - let the plugin continue loading other features
}
try {
this.addRibbonIcon('message-square', 'Ollama Chat', async () => {
const leaf = this.app.workspace.getLeaf();
await leaf.setViewState({
type: 'ollama-chat-view',
active: true,
});
await this.app.workspace.revealLeaf(leaf);
});
}
catch (error) {
utils_1.Logger.error('Failed to add ribbon icon: ' + error.message, 'plugin');
new obsidian_1.Notice('Failed to add Ollama ribbon icon');
// Don't throw - let the plugin continue loading other features
}
this.addSettingTab(new OllamaSettingTab(this.app, this));
}
async loadSettings() {
try {
const data = (await this.loadData());
if (data) {
utils_1.Logger.debug('Loading saved settings', 'settings');
this.settings = Object.assign({}, this.settings, data);
}
}
catch (error) {
error_handler_1.ErrorHandler.handleError(error, 'settings load');
}
}
async saveSettings() {
try {
// Validate settings before saving
const validationErrors = (0, utils_1.validatePluginSettings)(this.settings);
if (validationErrors.length > 0) {
utils_1.Logger.error('Validation errors prevented saving settings: ' + validationErrors.join('; '), 'settings');
new obsidian_1.Notice(`Cannot save settings: ${validationErrors[0]}`);
return false;
}
utils_1.Logger.debug('Saving settings: ' + JSON.stringify(this.settings), 'settings');
await this.saveData(this.settings);
utils_1.Logger.info('Settings saved successfully', 'settings');
return true;
}
catch (error) {
error_handler_1.ErrorHandler.handleError(error, 'settings save');
return false;
}
}
// Notify all open ChatView instances when settings change
notifyChatViews() {
const leaves = this.app.workspace.getLeavesOfType('ollama-chat-view');
leaves.forEach((leaf) => {
const view = leaf.view;
if (view instanceof chat_view_1.ChatView) {
view.onSettingsChange(this.settings);
}
});
}
}
exports.default = OllamaPlugin;
class OllamaSettingTab extends obsidian_1.PluginSettingTab {
constructor(app, plugin) {
super(app, plugin);
this.plugin = plugin;
}
display() {
// Clear any existing content first to prevent duplicates
this.containerEl.empty();
// Create container for settings
const container = this.containerEl.createDiv();
new obsidian_1.Setting(container)
.setName('Ollama URL')
.setDesc('URL of your Ollama instance')
.addText((text) => text.setValue(this.plugin.settings.ollamaUrl).onChange(async (value) => {
const urlValidation = (0, utils_1.validateOllamaUrl)(value);
if (urlValidation.valid) {
utils_1.Logger.debug('URL changed to: ' + value, 'settings');
this.plugin.settings.ollamaUrl = value;
await this.plugin.saveSettings();
this.plugin.notifyChatViews();
}
else {
utils_1.Logger.warn('Invalid URL format: ' + urlValidation.error, 'settings');
new obsidian_1.Notice(urlValidation.error || 'Invalid Ollama URL format.');
}
}));
new obsidian_1.Setting(container)
.setName('Model')
.setDesc('Model to use for chat')
.addText((text) => text.setValue(this.plugin.settings.model).onChange(async (value) => {
const modelValidation = (0, utils_1.validateModelName)(value);
if (modelValidation.valid) {
utils_1.Logger.debug('Model changed to: ' + value, 'settings');
this.plugin.settings.model = value;
await this.plugin.saveSettings();
this.plugin.notifyChatViews();
}
else {
utils_1.Logger.warn('Invalid model name format: ' + modelValidation.error, 'settings');
new obsidian_1.Notice(modelValidation.error || 'Invalid model name format.');
}
}));
}
hide() {
// Clear the container to prevent duplicate elements
this.containerEl.empty();
}
}
+173 -93
View File
@@ -1,150 +1,174 @@
import { Plugin, WorkspaceLeaf, App, Setting, Notice, PluginSettingTab } from 'obsidian';
import { ChatView } from './chat-view';
import { PluginSettings } from './types';
import { validatePluginSettings, validateOllamaUrl, validateModelName, Logger } from './utils';
import { DEFAULT_SETTINGS } from './constants';
import { ErrorHandler } from './error-handler';
import { SemanticCacheService } from './semantic-cache';
import { PluginSettings } from './types';
export default class OllamaPlugin extends Plugin {
settings: PluginSettings = DEFAULT_SETTINGS;
semanticCache?: SemanticCacheService;
async onload() {
// Initialize logging
Logger.info('Ollama Plugin loading...', 'plugin');
await this.loadSettings();
Logger.info('Plugin loaded successfully', 'plugin');
try {
this.registerView(
'ollama-chat-view',
(leaf: WorkspaceLeaf) => new ChatView(leaf, this.settings)
// Register the chat view
this.registerView(
'ollama-chat-view',
(leaf: WorkspaceLeaf) => new ChatView(leaf, this.settings)
);
// Add a command to open the chat view
this.addCommand({
id: 'open-ollama-chat',
name: 'Open Ollama Chat',
callback: async () => {
await this.activateChatView();
},
});
// Add a command to clear the semantic cache
this.addCommand({
id: 'clear-semantic-cache',
name: 'Clear Semantic Cache',
callback: async () => {
await this.clearSemanticCache();
new Notice('Semantic cache cleared.');
},
});
// Add a settings tab
this.addSettingTab(new OllamaSettingTab(this.app, this));
// Initialize the semantic cache
if (this.settings.cacheConfig) {
this.semanticCache = new SemanticCacheService(
this.settings.ollamaUrl,
this.settings.cacheConfig
);
} catch (error) {
Logger.error('Failed to register view: ' + (error as Error).message, 'plugin');
new Notice('Failed to register Ollama chat view');
// Don't throw - let the plugin continue loading other features
try {
await this.semanticCache.initialize();
} catch {
new Notice('Semantic cache initialization failed. Check console for details.');
}
}
}
try {
this.addRibbonIcon('message-square', 'Ollama Chat', async () => {
const leaf = this.app.workspace.getLeaf();
// eslint-disable-next-line @typescript-eslint/no-misused-promises
onunload() {
// Clean up any active semantic cache resources on plugin unload
// Using fire-and-forget pattern since onunload cannot be async per Obsidian API
if (this.semanticCache) {
void this.semanticCache.clearCache();
}
// No explicit unregisterView needed; relying on Obsidian lifecycle management.
}
async loadSettings() {
const loadedSettings = ((await this.loadData()) ?? {}) as Partial<PluginSettings>;
this.settings = Object.assign({}, DEFAULT_SETTINGS, loadedSettings);
}
async saveSettings() {
await this.saveData(this.settings);
}
async activateChatView() {
const existing = this.app.workspace.getLeavesOfType('ollama-chat-view');
if (existing.length > 0) {
await this.app.workspace.revealLeaf(existing[0]);
} else {
const leaf = this.app.workspace.getRightLeaf(false);
if (leaf) {
await leaf.setViewState({
type: 'ollama-chat-view',
active: true,
});
await this.app.workspace.revealLeaf(leaf);
});
} catch (error) {
Logger.error('Failed to add ribbon icon: ' + (error as Error).message, 'plugin');
new Notice('Failed to add Ollama ribbon icon');
// Don't throw - let the plugin continue loading other features
}
this.addSettingTab(new OllamaSettingTab(this.app, this));
}
async loadSettings() {
try {
const data = (await this.loadData()) as Partial<PluginSettings> | null;
if (data) {
Logger.debug('Loading saved settings', 'settings');
this.settings = Object.assign({}, this.settings, data);
}
} catch (error) {
ErrorHandler.handleError(error, 'settings load');
}
}
async saveSettings() {
try {
// Validate settings before saving
const validationErrors = validatePluginSettings(this.settings);
if (validationErrors.length > 0) {
Logger.error(
'Validation errors prevented saving settings: ' + validationErrors.join('; '),
'settings'
);
new Notice(`Cannot save settings: ${validationErrors[0]}`);
return false;
}
Logger.debug('Saving settings: ' + JSON.stringify(this.settings), 'settings');
await this.saveData(this.settings);
Logger.info('Settings saved successfully', 'settings');
return true;
} catch (error) {
ErrorHandler.handleError(error, 'settings save');
return false;
async clearSemanticCache() {
if (this.semanticCache) {
await this.semanticCache.clearCache();
}
}
// Notify all open ChatView instances when settings change
public notifyChatViews(): void {
notifyChatViews() {
const leaves = this.app.workspace.getLeavesOfType('ollama-chat-view');
leaves.forEach((leaf) => {
const view = leaf.view;
if (view instanceof ChatView) {
view.onSettingsChange(this.settings);
if (leaf.view instanceof ChatView) {
leaf.view.updateSettings(this.settings);
}
});
}
}
class OllamaSettingTab extends PluginSettingTab {
private plugin: OllamaPlugin;
plugin: OllamaPlugin;
constructor(app: App, plugin: OllamaPlugin) {
super(app, plugin);
this.plugin = plugin;
}
display(): void {
// Clear any existing content first to prevent duplicates
this.containerEl.empty();
display() {
const { containerEl } = this;
containerEl.empty();
containerEl.createEl('h2', { text: 'Ollama Settings' });
// Create container for settings
const container = this.containerEl.createDiv() as HTMLElement;
new Setting(container)
new Setting(containerEl)
.setName('Ollama URL')
.setDesc('URL of your Ollama instance')
.setDesc('URL for your Ollama instance (default: http://localhost:11434)')
.addText((text) =>
text.setValue(this.plugin.settings.ollamaUrl).onChange(async (value) => {
const urlValidation = validateOllamaUrl(value);
if (urlValidation.valid) {
Logger.debug('URL changed to: ' + value, 'settings');
this.plugin.settings.ollamaUrl = value;
await this.plugin.saveSettings();
this.plugin.notifyChatViews();
} else {
Logger.warn('Invalid URL format: ' + urlValidation.error, 'settings');
new Notice(urlValidation.error || 'Invalid Ollama URL format.');
}
this.plugin.settings.ollamaUrl = value;
await this.plugin.saveSettings();
})
);
new Setting(container)
new Setting(containerEl)
.setName('Model')
.setDesc('Model to use for chat')
.setDesc('Ollama model to use (default: llama3)')
.addText((text) =>
text.setValue(this.plugin.settings.model).onChange(async (value) => {
const modelValidation = validateModelName(value);
if (modelValidation.valid) {
Logger.debug('Model changed to: ' + value, 'settings');
this.plugin.settings.model = value;
this.plugin.settings.model = value;
await this.plugin.saveSettings();
})
);
new Setting(containerEl)
.setName('Vault Search Limit')
.setDesc('Maximum number of vault entries to include in context (default: 3)')
.addText((text) =>
text.setValue(String(this.plugin.settings.vaultSearchLimit)).onChange(async (value) => {
const parsed = parseInt(value);
if (!isNaN(parsed) && parsed > 0) {
this.plugin.settings.vaultSearchLimit = parsed;
await this.plugin.saveSettings();
this.plugin.notifyChatViews();
} else {
Logger.warn('Invalid model name format: ' + modelValidation.error, 'settings');
new Notice(modelValidation.error || 'Invalid model name format.');
new Notice('Vault search limit must be a positive integer.');
}
})
);
new Setting(container)
new Setting(containerEl)
.setName('Max Message History')
.setDesc('Maximum number of messages to keep in conversation history (default: 50)')
.addText((text) =>
text.setValue(String(this.plugin.settings.maxMessageHistory)).onChange(async (value) => {
const parsed = parseInt(value);
if (!isNaN(parsed) && parsed > 0) {
this.plugin.settings.maxMessageHistory = parsed;
await this.plugin.saveSettings();
} else {
new Notice('Max message history must be a positive integer.');
}
})
);
new Setting(containerEl)
.setName('Enable Semantic Cache')
.setDesc('Cache responses semantically to speed up repeated queries')
.setDesc('Use semantic cache to store and retrieve previous responses')
.addToggle((toggle) =>
toggle.setValue(this.plugin.settings.cacheConfig.enabled).onChange(async (value) => {
this.plugin.settings.cacheConfig.enabled = value;
@@ -152,9 +176,65 @@ class OllamaSettingTab extends PluginSettingTab {
this.plugin.notifyChatViews();
})
);
new Setting(containerEl)
.setName('ChromaDB URL')
.setDesc('URL for your ChromaDB instance (default: http://localhost:8000)')
.addText((text) =>
text
.setValue(this.plugin.settings.cacheConfig.chromaURL || 'http://localhost:8000')
.onChange(async (value) => {
this.plugin.settings.cacheConfig.chromaURL = value;
await this.plugin.saveSettings();
})
);
new Setting(containerEl)
.setName('Cache Embedding Model')
.setDesc('Ollama model used to generate embeddings for the semantic cache')
.addText((text) =>
text.setValue(this.plugin.settings.cacheConfig.embeddingModel).onChange(async (value) => {
this.plugin.settings.cacheConfig.embeddingModel = value;
await this.plugin.saveSettings();
this.plugin.notifyChatViews();
})
);
new Setting(containerEl)
.setName('Cache Similarity Threshold')
.setDesc(
'Minimum cosine similarity (01) for a cache hit. Higher values require closer matches.'
)
.addText((text) =>
text
.setValue(String(this.plugin.settings.cacheConfig.similarityThreshold))
.onChange(async (value) => {
const parsed = parseFloat(value);
if (!isNaN(parsed) && parsed >= 0 && parsed <= 1) {
this.plugin.settings.cacheConfig.similarityThreshold = parsed;
await this.plugin.saveSettings();
} else {
new Notice('Similarity threshold must be a number between 0 and 1.');
}
})
);
new Setting(containerEl)
.setName('Clear Semantic Cache')
.setDesc('Delete all cached responses from ChromaDB')
.addButton((button) =>
button.setButtonText('Clear Cache').onClick(async () => {
try {
await this.plugin.clearSemanticCache();
new Notice('Semantic cache cleared.');
} catch {
new Notice('Failed to clear semantic cache. Is ChromaDB running?');
}
})
);
}
hide(): void {
hide() {
// Clear the container to prevent duplicate elements
this.containerEl.empty();
}
-238
View File
@@ -1,238 +0,0 @@
"use strict";
// src/ollama-client.ts
Object.defineProperty(exports, "__esModule", { value: true });
exports.OllamaClient = void 0;
const types_1 = require("./types");
const utils_1 = require("./utils");
class OllamaClient {
constructor(baseURL, model, fetchFn) {
this.maxRetries = 3;
this.currentStreamController = null;
this.baseURL = baseURL;
this.model = model;
this.fetchFn = fetchFn ?? fetch;
}
cancelStream() {
if (this.currentStreamController) {
this.currentStreamController.abort();
this.currentStreamController = null;
}
}
async *streamChat(messages, tools = []) {
yield* this.streamChatWithRetry(messages, tools, 0);
}
async streamChatAsPromise(messages, tools = []) {
const chunks = [];
for await (const chunk of this.streamChat(messages, tools)) {
chunks.push(chunk);
}
return chunks;
}
async *streamChatWithRetry(messages, tools = [], attempt = 0) {
// Create a local controller for this request instead of using the instance variable
const controller = new AbortController();
this.currentStreamController = controller;
try {
const response = await this.fetchFn(`${this.baseURL}/api/chat`, {
method: 'POST',
headers: {
'Content-Type': 'application/json',
},
body: JSON.stringify({
model: this.model,
messages,
tools,
stream: true,
}),
signal: controller.signal,
});
if (!response.ok) {
if (response.status >= 500 && attempt < this.maxRetries) {
const retryDelay = Math.pow(2, attempt) * 100;
utils_1.Logger.warn(`Network error (status ${response.status}), retrying in ${retryDelay}ms (attempt ${attempt + 1}/${this.maxRetries})`, 'ollama-client');
const retryTimeout = new Promise((resolve) => setTimeout(resolve, retryDelay));
const abortListener = () => {
utils_1.Logger.info('Retry aborted by user', 'ollama-client');
};
const signal = controller.signal;
if (signal) {
signal.addEventListener('abort', abortListener);
try {
await Promise.race([
retryTimeout,
new Promise((resolve) => {
signal.addEventListener('abort', () => resolve(), {
once: true,
});
}),
]);
}
finally {
signal.removeEventListener('abort', abortListener);
}
}
else {
await retryTimeout;
}
yield* this.streamChatWithRetry(messages, tools, attempt + 1);
return;
}
throw new types_1.ApiError(`Ollama API error: ${response.status}`, response.status);
}
if (!response.body) {
throw new Error('No response body');
}
const contentType = response.headers.get('content-type');
if (!contentType || (!contentType.includes('ndjson') && !contentType.includes('json'))) {
throw new Error('Invalid response format');
}
const reader = response.body.getReader();
const decoder = new TextDecoder();
let buffer = '';
let malformedCount = 0;
const maxMalformed = 50;
try {
while (true) {
const { done, value } = await reader.read();
if (done)
break;
buffer += decoder.decode(value, { stream: true });
const lines = buffer.split('\n');
buffer = lines.pop() ?? '';
for (const line of lines) {
if (!line.trim())
continue;
try {
const parsed = JSON.parse(line);
this.throwIfOllamaError(parsed);
const message = this.toOllamaMessage(parsed.message);
if (!message) {
continue;
}
malformedCount = 0;
yield message;
}
catch (error) {
if (error instanceof Error && error.message.startsWith('Ollama error:')) {
throw error;
}
malformedCount++;
if (malformedCount > maxMalformed) {
throw new Error('Too many malformed chunks in stream');
}
utils_1.Logger.warn(`Skipped malformed chunk: ${line.substring(0, 80)}... - ${error.message}`, 'ollama-client');
}
}
}
if (buffer.trim()) {
try {
const parsed = JSON.parse(buffer);
this.throwIfOllamaError(parsed);
const message = this.toOllamaMessage(parsed.message);
if (message) {
yield message;
}
}
catch (error) {
if (error instanceof Error && error.message.startsWith('Ollama error:')) {
throw error;
}
utils_1.Logger.warn(`Failed to parse final chunk: ${buffer.substring(0, 80)}...`, 'ollama-client');
}
}
}
finally {
reader.releaseLock();
}
}
finally {
// Abort the local controller to release underlying fetch resources if not already aborted
if (!controller.signal.aborted) {
controller.abort();
}
// Clean up the reference only if this is still the current stream
if (this.currentStreamController === controller) {
this.currentStreamController = null;
}
}
}
async chat(messages, tools = []) {
return this.chatWithRetry(messages, tools, 0);
}
async chatWithRetry(messages, tools = [], attempt = 0) {
// Create a local controller for this request instead of using the instance variable
const controller = new AbortController();
try {
const response = await this.fetchFn(`${this.baseURL}/api/chat`, {
method: 'POST',
headers: {
'Content-Type': 'application/json',
},
body: JSON.stringify({
model: this.model,
messages,
tools,
stream: false,
}),
signal: controller.signal,
});
if (!response.ok) {
if (response.status >= 500 && attempt < this.maxRetries) {
const retryDelay = Math.pow(2, attempt) * 100;
utils_1.Logger.warn(`Network error (status ${response.status}), retrying in ${retryDelay}ms (attempt ${attempt + 1}/${this.maxRetries})`, 'ollama-client');
const retryTimeout = new Promise((resolve) => setTimeout(resolve, retryDelay));
const abortListener = () => {
utils_1.Logger.info('Retry aborted by user', 'ollama-client');
};
const signal = controller.signal;
if (signal) {
signal.addEventListener('abort', abortListener);
try {
await Promise.race([
retryTimeout,
new Promise((resolve) => {
signal.addEventListener('abort', () => resolve(), {
once: true,
});
}),
]);
}
finally {
signal.removeEventListener('abort', abortListener);
}
}
else {
await retryTimeout;
}
return this.chatWithRetry(messages, tools, attempt + 1);
}
throw new types_1.ApiError(`Ollama API error: ${response.status}`, response.status);
}
const data = (await response.json());
return (this.toOllamaMessage(data.message) ?? { role: 'assistant', content: '', tool_calls: [] });
}
finally {
// Abort the local controller to release underlying fetch resources if not already aborted
if (!controller.signal.aborted) {
controller.abort();
}
}
}
throwIfOllamaError(parsed) {
if (parsed.error) {
throw new Error(`Ollama error: ${String(parsed.error)}`);
}
}
toOllamaMessage(value) {
if (!value || typeof value !== 'object') {
return null;
}
const record = value;
return {
role: record.role ?? 'assistant',
content: typeof record.content === 'string' ? record.content : '',
tool_calls: record.tool_calls ?? [],
};
}
}
exports.OllamaClient = OllamaClient;
+261 -242
View File
@@ -1,12 +1,13 @@
// src/ollama-client.ts
import type { OllamaMessage, OllamaTool } from './types';
import type { CacheConfig, OllamaMessage, OllamaTool } from './types';
import { ApiError } from './types';
import { Logger } from './utils';
import { SemanticCacheService, CacheConfig } from './semantic-cache';
import { SemanticCacheService } from './semantic-cache';
interface OllamaChatResponse {
message?: Partial<OllamaMessage>;
error?: string;
}
export class OllamaClient {
@@ -14,6 +15,7 @@ export class OllamaClient {
private model: string;
private fetchFn: typeof fetch;
private readonly maxRetries: number = 3;
private readonly maxMalformedChunks: number = 50;
private currentStreamController: AbortController | null = null;
private cacheService?: SemanticCacheService;
@@ -24,6 +26,7 @@ export class OllamaClient {
if (cacheConfig?.enabled) {
this.cacheService = new SemanticCacheService(baseURL, cacheConfig);
void this.cacheService.initialize();
}
}
@@ -33,6 +36,12 @@ export class OllamaClient {
}
}
async clearCache(): Promise<void> {
if (this.cacheService) {
await this.cacheService.clearCache();
}
}
cancelStream(): void {
if (this.currentStreamController) {
this.currentStreamController.abort();
@@ -50,6 +59,7 @@ export class OllamaClient {
return;
}
// Find the last user message
const lastUserMsg = [...messages].reverse().find((m) => m.role === 'user');
if (lastUserMsg && this.cacheService) {
const cached = await this.cacheService.getCache(lastUserMsg.content);
@@ -65,53 +75,176 @@ export class OllamaClient {
yield chunk;
}
// Populate cache in background after successful stream
const fullContent = chunks.map((c) => c.content).join('');
if (this.cacheService && lastUserMsg) {
const fullContent = chunks.map((c) => c.content).join('');
void this.cacheService.setCache(lastUserMsg.content, fullContent);
}
}
async chat(messages: OllamaMessage[], tools: OllamaTool[] = []): Promise<OllamaMessage> {
// Bypass cache if tools are involved to prevent state corruption
if (tools.length > 0) {
return this.chatWithRetry(messages, tools, 0);
}
// Find the last user message
const lastUserMsg = [...messages].reverse().find((m) => m.role === 'user');
if (lastUserMsg && this.cacheService) {
const cached = await this.cacheService.getCache(lastUserMsg.content);
if (cached) {
return { role: 'assistant', content: cached };
}
}
const response = await this.chatWithRetry(messages, tools, 0);
if (this.cacheService && lastUserMsg) {
void this.cacheService.setCache(lastUserMsg.content, response.content);
}
return response;
}
async streamChatAsPromise(
messages: OllamaMessage[],
tools: OllamaTool[] = []
): Promise<OllamaMessage[]> {
const chunks: OllamaMessage[] = [];
): Promise<OllamaMessage> {
let content = '';
let role: OllamaMessage['role'] = 'assistant';
let toolCalls: OllamaMessage['tool_calls'];
for await (const chunk of this.streamChat(messages, tools)) {
chunks.push(chunk);
}
return chunks;
}
async chat(messages: OllamaMessage[], tools: OllamaTool[] = []): Promise<OllamaMessage> {
// Bypass cache if tools are involved
if (tools.length > 0) {
return this.chatWithRetry(messages, tools, 0);
}
const lastUserMsg = [...messages].reverse().find((m) => m.role === 'user');
if (lastUserMsg && this.cacheService) {
const cached = await this.cacheService.getCache(lastUserMsg.content);
if (cached) {
return { role: 'assistant', content: cached, tool_calls: [] };
role = chunk.role ?? role;
content += chunk.content ?? '';
if (chunk.tool_calls) {
toolCalls = [...(toolCalls ?? []), ...chunk.tool_calls];
}
}
const response = await this.chatWithRetry(messages, tools, 0);
if (this.cacheService && lastUserMsg) {
void this.cacheService.setCache(lastUserMsg.content, response.content);
}
return response;
return { role, content, tool_calls: toolCalls };
}
private async *streamChatWithRetry(
async *streamChatWithRetry(
messages: OllamaMessage[],
tools: OllamaTool[] = [],
attempt: number = 0
retryCount: number
): AsyncGenerator<OllamaMessage, void, unknown> {
// Create a local controller for this request instead of using the instance variable
const controller = new AbortController();
this.currentStreamController = controller;
let reader: ReadableStreamDefaultReader<Uint8Array> | null = null;
try {
const response = await this.fetchFn(`${this.baseURL}/api/chat`, {
method: 'POST',
headers: {
'Content-Type': 'application/json',
},
body: JSON.stringify({
model: this.model,
messages: messages,
tools: tools,
stream: true,
}),
signal: controller.signal,
});
if (!response.ok) {
throw new ApiError(`Ollama API error: ${response.status}`, response.status);
}
if (!response.body) {
throw new Error('No response body');
}
const contentType = response.headers?.get?.('content-type');
if (contentType && !contentType.includes('application/x-ndjson')) {
throw new Error('Invalid response format');
}
reader = response.body.getReader();
const decoder = new TextDecoder();
let buffer = '';
let malformedChunks = 0;
while (true) {
const { done, value } = await reader.read();
if (done) {
break;
}
buffer += decoder.decode(value);
const lines = buffer.split('\n');
buffer = lines.pop() || '';
for (const line of lines) {
if (line.trim() === '') {
continue;
}
let parsed: OllamaChatResponse;
try {
parsed = this.parseChatResponse(line);
} catch (error) {
malformedChunks++;
const errorMessage = error instanceof Error ? error.message : String(error);
Logger.warn(
`Skipped malformed chunk: ${line.slice(0, 50)}... - ${errorMessage}`,
'ollama-client'
);
if (malformedChunks > this.maxMalformedChunks) {
throw new Error('Too many malformed chunks in Ollama response');
}
continue;
}
if (parsed.error) {
throw new Error(`Ollama error: ${parsed.error}`);
}
yield this.normalizeMessage(parsed.message);
}
}
if (buffer.trim() !== '') {
let parsed: OllamaChatResponse | null = null;
try {
parsed = this.parseChatResponse(buffer);
} catch (error) {
const errorMessage = error instanceof Error ? error.message : String(error);
Logger.warn(
`Skipped malformed chunk: ${buffer.slice(0, 50)}... - ${errorMessage}`,
'ollama-client'
);
}
if (parsed?.error) {
throw new Error(`Ollama error: ${parsed.error}`);
}
if (parsed?.message) {
yield this.normalizeMessage(parsed.message);
}
}
} catch (error) {
if (retryCount < this.maxRetries && this.isRetryableError(error, controller)) {
const errorMessage = error instanceof Error ? error.message : String(error);
Logger.warn(`Retrying after error (attempt ${retryCount + 1}): ${errorMessage}`, 'ollama-client');
await new Promise((resolve) => setTimeout(resolve, 10 * Math.pow(2, retryCount)));
yield* this.streamChatWithRetry(messages, tools, retryCount + 1);
} else {
throw error;
}
} finally {
reader?.releaseLock();
if (this.currentStreamController === controller) {
this.currentStreamController = null;
}
}
}
async chatWithRetry(
messages: OllamaMessage[],
tools: OllamaTool[] = [],
retryCount: number
): Promise<OllamaMessage> {
const controller = new AbortController();
this.currentStreamController = controller;
@@ -123,231 +256,117 @@ export class OllamaClient {
},
body: JSON.stringify({
model: this.model,
messages,
tools,
stream: true,
}),
signal: controller.signal,
});
if (!response.ok) {
if (response.status >= 500 && attempt < this.maxRetries) {
const retryDelay = Math.pow(2, attempt) * 100;
Logger.warn(
`Network error (status ${response.status}), retrying in ${retryDelay}ms (attempt ${attempt + 1}/${this.maxRetries})`,
'ollama-client'
);
const retryTimeout = new Promise((resolve) => setTimeout(resolve, retryDelay));
const abortListener = () => {
Logger.info('Retry aborted by user', 'ollama-client');
};
const signal = controller.signal;
if (signal) {
signal.addEventListener('abort', abortListener);
try {
await Promise.race([
retryTimeout,
new Promise<void>((resolve) => {
signal.addEventListener('abort', () => resolve(), {
once: true,
});
}),
]);
} finally {
signal.removeEventListener('abort', abortListener);
}
// Check if signal was aborted before retrying
if (signal.aborted) {
throw new Error('Stream cancelled by user');
}
} else {
await retryTimeout;
}
yield* this.streamChatWithRetry(messages, tools, attempt + 1);
return;
}
throw new ApiError(`Ollama API error: ${response.status}`, response.status);
}
if (!response.body) {
throw new Error('No response body');
}
const contentType = response.headers.get('content-type');
if (!contentType || (!contentType.includes('ndjson') && !contentType.includes('json'))) {
throw new Error('Invalid response format');
}
const reader = response.body.getReader();
const decoder = new TextDecoder();
let buffer = '';
let malformedCount = 0;
const maxMalformed = 50;
try {
while (true) {
const { done, value } = await reader.read();
if (done) break;
buffer += decoder.decode(value, { stream: true });
const lines = buffer.split('\n');
buffer = lines.pop() ?? '';
for (const line of lines) {
if (!line.trim()) continue;
try {
const parsed = JSON.parse(line) as Record<string, unknown>;
this.throwIfOllamaError(parsed);
const message = this.toOllamaMessage(parsed.message);
if (!message) {
continue;
}
malformedCount = 0;
yield message;
} catch (error) {
if (error instanceof Error && error.message.startsWith('Ollama error:')) {
throw error;
}
malformedCount++;
if (malformedCount > maxMalformed) {
throw new Error('Too many malformed chunks in stream');
}
Logger.warn(
`Skipped malformed chunk: ${line.substring(0, 80)}... - ${(error as Error).message}`,
'ollama-client'
);
}
}
}
if (buffer.trim()) {
try {
const parsed = JSON.parse(buffer) as Record<string, unknown>;
this.throwIfOllamaError(parsed);
const message = this.toOllamaMessage(parsed.message);
if (message) {
yield message;
}
} catch (error) {
if (error instanceof Error && error.message.startsWith('Ollama error:')) {
throw error;
}
Logger.warn(
`Failed to parse final chunk: ${buffer.substring(0, 80)}...`,
'ollama-client'
);
}
}
} finally {
reader.releaseLock();
}
} finally {
// Abort the local controller to release underlying fetch resources if not already aborted
if (!controller.signal.aborted) {
controller.abort();
}
// Clean up the reference only if this is still the current stream
if (this.currentStreamController === controller) {
this.currentStreamController = null;
}
}
}
private async chatWithRetry(
messages: OllamaMessage[],
tools: OllamaTool[] = [],
attempt: number = 0
): Promise<OllamaMessage> {
// Create a local controller for this request instead of using the instance variable
const controller = new AbortController();
try {
const response = await this.fetchFn(`${this.baseURL}/api/chat`, {
method: 'POST',
headers: {
'Content-Type': 'application/json',
},
body: JSON.stringify({
model: this.model,
messages,
tools,
messages: messages,
tools: tools,
stream: false,
}),
signal: controller.signal,
});
if (!response.ok) {
if (response.status >= 500 && attempt < this.maxRetries) {
const retryDelay = Math.pow(2, attempt) * 100;
Logger.warn(
`Network error (status ${response.status}), retrying in ${retryDelay}ms (attempt ${attempt + 1}/${this.maxRetries})`,
'ollama-client'
);
const retryTimeout = new Promise((resolve) => setTimeout(resolve, retryDelay));
const abortListener = () => {
Logger.info('Retry aborted by user', 'ollama-client');
};
const signal = controller.signal;
if (signal) {
signal.addEventListener('abort', abortListener);
try {
await Promise.race([
retryTimeout,
new Promise<void>((resolve) => {
signal.addEventListener('abort', () => resolve(), {
once: true,
});
}),
]);
} finally {
signal.removeEventListener('abort', abortListener);
}
// Check if signal was aborted before retrying
if (signal.aborted) {
throw new Error('Stream cancelled by user');
}
} else {
await retryTimeout;
}
return this.chatWithRetry(messages, tools, attempt + 1);
}
throw new ApiError(`Ollama API error: ${response.status}`, response.status);
}
const data = (await response.json()) as OllamaChatResponse;
return (
this.toOllamaMessage(data.message) ?? { role: 'assistant', content: '', tool_calls: [] }
);
const data = await response.json() as unknown;
if (!this.isChatResponse(data)) {
return this.normalizeMessage();
}
return this.normalizeMessage(data.message);
} catch (error) {
if (retryCount < this.maxRetries && this.isRetryableError(error, controller)) {
const errorMessage = error instanceof Error ? error.message : String(error);
Logger.warn(`Retrying after error (attempt ${retryCount + 1}): ${errorMessage}`, 'ollama-client');
await new Promise((resolve) => setTimeout(resolve, 10 * Math.pow(2, retryCount)));
return this.chatWithRetry(messages, tools, retryCount + 1);
} else {
throw error;
}
} finally {
// Abort the local controller to release underlying fetch resources if not already aborted
if (!controller.signal.aborted) {
controller.abort();
if (this.currentStreamController === controller) {
this.currentStreamController = null;
}
}
}
private throwIfOllamaError(parsed: Record<string, unknown>): void {
if (parsed.error) {
throw new Error(`Ollama error: ${String(parsed.error)}`);
}
}
private toOllamaMessage(value: unknown): OllamaMessage | null {
if (!value || typeof value !== 'object') {
return null;
}
const record = value as Partial<OllamaMessage>;
private normalizeMessage(message?: Partial<OllamaMessage>): OllamaMessage {
return {
role: record.role ?? 'assistant',
content: typeof record.content === 'string' ? record.content : '',
tool_calls: record.tool_calls ?? [],
role: message?.role ?? 'assistant',
content: message?.content ?? '',
tool_calls: message?.tool_calls ?? [],
tool_call_id: message?.tool_call_id,
};
}
private parseChatResponse(raw: string): OllamaChatResponse {
const parsed: unknown = JSON.parse(raw);
if (!this.isChatResponse(parsed)) {
throw new Error('Invalid chat response');
}
return parsed;
}
private isChatResponse(data: unknown): data is OllamaChatResponse {
if (typeof data !== 'object' || data === null) {
return false;
}
const response = data as { message?: unknown; error?: unknown };
return (
(response.error === undefined || typeof response.error === 'string') &&
(response.message === undefined || this.isPartialMessage(response.message))
);
}
private isPartialMessage(data: unknown): data is Partial<OllamaMessage> {
if (typeof data !== 'object' || data === null) {
return false;
}
const message = data as {
role?: unknown;
content?: unknown;
tool_calls?: unknown;
tool_call_id?: unknown;
};
const validRole =
message.role === undefined ||
message.role === 'system' ||
message.role === 'user' ||
message.role === 'assistant' ||
message.role === 'tool';
return (
validRole &&
(message.content === undefined || typeof message.content === 'string') &&
(message.tool_calls === undefined || Array.isArray(message.tool_calls)) &&
(message.tool_call_id === undefined || typeof message.tool_call_id === 'string')
);
}
private isRetryableError(error: unknown, controller: AbortController): boolean {
if (controller.signal.aborted) {
return false;
}
if (error instanceof ApiError && error.statusCode >= 400 && error.statusCode < 500) {
return false;
}
if (error instanceof Error) {
if (error.name === 'AbortError') {
return false;
}
if (
error.message.startsWith('Ollama error:') ||
error.message.includes('Too many malformed chunks') ||
error.message === 'No response body' ||
error.message === 'Invalid response format'
) {
return false;
}
}
return true;
}
}
+67 -62
View File
@@ -1,107 +1,112 @@
// src/semantic-cache.ts
import { ChromaClient } from 'chromadb';
import { Logger } from './utils';
import { CacheConfig } from './types';
export class SemanticCacheService {
private client: ChromaClient;
private collection: ReturnType<ChromaClient['getOrCreateCollection']> | null = null;
// eslint-disable-next-line @typescript-eslint/no-explicit-any
private client: any | null = null;
// eslint-disable-next-line @typescript-eslint/no-explicit-any
private collection: any | null = null;
private config: CacheConfig;
private ollamaURL: string;
constructor(ollamaURL: string, config: CacheConfig) {
this.ollamaURL = ollamaURL.replace(/\/+$/, '');
this.config = config;
this.client = new ChromaClient({ path: 'http://localhost:8000' });
}
async initialize() {
async initialize(): Promise<void> {
if (!this.config.enabled) return;
try {
// Dynamic import — chromadb is optional and may not be installed.
// This prevents the plugin from crashing at load time if chromadb is absent.
const { ChromaClient } = await import('chromadb');
const chromaURL = this.config.chromaURL || 'http://localhost:8000';
this.client = new ChromaClient({ path: chromaURL });
this.collection = await this.client.getOrCreateCollection({
name: this.config.collectionName,
metadata: { 'hnsw:space': 'cosine' },
});
Logger.info(`Semantic cache initialized: ${this.config.collectionName}`, 'semantic-cache');
} catch (error) {
Logger.error(`Failed to initialize semantic cache: ${String(error)}`, 'semantic-cache');
const errorMessage = error instanceof Error ? error.message : String(error);
Logger.error(`Failed to initialize semantic cache: ${errorMessage}`, 'semantic-cache');
throw error;
}
}
private async getEmbedding(text: string): Promise<number[]> {
async getCache(query: string): Promise<string | null> {
if (!this.config.enabled || !this.collection) return null;
try {
const response = await fetch(`${this.ollamaURL}/api/embeddings`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
model: this.config.embeddingModel,
prompt: text,
}),
const results = await this.collection.query({
query_embeddings: await this.generateEmbedding(query),
n_results: 1,
where: { source: 'ollama' },
});
if (!response.ok) {
throw new Error(`Embedding failed with status ${response.status}`);
if (results.ids[0] && results.ids[0].length > 0) {
if (results.distances[0] && results.distances[0][0] > this.config.similarityThreshold) {
return results.documents[0][0];
}
}
const data = await response.json();
return data.embedding;
return null;
} catch (error) {
Logger.warn(`Failed to generate embedding: ${String(error)}`, 'semantic-cache');
return [];
}
}
async getCache(prompt: string): Promise<string | null> {
if (!this.collection || !this.config.enabled || !prompt.trim()) {
const errorMessage = error instanceof Error ? error.message : String(error);
Logger.warn(`Cache lookup failed: ${errorMessage}`, 'semantic-cache');
return null;
}
try {
const embedding = await this.getEmbedding(prompt);
if (!embedding.length) return null;
const results = await this.collection.query({
queryEmbeddings: [embedding],
nResults: 1,
include: ['metadatas', 'distances'],
});
// Cosine distance = 1 - cosine_similarity
// We want distance < (1 - threshold)
if (
results.distances &&
results.distances[0] &&
results.distances[0][0] < 1 - this.config.similarityThreshold
) {
Logger.debug('Semantic cache hit', 'semantic-cache');
return results.metadatas?.[0]?.[0]?.fullResponse ?? null;
}
} catch (error) {
Logger.warn(`Cache lookup failed: ${String(error)}`, 'semantic-cache');
}
return null;
}
async setCache(prompt: string, response: string): Promise<void> {
if (!this.collection || !this.config.enabled || !prompt.trim() || !response.trim()) {
return;
}
async setCache(query: string, response: string): Promise<void> {
if (!this.config.enabled || !this.collection) return;
try {
const embedding = await this.getEmbedding(prompt);
if (!embedding.length) return;
await this.collection.add({
ids: [crypto.randomUUID()],
embeddings: [embedding],
metadatas: [{ fullResponse: response }],
documents: [response],
embeddings: await this.generateEmbedding(query),
metadatas: [{ source: 'ollama' }],
});
Logger.debug('Cached new response', 'semantic-cache');
} catch (error) {
Logger.warn(`Cache write failed: ${String(error)}`, 'semantic-cache');
const errorMessage = error instanceof Error ? error.message : String(error);
Logger.warn(`Cache set failed: ${errorMessage}`, 'semantic-cache');
}
}
async clearCache(): Promise<void> {
if (!this.config.enabled || !this.collection) return;
try {
await this.collection.reset();
Logger.info('Semantic cache cleared', 'semantic-cache');
} catch (error) {
const errorMessage = error instanceof Error ? error.message : String(error);
Logger.error(`Failed to clear semantic cache: ${errorMessage}`, 'semantic-cache');
}
}
private async generateEmbedding(text: string): Promise<number[]> {
const response = await fetch(`${this.ollamaURL}/api/embeddings`, {
method: 'POST',
headers: {
'Content-Type': 'application/json',
},
body: JSON.stringify({
model: this.config.embeddingModel,
prompt: text,
}),
});
if (!response.ok) {
throw new Error(`Failed to generate embedding: ${response.status} ${response.statusText}`);
}
const data = await response.json();
return data.embedding;
}
}
-114
View File
@@ -1,114 +0,0 @@
"use strict";
// src/tool-executor.ts
Object.defineProperty(exports, "__esModule", { value: true });
exports.ToolExecutor = void 0;
const utils_1 = require("./utils");
// Disallow characters that are invalid in file paths
const INVALID_PATH_CHARS = /[<>:"|?*~]/;
const MAX_PATH_LENGTH = 200;
const FORBIDDEN_DIRS = ['.obsidian', '.git'];
class ToolExecutor {
constructor(vault, app) {
this.vault = vault;
this.app = app;
}
isSafePath(path) {
// Reject empty paths
if (!path || path.trim().length === 0) {
return false;
}
// Reject paths that are too long
if (path.length > MAX_PATH_LENGTH) {
return false;
}
// Reject paths with invalid characters
if (INVALID_PATH_CHARS.test(path)) {
return false;
}
// Reject absolute paths
if (path.startsWith('/') || path.startsWith('\\')) {
return false;
}
// Reject Windows drive letters (e.g., C:)
if (/^[a-zA-Z]:/.test(path)) {
return false;
}
// Reject paths containing backslashes (Windows-style path separators)
if (path.includes('\\')) {
return false;
}
// Reject paths that traverse to parent directories
const normalized = path.replace(/^(\.\/)+/, '');
if (normalized.split('/').includes('..')) {
return false;
}
// Reject forbidden directories
for (const dir of FORBIDDEN_DIRS) {
if (normalized.startsWith(`${dir}/`) || normalized.startsWith(`${dir}\\`)) {
return false;
}
if (normalized.includes(`/${dir}/`) || normalized.includes(`\\${dir}\\`)) {
return false;
}
}
return true;
}
async handleToolCall(toolCall) {
try {
const toolName = toolCall.function?.name;
const rawArgs = toolCall.function?.arguments;
if (!toolName) {
throw new Error('Tool name is required');
}
// Parse arguments whether they're a string or object
let parsedArgs;
if (typeof rawArgs === 'string') {
try {
parsedArgs = (0, utils_1.safeParseJson)(rawArgs);
}
catch {
throw new Error('Invalid JSON arguments');
}
}
else if (rawArgs && typeof rawArgs === 'object') {
parsedArgs = rawArgs;
}
else {
throw new Error('Arguments must be an object or JSON string');
}
// Process the tool call based on its type
switch (toolName) {
case 'create_file':
return await this.handleCreateFile(parsedArgs);
default:
return { success: false, message: `Unknown tool: ${toolName}` };
}
}
catch (error) {
const errorMessage = error instanceof Error ? error.message : String(error);
throw new Error(errorMessage);
}
}
async handleCreateFile(args) {
const path = args.path;
const content = args.content;
if (typeof path !== 'string') {
throw new Error('Path must be a string');
}
if (typeof content !== 'string') {
throw new Error('Content must be a string');
}
if (!this.isSafePath(path)) {
throw new Error('Invalid file path detected');
}
try {
await this.vault.create(path, content);
return { success: true, message: 'File created successfully' };
}
catch (error) {
const errorMessage = error instanceof Error ? error.message : String(error);
throw new Error(errorMessage);
}
}
}
exports.ToolExecutor = ToolExecutor;
+63 -1
View File
@@ -1,6 +1,6 @@
// src/tool-executor.ts
import { Vault, App } from 'obsidian';
import { Vault, App, TFile } from 'obsidian';
import type { ToolCall, ToolResult } from './types';
import { safeParseJson } from './utils';
@@ -95,6 +95,10 @@ export class ToolExecutor {
switch (toolName) {
case 'create_file':
return await this.handleCreateFile(parsedArgs);
case 'read_vault_file':
return await this.handleReadVaultFile(parsedArgs);
case 'search_vault_files':
return this.handleSearchVaultFiles(parsedArgs);
default:
return { success: false, message: `Unknown tool: ${toolName}` };
}
@@ -128,4 +132,62 @@ export class ToolExecutor {
throw new Error(errorMessage);
}
}
async executeTool(name: string, args: string | Record<string, unknown>): Promise<ToolResult> {
return this.handleToolCall({
id: crypto.randomUUID(),
type: 'function',
function: {
name,
arguments: args as string,
},
});
}
private async handleReadVaultFile(args: Record<string, unknown>): Promise<ToolResult> {
const path = args.path;
if (typeof path !== 'string') {
throw new Error('Path must be a string');
}
if (!this.isSafePath(path)) {
throw new Error('Invalid file path detected');
}
const file = this.vault.getAbstractFileByPath(path);
if (!(file instanceof TFile)) {
throw new Error(`File not found: ${path}`);
}
const content = await this.vault.cachedRead(file);
return {
success: true,
message: 'File read successfully',
data: { path, content },
};
}
private handleSearchVaultFiles(args: Record<string, unknown>): ToolResult {
const query = args.query;
const limitArg = args.limit;
if (typeof query !== 'string') {
throw new Error('Query must be a string');
}
const limit = typeof limitArg === 'number' && Number.isFinite(limitArg) ? limitArg : 10;
const normalizedQuery = query.toLowerCase();
const files = this.vault
.getMarkdownFiles()
.filter((file) => file.path.toLowerCase().includes(normalizedQuery))
.slice(0, limit)
.map((file) => ({ path: file.path, basename: file.basename }));
return {
success: true,
message: `Found ${files.length} matching files`,
data: files,
};
}
}
-72
View File
@@ -1,72 +0,0 @@
"use strict";
// src/types.ts
Object.defineProperty(exports, "__esModule", { value: true });
exports.PathValidationError = exports.ToolExecutionError = exports.StreamingError = exports.ValidationError = exports.ApiError = exports.NetworkError = exports.OllamaError = exports.ErrorType = void 0;
// ============================================================
// Error Type Hierarchy
// ============================================================
var ErrorType;
(function (ErrorType) {
ErrorType["NETWORK_ERROR"] = "network_error";
ErrorType["API_ERROR"] = "api_error";
ErrorType["VALIDATION_ERROR"] = "validation_error";
ErrorType["STREAMING_ERROR"] = "streaming_error";
ErrorType["TOOL_EXECUTION_ERROR"] = "tool_execution_error";
ErrorType["PATH_VALIDATION_ERROR"] = "path_validation_error";
ErrorType["UNKNOWN_ERROR"] = "unknown_error";
})(ErrorType || (exports.ErrorType = ErrorType = {}));
class OllamaError extends Error {
constructor(message, type) {
super(message);
this.type = type;
Object.setPrototypeOf(this, OllamaError.prototype);
}
}
exports.OllamaError = OllamaError;
class NetworkError extends OllamaError {
constructor(message, statusCode) {
super(message, ErrorType.NETWORK_ERROR);
this.statusCode = statusCode;
Object.setPrototypeOf(this, NetworkError.prototype);
}
}
exports.NetworkError = NetworkError;
class ApiError extends OllamaError {
constructor(message, statusCode) {
super(message, ErrorType.API_ERROR);
this.statusCode = statusCode;
Object.setPrototypeOf(this, ApiError.prototype);
}
}
exports.ApiError = ApiError;
class ValidationError extends OllamaError {
constructor(message, details) {
super(message, ErrorType.VALIDATION_ERROR);
this.details = details;
Object.setPrototypeOf(this, ValidationError.prototype);
}
}
exports.ValidationError = ValidationError;
class StreamingError extends OllamaError {
constructor(message) {
super(message, ErrorType.STREAMING_ERROR);
Object.setPrototypeOf(this, StreamingError.prototype);
}
}
exports.StreamingError = StreamingError;
class ToolExecutionError extends OllamaError {
constructor(message, toolName) {
super(message, ErrorType.TOOL_EXECUTION_ERROR);
this.toolName = toolName;
Object.setPrototypeOf(this, ToolExecutionError.prototype);
}
}
exports.ToolExecutionError = ToolExecutionError;
class PathValidationError extends OllamaError {
constructor(message, path) {
super(message, ErrorType.PATH_VALIDATION_ERROR);
this.path = path;
Object.setPrototypeOf(this, PathValidationError.prototype);
}
}
exports.PathValidationError = PathValidationError;
+205 -80
View File
@@ -35,24 +35,19 @@ export class NetworkError extends OllamaError {
}
export class ApiError extends OllamaError {
public readonly statusCode?: number;
public readonly statusCode: number;
constructor(message: string, statusCode?: number) {
constructor(message: string, statusCode: number) {
super(message, ErrorType.API_ERROR);
this.statusCode = statusCode;
Object.setPrototypeOf(this, ApiError.prototype);
}
}
export interface ValidationFieldDetails {
field?: string;
message?: string;
}
export class ValidationError extends OllamaError {
public readonly details?: ValidationFieldDetails;
public readonly details?: { field?: string; message?: string };
constructor(message: string, details?: ValidationFieldDetails) {
constructor(message: string, details?: { field?: string; message?: string }) {
super(message, ErrorType.VALIDATION_ERROR);
this.details = details;
Object.setPrototypeOf(this, ValidationError.prototype);
@@ -69,7 +64,7 @@ export class StreamingError extends OllamaError {
export class ToolExecutionError extends OllamaError {
public readonly toolName: string;
constructor(message: string, toolName: string) {
constructor(message: string, toolName: string = 'unknown') {
super(message, ErrorType.TOOL_EXECUTION_ERROR);
this.toolName = toolName;
Object.setPrototypeOf(this, ToolExecutionError.prototype);
@@ -79,13 +74,95 @@ export class ToolExecutionError extends OllamaError {
export class PathValidationError extends OllamaError {
public readonly path: string;
constructor(message: string, path: string) {
constructor(message: string, path: string = '') {
super(message, ErrorType.PATH_VALIDATION_ERROR);
this.path = path;
Object.setPrototypeOf(this, PathValidationError.prototype);
}
}
// ============================================================
// Ollama Types
// ============================================================
export interface OllamaMessage {
role: 'system' | 'user' | 'assistant' | 'tool';
content: string;
tool_calls?: OllamaToolCall[];
tool_call_id?: string;
}
export interface OllamaToolCall {
id: string;
type: 'function';
function: {
name: string;
arguments: string;
};
}
export interface OllamaTool {
type: 'function';
function: {
name: string;
description: string;
parameters: {
type: 'object';
properties: {
[key: string]: {
type: string;
description?: string;
};
};
required?: string[];
};
};
}
export type ToolCall = OllamaToolCall;
export interface ToolResult {
success: boolean;
message: string;
data?: unknown;
id?: string;
}
export interface VaultIndexEntry {
path: string;
title: string;
content: string;
score: number;
}
export interface ChatMessage {
id: string;
role: 'user' | 'assistant';
content: string;
timestamp: number;
isStreaming?: boolean;
tool_calls?: OllamaToolCall[];
// Refinement tracking
originalQuery?: string;
originalAssistantAnswer?: string;
userCritique?: string;
isRefinement?: boolean;
}
export interface DependencyGraph {
nodes: {
id: string;
concept: string;
filePath: string;
preview: string;
}[];
edges: {
source: string;
target: string;
weight?: number;
}[];
}
// ============================================================
// Plugin Configuration
// ============================================================
@@ -95,6 +172,7 @@ export interface CacheConfig {
similarityThreshold: number;
collectionName: string;
embeddingModel: string;
chromaURL?: string;
}
export interface PluginSettings {
@@ -107,80 +185,127 @@ export interface PluginSettings {
}
// ============================================================
// Ollama Protocol Types
// Workflow Engine Types (Multi-Step/Chained Reasoning)
// ============================================================
export interface OllamaTool {
type: 'function';
function: {
name: string;
description: string;
parameters: {
type: 'object';
properties: Record<string, unknown>;
required: string[];
};
};
}
/**
* Supported step types in a workflow.
* - 'llm': Call the LLM with a prompt (supports variable interpolation).
* - 'vault_search': Search the vault using VaultIndexer.
* - 'tool': Execute a tool via ToolExecutor.
* - 'format': Transform/format data using a template string.
*/
export type WorkflowStepType = 'llm' | 'vault_search' | 'tool' | 'format';
export interface OllamaToolCall {
id: string;
type: 'function';
function: {
name: string;
arguments: string | Record<string, unknown>;
};
}
export interface OllamaMessage {
role: 'system' | 'user' | 'assistant' | 'tool';
content: string;
tool_calls?: OllamaToolCall[];
}
// ============================================================
// Tool Execution Types
// ============================================================
export interface ToolCall {
id: string;
type: 'function';
function: {
name: string;
arguments: string | Record<string, unknown>;
};
}
export interface ToolResult {
/**
* Result of executing a single workflow step.
*/
export interface WorkflowStepResult {
stepId: string;
stepName: string;
success: boolean;
message: string;
}
export interface ExecutionResult {
success: boolean;
output: string;
}
// ============================================================
// Chat Message Types
// ============================================================
export interface ChatMessage {
id: string;
role: 'user' | 'assistant';
content: string;
data: unknown;
error?: string;
timestamp: number;
isStreaming?: boolean;
tool_calls?: ToolCall[];
}
// ============================================================
// Vault Index Types
// ============================================================
export interface VaultIndexEntry {
path: string;
title: string;
content: string;
score: number;
/**
* Configuration for a single workflow step.
*/
export interface WorkflowStep {
id: string;
type: WorkflowStepType;
name: string;
description?: string;
config: WorkflowStepConfig;
/** Optional: if set, this step waits for the named step to complete before running */
dependsOn?: string;
}
/**
* Union of all possible step configurations based on step type.
*/
export type WorkflowStepConfig =
| LlmStepConfig
| VaultSearchStepConfig
| ToolStepConfig
| FormatStepConfig;
/**
* Call the LLM with a system prompt and user prompt.
* Both fields support variable interpolation via {{variable_name}} syntax.
* Variables can reference prior step outputs as {{stepId.output}}.
*/
export interface LlmStepConfig {
type: 'llm';
systemPrompt?: string;
userPrompt: string;
includeToolCalls?: boolean;
}
/**
* Search the vault for relevant notes.
* The query field supports variable interpolation.
*/
export interface VaultSearchStepConfig {
type: 'vault_search';
query: string;
limit?: number;
tagFilter?: string;
}
/**
* Execute a tool (e.g., read_vault_file, create_file, search_vault_files).
* The args field supports variable interpolation.
*/
export interface ToolStepConfig {
type: 'tool';
toolName: string;
args: Record<string, unknown>;
}
/**
* Format/transform previous step output into a template.
* The template supports {{stepId.output}} and {{stepId.output.property}} syntax.
*/
export interface FormatStepConfig {
type: 'format';
template: string;
outputFormat?: 'markdown' | 'text' | 'json';
}
/**
* Full definition of a workflow (chain of steps).
*/
export interface WorkflowDefinition {
id: string;
name: string;
description: string;
steps: WorkflowStep[];
}
/**
* Mutable context passed through workflow execution.
* Stores intermediate results for variable interpolation.
*/
export interface WorkflowExecutionContext {
/** Keyed by step ID -> step result data */
variables: Map<string, unknown>;
/** Full step results for debugging/logging */
stepResults: WorkflowStepResult[];
/** Conversation history to pass to LLM steps */
conversationHistory: OllamaMessage[];
}
/**
* Final result after executing all workflow steps.
*/
export interface WorkflowExecutionResult {
workflowId: string;
workflowName: string;
success: boolean;
stepResults: WorkflowStepResult[];
/** Output of the last successfully executed step */
finalOutput: unknown;
error?: string;
}
-167
View File
@@ -1,167 +0,0 @@
"use strict";
Object.defineProperty(exports, "__esModule", { value: true });
exports.Logger = exports.LogLevel = void 0;
exports.validateOllamaUrl = validateOllamaUrl;
exports.validateModelName = validateModelName;
exports.validatePluginSettings = validatePluginSettings;
exports.safeParseJson = safeParseJson;
var LogLevel;
(function (LogLevel) {
LogLevel[LogLevel["DEBUG"] = 0] = "DEBUG";
LogLevel[LogLevel["INFO"] = 1] = "INFO";
LogLevel[LogLevel["WARN"] = 2] = "WARN";
LogLevel[LogLevel["ERROR"] = 3] = "ERROR";
})(LogLevel || (exports.LogLevel = LogLevel = {}));
const SEVERITY_ORDER = {
debug: LogLevel.DEBUG,
info: LogLevel.INFO,
warn: LogLevel.WARN,
error: LogLevel.ERROR,
};
class Logger {
static setLevel(level) {
if (typeof level === 'string') {
const lowerLevel = level.toLowerCase();
Logger.minLevel = SEVERITY_ORDER[lowerLevel] ?? LogLevel.DEBUG;
}
else {
Logger.minLevel = level;
}
}
static debug(message, category = 'general') {
if (LogLevel.DEBUG >= Logger.minLevel) {
console.debug(`[${category}] DEBUG: ${message}`);
}
}
static info(message, category = 'general') {
if (LogLevel.INFO >= Logger.minLevel) {
console.info(`[${category}] INFO: ${message}`);
}
}
static warn(message, category = 'general') {
if (LogLevel.WARN >= Logger.minLevel) {
console.warn(`[${category}] WARN: ${message}`);
}
}
static error(message, category = 'general') {
if (LogLevel.ERROR >= Logger.minLevel) {
console.error(`[${category}] ERROR: ${message}`);
}
}
}
exports.Logger = Logger;
Logger.minLevel = LogLevel.DEBUG;
// ==================== URL & Model Validation ====================
function validateOllamaUrl(url) {
if (typeof url !== 'string' || !url.trim()) {
return { valid: false, error: 'URL cannot be empty' };
}
const trimmedUrl = url.trim();
if (trimmedUrl.endsWith('/')) {
return { valid: false, error: 'URL should not end with a slash' };
}
try {
const parsed = new URL(trimmedUrl);
if (parsed.protocol !== 'http:' && parsed.protocol !== 'https:') {
return { valid: false, error: 'Must be a valid HTTP or HTTPS URL' };
}
return { valid: true };
}
catch {
return { valid: false, error: 'Must be a valid HTTP or HTTPS URL' };
}
}
function validateModelName(model) {
if (typeof model !== 'string') {
return { valid: false, error: 'Model name must be a string' };
}
const trimmedModel = model.trim();
// Explicit check for empty string after trimming
if (!trimmedModel || trimmedModel.length === 0) {
return { valid: false, error: 'Model name cannot be empty' };
}
if (trimmedModel.length < 2) {
return { valid: false, error: 'Model name must be at least 2 characters long' };
}
if (trimmedModel.length > 100) {
return { valid: false, error: 'Model name must be less than 100 characters long' };
}
if (!/^[a-zA-Z0-9._:-]+$/.test(trimmedModel)) {
return {
valid: false,
error: 'Model name can only contain letters, numbers, dots, dashes, underscores, and colons',
};
}
return { valid: true };
}
function validatePluginSettings(settings) {
const errors = [];
const urlValidation = validateOllamaUrl(settings.ollamaUrl);
if (!urlValidation.valid) {
errors.push(`Invalid Ollama URL: ${urlValidation.error}`);
}
const modelValidation = validateModelName(settings.model);
if (!modelValidation.valid) {
errors.push(`Invalid Model Name: ${modelValidation.error}`);
}
return errors;
}
// ==================== Safe JSON Parsing ====================
const MAX_JSON_SIZE = 1000000;
const MAX_JSON_NESTING = 24;
function countNestingDepth(value, depth = 0) {
if (depth > MAX_JSON_NESTING) {
return depth;
}
if (Array.isArray(value)) {
return Math.max(...value.map((item) => countNestingDepth(item, depth + 1)), depth);
}
if (value !== null && typeof value === 'object') {
const entries = Object.values(value);
if (entries.length === 0)
return depth;
return Math.max(...entries.map((item) => countNestingDepth(item, depth + 1)), depth);
}
return depth;
}
function safeParseJson(jsonString) {
if (typeof jsonString !== 'string') {
throw new Error('Input must be a string');
}
if (jsonString.length > MAX_JSON_SIZE) {
throw new Error('JSON input too large');
}
let parsed;
try {
parsed = JSON.parse(jsonString);
}
catch {
throw new Error('Invalid JSON');
}
// Check for dangerous prototype pollution patterns in object keys only
const checkDangerousPatterns = (obj) => {
if (typeof obj !== 'object' || obj === null) {
return false;
}
const dangerousKeys = ['constructor', 'prototype', '__proto__'];
if (dangerousKeys.some((key) => Object.keys(obj).includes(key))) {
return true;
}
// Recursively check nested objects (own properties only)
for (const key of Object.keys(obj)) {
if (checkDangerousPatterns(obj[key])) {
return true;
}
}
return false;
};
if (checkDangerousPatterns(parsed)) {
throw new Error('dangerous code pattern detected');
}
// Check nesting depth
if (countNestingDepth(parsed) > MAX_JSON_NESTING) {
throw new Error('JSON nesting too deep');
}
return parsed;
}
// ==================== Markdown Utilities ====================
+3 -2
View File
@@ -167,8 +167,9 @@ export function safeParseJson(jsonString: string): unknown {
}
// Recursively check nested objects (own properties only)
for (const key of Object.keys(obj as Record<string, unknown>)) {
if (checkDangerousPatterns((obj as Record<string, unknown>)[key])) {
const record = obj as Record<string, unknown>;
for (const key of Object.keys(obj)) {
if (checkDangerousPatterns(record[key])) {
return true;
}
}
-254
View File
@@ -1,254 +0,0 @@
"use strict";
// src/vault-indexer.ts
Object.defineProperty(exports, "__esModule", { value: true });
exports.InMemoryCache = exports.VaultIndexer = void 0;
exports.createVaultIndexerWithCache = createVaultIndexerWithCache;
const utils_1 = require("./utils");
class InMemoryCache {
constructor() {
this.store = new Map();
}
get(key) {
return Promise.resolve(this.store.get(key) || null);
}
put(key, value) {
this.store.set(key, value);
return Promise.resolve();
}
clear() {
this.store.clear();
return Promise.resolve();
}
}
exports.InMemoryCache = InMemoryCache;
class VaultIndexer {
constructor(vault, cache) {
this.vault = null;
this.vault = vault;
this.cache = cache;
}
async searchVault(query, limit = 5) {
if (!query || !query.trim()) {
return [];
}
if (!this.vault) {
throw new Error('Vault-like object not provided to VaultIndexer');
}
const cacheKey = `query:${query.trim()}:limit:${limit}`;
if (this.cache) {
let cachedResults;
try {
cachedResults = await this.cache.get(cacheKey);
}
catch {
// Ignore cache retrieval errors and continue with normal processing
cachedResults = null;
}
if (cachedResults) {
try {
const parsedResults = JSON.parse(cachedResults);
return parsedResults.slice(0, limit);
}
catch {
// Ignore cache parse errors and continue with normal processing
}
}
}
const queryTokens = this.tokenize(query.trim());
const vault = this.vault;
const allFiles = vault.getMarkdownFiles();
const results = await this.processFilesInBatches(vault, allFiles, queryTokens);
const filteredResults = results.sort((a, b) => b.score - a.score).slice(0, limit);
if (this.cache) {
try {
await this.cache.put(cacheKey, JSON.stringify(filteredResults));
}
catch (error) {
utils_1.Logger.warn(`Failed to cache results for query "${query}": ${error instanceof Error ? error.message : String(error)}`, 'vault-indexer');
}
}
return filteredResults;
}
async processFilesInBatches(vault, files, queryTokens) {
const batchSize = 10;
const results = [];
const seenPaths = new Set();
for (let i = 0; i < files.length; i += batchSize) {
const batch = files.slice(i, i + batchSize);
const batchResults = await Promise.all(batch.map(async (file) => {
try {
const content = await vault.read(file);
const tokenized = this.tokenizeContent(content);
const scoreResult = this.calculateWeightedScore(tokenized, queryTokens, file);
if (scoreResult.score > 0) {
const entry = {
path: file.path,
title: file.basename.replace(/\.md$/, ''),
content: content.substring(0, 500),
score: scoreResult.score,
};
if (!seenPaths.has(entry.path)) {
seenPaths.add(entry.path);
return entry;
}
return null;
}
return null;
}
catch (error) {
utils_1.Logger.warn(`Failed to process ${file.path}: ${error instanceof Error ? error.message : String(error)}`, 'vault-indexer');
return null;
}
}));
const validResults = batchResults.filter((result) => result !== null);
results.push(...validResults);
}
return results;
}
tokenize(text) {
const stopWords = new Set([
'the',
'a',
'an',
'and',
'or',
'but',
'is',
'are',
'was',
'were',
'in',
'on',
'at',
'to',
'of',
'for',
'with',
'as',
'by',
'it',
'its',
'that',
'this',
'these',
'those',
]);
return text
.toLowerCase()
.split(/\W+/)
.filter((token) => token.length > 1 && !stopWords.has(token));
}
tokenizeContent(content) {
const tokens = [];
const headings = [];
const frontmatter = {};
let firstParagraph;
const frontmatterMatch = content.match(/^---(.*?)---/s);
if (frontmatterMatch) {
try {
const frontmatterContent = frontmatterMatch[1];
const lines = frontmatterContent.trim().split('\n');
for (const line of lines) {
const [key, ...valueParts] = line.split(':');
if (!key)
continue;
const value = valueParts.join(':').trim();
if (key.trim() === 'title') {
if (value) {
frontmatter.title = value;
}
}
else if (key.trim() === 'tags') {
if (value) {
frontmatter.tags = value;
}
}
}
}
catch {
utils_1.Logger.warn('Failed to parse frontmatter', 'vault-indexer');
}
}
const headingMatches = content.match(/^#{1,6} (.*?)$/gm);
if (headingMatches) {
headings.push(...headingMatches.map((h) => h.replace(/^#{1,6} /, '')));
}
const paragraphMatch = content.match(/^([^#]*?)(?=#|\n\n|$)/s);
if (paragraphMatch) {
firstParagraph = paragraphMatch[1].trim();
}
const allText = content
.replace(/^---.*?---/s, '')
.replace(/^#.*?$/gm, '')
.replace(/```.*?```/gs, '')
.replace(/`.*?`/g, '')
.replace(/\[.*?\]\(.*?\)/g, '');
tokens.push(...this.tokenize(allText));
return { tokens, headings, frontmatter, firstParagraph };
}
calculateWeightedScore(tokenized, queryTokens, file) {
let totalScore = 0;
const matchedTokens = new Set();
for (const queryToken of queryTokens) {
let tokenScore = 0;
const stemmed = this.stemToken(queryToken);
let matched = false;
if (tokenized.frontmatter?.title &&
this.exactMatch(tokenized.frontmatter.title, queryToken)) {
tokenScore += 3;
matched = true;
}
else if (file &&
file.basename &&
this.exactMatch(file.basename.replace(/\.md$/, ''), queryToken)) {
tokenScore += 3;
matched = true;
}
if (tokenized.frontmatter?.tags && this.exactMatch(tokenized.frontmatter.tags, queryToken)) {
tokenScore += 2.5;
matched = true;
}
if (tokenized.headings.some((heading) => heading.toLowerCase().includes(stemmed))) {
tokenScore += 5;
matched = true;
}
if (tokenized.firstParagraph && tokenized.firstParagraph.toLowerCase().includes(stemmed)) {
tokenScore += 1.5;
matched = true;
}
if (tokenized.tokens.includes(stemmed)) {
tokenScore += 1;
matched = true;
}
if (matched) {
totalScore += tokenScore;
matchedTokens.add(queryToken);
}
}
return {
score: totalScore,
matchedFields: Array.from(matchedTokens),
};
}
stemToken(token) {
// Improved stemmer that handles edge cases
if (token.length <= 3)
return token; // Don't stem very short tokens
if (token.endsWith('s'))
return token.slice(0, -1);
if (token.endsWith('ed') && token.length > 4)
return token.slice(0, -2); // Don't stem 3-letter words ending in ed
if (token.endsWith('ing') && token.length > 5)
return token.slice(0, -3); // Don't stem 4-letter words ending in ing
return token;
}
exactMatch(content, token) {
const stemmedToken = this.stemToken(token);
return content.toLowerCase().includes(stemmedToken);
}
}
exports.VaultIndexer = VaultIndexer;
// Convenience method to create a VaultIndexer with an in-memory cache
function createVaultIndexerWithCache(vault) {
return new VaultIndexer(vault, new InMemoryCache());
}
+194 -256
View File
@@ -1,334 +1,272 @@
// src/vault-indexer.ts
import { VaultIndexEntry } from './types';
import { Vault, TFile } from 'obsidian';
import { Logger } from './utils';
import { Cache } from './cache';
interface Cache {
get(key: string): Promise<string | null>;
put(key: string, value: string): Promise<void>;
clear(): Promise<void>;
interface ParsedFrontmatter {
title?: string;
tags?: string;
[key: string]: unknown;
}
class InMemoryCache implements Cache {
private store: Map<string, string>;
interface TokenizedContent {
title: string;
headings: string[];
frontmatter: ParsedFrontmatter;
firstParagraph: string;
content: string;
basename: string;
}
constructor() {
this.store = new Map();
}
interface VaultEntry {
file: TFile;
title: string;
frontmatter: ParsedFrontmatter;
headings: string[];
content: string;
basename: string;
score: number;
}
export class InMemoryCache implements Cache {
private store = new Map<string, string>();
get(key: string): Promise<string | null> {
return Promise.resolve(this.store.get(key) || null);
return Promise.resolve(this.store.get(key) ?? null);
}
put(key: string, value: string): Promise<void> {
this.store.set(key, value);
return Promise.resolve();
}
clear(): Promise<void> {
this.store.clear();
return Promise.resolve();
}
}
interface Frontmatter {
title?: string;
tags?: string;
}
const STOP_WORDS = new Set([
'a', 'an', 'the', 'is', 'it', 'in', 'on', 'at', 'to', 'for', 'of',
'and', 'or', 'but', 'with', 'by', 'from', 'up', 'about', 'into',
'this', 'that', 'these', 'those', 'be', 'been', 'being', 'have',
'has', 'had', 'do', 'does', 'did', 'will', 'would', 'could', 'should',
'may', 'might', 'can', 'are', 'was', 'were', 'as', 'so', 'if', 'not',
'no', 'my', 'your', 'our', 'its', 'we', 'you', 'he', 'she', 'they',
]);
interface VaultFile {
basename: string;
path: string;
}
const CONTENT_PREVIEW_LENGTH = 500;
interface VaultLike {
getMarkdownFiles(): VaultFile[];
read(file: VaultFile): Promise<string>;
}
interface TokenizedContent {
tokens: string[];
headings: string[];
frontmatter: Frontmatter;
firstParagraph?: string;
}
interface ScoreResult {
score: number;
matchedFields: string[];
}
class VaultIndexer {
private vault: VaultLike | null = null;
export class VaultIndexer {
private vault: Vault;
private cache?: Cache;
private readonly SCORING_WEIGHTS = {
TITLE: 5,
FRONTMATTER_TITLE: 4,
FRONTMATTER_TAGS: 3,
HEADINGS: 2,
CONTENT: 1,
};
constructor(vault: VaultLike, cache?: Cache) {
constructor(vault: Vault, cache?: Cache) {
this.vault = vault;
this.cache = cache;
}
async searchVault(query: string, limit: number = 5): Promise<VaultIndexEntry[]> {
tokenize(text: string): string[] {
return text
.toLowerCase()
.replace(/[^\w\s]/g, '')
.split(/\s+/)
.filter((token) => token.length > 1 && !STOP_WORDS.has(token));
}
tokenizeContent(content: string, file: TFile): TokenizedContent {
const parsed = this.parseMarkdown(content);
const bodyWithoutFrontmatter = content.replace(/^---[\s\S]*?---\n/, '');
const paragraphs = bodyWithoutFrontmatter
.split(/\n\n+/)
.map((p) => p.trim())
.filter((p) => p && !p.startsWith('#'));
const firstParagraph = paragraphs[0] || '';
return {
title: parsed.title || file.basename,
headings: parsed.headings,
frontmatter: parsed.frontmatter,
firstParagraph,
content: parsed.content,
basename: file.basename,
};
}
calculateWeightedScore(
tokenized: TokenizedContent,
queryTokens: string[]
): { score: number } {
let score = 0;
for (const token of queryTokens) {
if (tokenized.frontmatter?.title && this.exactMatch(tokenized.frontmatter.title, token)) {
score += this.SCORING_WEIGHTS.FRONTMATTER_TITLE;
}
if (tokenized.basename && this.exactMatch(tokenized.basename, token)) {
score += this.SCORING_WEIGHTS.FRONTMATTER_TITLE;
}
if (tokenized.frontmatter?.tags && this.exactMatch(tokenized.frontmatter.tags, token)) {
score += this.SCORING_WEIGHTS.FRONTMATTER_TAGS;
}
if (tokenized.headings.some((h) => h.toLowerCase().includes(token.toLowerCase()))) {
score += this.SCORING_WEIGHTS.HEADINGS;
}
if (tokenized.content.toLowerCase().includes(token.toLowerCase())) {
score += this.SCORING_WEIGHTS.CONTENT;
}
if (tokenized.title && this.exactMatch(tokenized.title, token)) {
score += this.SCORING_WEIGHTS.TITLE;
}
}
return { score };
}
async getVaultEntries(): Promise<VaultEntry[]> {
const files = this.vault.getMarkdownFiles();
const entries: VaultEntry[] = [];
for (const file of files) {
try {
const content =
typeof this.vault.cachedRead === 'function'
? await this.vault.cachedRead(file)
: await this.vault.read(file);
const parsed = this.parseMarkdown(content);
entries.push({
file: file,
title: parsed.frontmatter.title || file.basename,
frontmatter: parsed.frontmatter,
headings: parsed.headings,
content: parsed.content.slice(0, CONTENT_PREVIEW_LENGTH),
basename: file.basename,
score: 0,
});
} catch (error) {
const errorMessage = error instanceof Error ? error.message : String(error);
Logger.warn(`Failed to read file ${file.path}: ${errorMessage}`, 'vault-indexer');
}
}
return entries;
}
async searchVault(query: string, limit = 3): Promise<VaultEntry[]> {
if (!query || !query.trim()) {
return [];
}
if (!this.vault) {
throw new Error('Vault-like object not provided to VaultIndexer');
}
const cacheKey = `query:${query.trim()}:limit:${limit}`;
if (this.cache) {
let cachedResults;
let cachedResults: string | null = null;
try {
cachedResults = await this.cache.get(cacheKey);
} catch {
// Ignore cache retrieval errors and continue with normal processing
cachedResults = null;
}
if (cachedResults) {
try {
const parsedResults = JSON.parse(cachedResults) as VaultIndexEntry[];
// eslint-disable-next-line @typescript-eslint/no-unsafe-assignment
const parsedResults: VaultEntry[] = JSON.parse(cachedResults);
return parsedResults.slice(0, limit);
} catch {
// Ignore cache parse errors and continue with normal processing
// ignore parse errors
}
}
}
const queryTokens = this.tokenize(query.trim());
const vault = this.vault;
const allFiles = vault.getMarkdownFiles();
const results = await this.processFilesInBatches(vault, allFiles, queryTokens);
const queryTokens = this.tokenize(query);
if (queryTokens.length === 0) {
return [];
}
const filteredResults = results.sort((a, b) => b.score - a.score).slice(0, limit);
const entries = await this.getVaultEntries();
const scored = entries
.map((entry) => {
const { score } = this.calculateWeightedScore(
{
title: entry.title,
headings: entry.headings,
frontmatter: entry.frontmatter,
firstParagraph: '',
content: entry.content,
basename: entry.basename,
},
queryTokens
);
return { ...entry, score };
})
.filter((e) => e.score > 0);
scored.sort((a, b) => b.score - a.score);
const results = scored.slice(0, limit);
if (this.cache) {
try {
await this.cache.put(cacheKey, JSON.stringify(filteredResults));
await this.cache.put(cacheKey, JSON.stringify(results));
} catch (error) {
const errorMessage = error instanceof Error ? error.message : String(error);
Logger.warn(
`Failed to cache results for query "${query}": ${error instanceof Error ? error.message : String(error)}`,
`Failed to cache results for query "${query}": ${errorMessage}`,
'vault-indexer'
);
}
}
return filteredResults;
}
private async processFilesInBatches(
vault: VaultLike,
files: VaultFile[],
queryTokens: string[]
): Promise<VaultIndexEntry[]> {
const batchSize = 10;
const results: VaultIndexEntry[] = [];
const seenPaths = new Set<string>();
for (let i = 0; i < files.length; i += batchSize) {
const batch = files.slice(i, i + batchSize);
const batchResults = await Promise.all(
batch.map(async (file) => {
try {
const content = await vault.read(file);
const tokenized = this.tokenizeContent(content);
const scoreResult = this.calculateWeightedScore(tokenized, queryTokens, file);
if (scoreResult.score > 0) {
const entry: VaultIndexEntry = {
path: file.path,
title: file.basename.replace(/\.md$/, ''),
content: content.substring(0, 500),
score: scoreResult.score,
};
if (!seenPaths.has(entry.path)) {
seenPaths.add(entry.path);
return entry;
}
return null;
}
return null;
} catch (error) {
Logger.warn(
`Failed to process ${file.path}: ${error instanceof Error ? error.message : String(error)}`,
'vault-indexer'
);
return null;
}
})
);
const validResults = batchResults.filter(
(result): result is NonNullable<typeof result> => result !== null
);
results.push(...validResults);
}
return results;
}
private tokenize(text: string): string[] {
const stopWords = new Set([
'the',
'a',
'an',
'and',
'or',
'but',
'is',
'are',
'was',
'were',
'in',
'on',
'at',
'to',
'of',
'for',
'with',
'as',
'by',
'it',
'its',
'that',
'this',
'these',
'those',
]);
return text
.toLowerCase()
.split(/\W+/)
.filter((token) => token.length > 1 && !stopWords.has(token));
private stemToken(token: string): string {
if (token.endsWith('ing') && token.length > 4) return token.slice(0, -3);
if (token.endsWith('ed') && token.length > 3) return token.slice(0, -2);
if (token.endsWith('s') && token.length > 2) return token.slice(0, -1);
return token;
}
private tokenizeContent(content: string): TokenizedContent {
const tokens: string[] = [];
const headings: string[] = [];
const frontmatter: Frontmatter = {};
let firstParagraph: string | undefined;
private exactMatch(text: string | undefined, queryToken: string): boolean {
if (!text) return false;
const textLower = text.toLowerCase();
const queryLower = queryToken.toLowerCase();
const queryStem = this.stemToken(queryLower);
return textLower.includes(queryLower) || textLower.includes(queryStem);
}
const frontmatterMatch = content.match(/^---(.*?)---/s);
private parseMarkdown(content: string) {
const frontmatterRegex = /^---\n([\s\S]*?)\n---\n/;
const frontmatterMatch = content.match(frontmatterRegex);
const frontmatter: ParsedFrontmatter = {};
if (frontmatterMatch) {
try {
const frontmatterContent = frontmatterMatch[1];
const lines = frontmatterContent.trim().split('\n');
const lines = frontmatterMatch[1].trim().split('\n');
for (const line of lines) {
const [key, ...valueParts] = line.split(':');
if (!key) continue;
const value = valueParts.join(':').trim();
if (key.trim() === 'title') {
if (value) {
frontmatter.title = value;
}
} else if (key.trim() === 'tags') {
if (value) {
frontmatter.tags = value;
}
}
if (key.trim() === 'title' && value) frontmatter.title = value;
else if (key.trim() === 'tags' && value) frontmatter.tags = value;
}
} catch {
Logger.warn('Failed to parse frontmatter', 'vault-indexer');
}
}
const headingMatches = content.match(/^#{1,6} (.*?)$/gm);
if (headingMatches) {
headings.push(...headingMatches.map((h: string) => h.replace(/^#{1,6} /, '')));
const titleMatch = content.match(/^# (.+)$/m);
const title = titleMatch ? titleMatch[1] : '';
const headings: string[] = [];
const headingRegex = /^#{1,6} (.+)$/gm;
let headingMatch;
while ((headingMatch = headingRegex.exec(content)) !== null) {
headings.push(headingMatch[1]);
}
const paragraphMatch = content.match(/^([^#]*?)(?=#|\n\n|$)/s);
if (paragraphMatch) {
firstParagraph = paragraphMatch[1].trim();
}
const bodyWithoutFrontmatter = frontmatterMatch
? content.substring(frontmatterMatch[0].length)
: content;
const bodyText = bodyWithoutFrontmatter
.replace(/#{1,6} .+/g, '')
.replace(/^\s*[\r\n]/gm, '')
.trim();
const allText = content
.replace(/^---.*?---/s, '')
.replace(/^#.*?$/gm, '')
.replace(/```.*?```/gs, '')
.replace(/`.*?`/g, '')
.replace(/\[.*?\]\(.*?\)/g, '');
tokens.push(...this.tokenize(allText));
return { tokens, headings, frontmatter, firstParagraph };
}
private calculateWeightedScore(
tokenized: TokenizedContent,
queryTokens: string[],
file?: VaultFile
): ScoreResult {
let totalScore = 0;
const matchedTokens: Set<string> = new Set<string>();
for (const queryToken of queryTokens) {
let tokenScore = 0;
const stemmed = this.stemToken(queryToken);
let matched = false;
if (
tokenized.frontmatter?.title &&
this.exactMatch(tokenized.frontmatter.title, queryToken)
) {
tokenScore += 3;
matched = true;
} else if (
file &&
file.basename &&
this.exactMatch(file.basename.replace(/\.md$/, ''), queryToken)
) {
tokenScore += 3;
matched = true;
}
if (tokenized.frontmatter?.tags && this.exactMatch(tokenized.frontmatter.tags, queryToken)) {
tokenScore += 2.5;
matched = true;
}
if (tokenized.headings.some((heading: string) => heading.toLowerCase().includes(stemmed))) {
tokenScore += 5;
matched = true;
}
if (tokenized.firstParagraph && tokenized.firstParagraph.toLowerCase().includes(stemmed)) {
tokenScore += 1.5;
matched = true;
}
if (tokenized.tokens.includes(stemmed)) {
tokenScore += 1;
matched = true;
}
if (matched) {
totalScore += tokenScore;
matchedTokens.add(queryToken);
}
}
return {
score: totalScore,
matchedFields: Array.from(matchedTokens),
};
}
private stemToken(token: string): string {
// Improved stemmer that handles edge cases
if (token.length <= 3) return token; // Don't stem very short tokens
if (token.endsWith('s')) return token.slice(0, -1);
if (token.endsWith('ed') && token.length > 4) return token.slice(0, -2); // Don't stem 3-letter words ending in ed
if (token.endsWith('ing') && token.length > 5) return token.slice(0, -3); // Don't stem 4-letter words ending in ing
return token;
}
private exactMatch(content: string, token: string): boolean {
const stemmedToken = this.stemToken(token);
return content.toLowerCase().includes(stemmedToken);
return { frontmatter, title, headings, content: bodyText };
}
}
export { VaultIndexer, Cache, InMemoryCache };
// Convenience method to create a VaultIndexer with an in-memory cache
export function createVaultIndexerWithCache(vault: VaultLike): VaultIndexer {
return new VaultIndexer(vault, new InMemoryCache());
}
+3
View File
@@ -0,0 +1,3 @@
// src/workflow-engine/index.ts
export { WorkflowEngine } from './workflow-engine';
+817
View File
@@ -0,0 +1,817 @@
// src/workflow-engine/workflow-engine.ts
import { Vault, App } from 'obsidian';
import {
WorkflowStep,
WorkflowStepType,
WorkflowStepResult,
WorkflowDefinition,
WorkflowExecutionContext,
WorkflowExecutionResult,
LlmStepConfig,
VaultSearchStepConfig,
ToolStepConfig,
FormatStepConfig,
OllamaMessage,
OllamaTool,
} from '../types';
import { VaultIndexer } from '../vault-indexer';
import { ToolExecutor } from '../tool-executor';
import { OllamaClient } from '../ollama-client';
import { ConversationStateManager } from '../conversation-state';
import { Logger } from '../utils';
import { safeParseJson } from '../utils';
/**
* Pattern to match {{variable}} or {{variable.property}} syntax in strings.
*/
const VARIABLE_PATTERN = /\{\{([\w.]+)\}\}/g;
/**
* WorkflowEngine orchestrates multi-step/chained reasoning workflows.
*
* Each workflow is a sequence of steps that can:
* - Call the LLM with prompts
* - Search the vault for relevant notes
* - Execute tools (file operations, vault queries)
* - Format/transform data
*
* Steps can reference outputs from previous steps via {{stepId.output}} syntax.
*/
export class WorkflowEngine {
private vaultIndexer: VaultIndexer;
private toolExecutor: ToolExecutor;
private ollamaClient: OllamaClient;
private conversationStateManager: ConversationStateManager;
private maxSteps: number;
private maxWorkflowDuration: number;
constructor(
vault: Vault,
app: App,
ollamaUrl: string,
model: string,
options?: {
maxSteps?: number;
maxWorkflowDuration?: number;
cacheConfig?: import('../types').CacheConfig;
}
) {
this.vaultIndexer = new VaultIndexer(vault);
this.toolExecutor = new ToolExecutor(vault, app);
this.ollamaClient = new OllamaClient(ollamaUrl, model, undefined, options?.cacheConfig);
this.conversationStateManager = new ConversationStateManager();
this.maxSteps = options?.maxSteps ?? 20;
this.maxWorkflowDuration = options?.maxWorkflowDuration ?? 300_000; // 5 minutes
}
/**
* Execute a workflow definition.
* @param definition The workflow to execute
* @param initialVariables Optional initial variables to seed the context
* @returns The execution result with all step outputs
*/
async executeWorkflow(
definition: WorkflowDefinition,
initialVariables?: Record<string, unknown>
): Promise<WorkflowExecutionResult> {
Logger.info(`Starting workflow: ${definition.name} (${definition.id})`, 'workflow-engine');
const startTime = Date.now();
const context = this.createExecutionContext(initialVariables);
// Validate the workflow before execution
const validationError = this.validateWorkflow(definition);
if (validationError) {
Logger.error(`Workflow validation failed: ${validationError}`, 'workflow-engine');
return {
workflowId: definition.id,
workflowName: definition.name,
success: false,
stepResults: [],
finalOutput: null,
error: validationError,
};
}
try {
// Execute steps in topological order
const orderedSteps = this.topologicalSort(definition.steps);
let stepCount = 0;
for (const step of orderedSteps) {
// Check timeout
const elapsed = Date.now() - startTime;
if (elapsed > this.maxWorkflowDuration) {
throw new Error(`Workflow exceeded maximum duration of ${this.maxWorkflowDuration}ms`);
}
// Check max steps
stepCount++;
if (stepCount > this.maxSteps) {
throw new Error(`Workflow exceeded maximum step count of ${this.maxSteps}`);
}
// Wait for dependency if exists
if (step.dependsOn) {
const depResult = context.stepResults.find((r) => r.stepId === step.dependsOn);
if (!depResult) {
throw new Error(`Step ${step.id}: dependency '${step.dependsOn}' not found`);
}
if (!depResult.success) {
Logger.warn(
`Step ${step.id}: dependency '${step.dependsOn}' failed, skipping`,
'workflow-engine'
);
// Record a failure for this step due to failed dependency
context.stepResults.push({
stepId: step.id,
stepName: step.name,
success: false,
data: null,
error: `Dependency '${step.dependsOn}' failed: ${depResult.error}`,
timestamp: Date.now(),
});
continue;
}
}
// Execute the step
Logger.debug(`Executing step: ${step.name} (${step.id})`, 'workflow-engine');
const result = await this.executeStep(step, context);
context.stepResults.push(result);
// Store the result data in variables for interpolation
context.variables.set(step.id, result.data);
// Update conversation history for LLM steps
if (step.type === 'llm' && result.success) {
this.conversationStateManager.updateShortTermContext({
role: 'assistant',
content: String(result.data),
});
}
}
// Determine final output (last successful step's data)
const lastSuccessfulResult = [...context.stepResults].reverse().find((r) => r.success);
const finalOutput = lastSuccessfulResult?.data ?? null;
const success = context.stepResults.every((r) => r.success);
Logger.info(
`Workflow completed: ${definition.name} (${success ? 'success' : 'partial'})`,
'workflow-engine'
);
return {
workflowId: definition.id,
workflowName: definition.name,
success,
stepResults: context.stepResults,
finalOutput,
};
} catch (error) {
const errorMessage = error instanceof Error ? error.message : String(error);
Logger.error(`Workflow failed: ${errorMessage}`, 'workflow-engine');
return {
workflowId: definition.id,
workflowName: definition.name,
success: false,
stepResults: context.stepResults,
finalOutput: null,
error: errorMessage,
};
}
}
/**
* Execute a workflow from a natural language description.
* The LLM will generate the workflow steps, then we execute them.
*
* @param userQuery The user's natural language request
* @param availableTools Optional list of available tools to inform the LLM
* @returns The execution result
*/
async executeWorkflowFromQuery(
userQuery: string,
availableTools?: OllamaTool[]
): Promise<WorkflowExecutionResult> {
Logger.info(`Generating workflow from query: ${userQuery}`, 'workflow-engine');
// First, ask the LLM to generate a workflow plan
const workflow = await this.generateWorkflowFromQuery(userQuery, availableTools);
if (!workflow) {
return {
workflowId: 'auto-generated',
workflowName: 'Auto-generated workflow',
success: false,
stepResults: [],
finalOutput: null,
error: 'Failed to generate workflow from query',
};
}
// Add a system variable with the original query
const initialVariables: Record<string, unknown> = {
original_query: userQuery,
};
return this.executeWorkflow(workflow, initialVariables);
}
/**
* Generate a workflow definition from a natural language query using the LLM.
*/
private async generateWorkflowFromQuery(
query: string,
availableTools?: OllamaTool[]
): Promise<WorkflowDefinition | null> {
const toolDescriptions =
availableTools?.map((t) => `- ${t.function.name}: ${t.function.description}`).join('\n') ??
'';
const systemPrompt = `You are a workflow planner. Given a user query, break it down into a sequence of workflow steps.
Available step types:
- vault_search: Search the vault for notes. Config: { type: 'vault_search', query: string, limit?: number, tagFilter?: string }
- llm: Call an LLM. Config: { type: 'llm', systemPrompt?: string, userPrompt: string }
- tool: Execute a tool. Config: { type: 'tool', toolName: string, args: object }
- format: Format output. Config: { type: 'format', template: string, outputFormat?: 'markdown' | 'text' | 'json' }
Available tools:
${toolDescriptions}
Variable interpolation syntax:
- Use {{stepId.output}} to reference a previous step's output
- Use {{stepId.output.property}} to reference a property of a step's output
- Use {{original_query}} to reference the original user query
Return a JSON object with this structure:
{
"id": "workflow-uuid",
"name": "workflow name",
"description": "description",
"steps": [
{
"id": "step_1",
"type": "vault_search" | "llm" | "tool" | "format",
"name": "step name",
"description": "optional description",
"config": { /* step-specific config */ },
"dependsOn": "optional_step_id"
}
]
}
Rules:
1. Number steps sequentially (step_1, step_2, etc.)
2. Use dependsOn to specify ordering when needed
3. Use variable interpolation to pass data between steps
4. Keep the workflow minimal but effective
5. End with a format step if the user wants structured output`;
const userPrompt = `User query: ${query}`;
try {
const messages: OllamaMessage[] = [
{ role: 'system', content: systemPrompt },
{ role: 'user', content: userPrompt },
];
const response = await this.ollamaClient.chat(messages, []);
const content = response.content?.trim();
if (!content) {
Logger.error('Empty response from LLM when generating workflow', 'workflow-engine');
return null;
}
// Extract JSON from the response (handle markdown code blocks)
const jsonMatch =
content.match(/\```(?:json)?\s*([\s\S]*?)\```/) ?? content.match(/\{[\s\S]*\}/);
const jsonString = jsonMatch ? jsonMatch[1] : content;
const parsed = safeParseJson(jsonString) as unknown;
if (!parsed || typeof parsed !== 'object') {
Logger.error('Invalid workflow JSON from LLM', 'workflow-engine');
return null;
}
return parsed as WorkflowDefinition;
} catch (error) {
const errorMessage = error instanceof Error ? error.message : String(error);
Logger.error(`Failed to generate workflow: ${errorMessage}`, 'workflow-engine');
return null;
}
}
/**
* Create a fresh execution context.
*/
private createExecutionContext(
initialVariables?: Record<string, unknown>
): WorkflowExecutionContext {
const variables = new Map<string, unknown>();
if (initialVariables) {
for (const [key, value] of Object.entries(initialVariables)) {
variables.set(key, value);
}
}
return {
variables,
stepResults: [],
conversationHistory: [],
};
}
/**
* Execute a single workflow step.
*/
private async executeStep(
step: WorkflowStep,
context: WorkflowExecutionContext
): Promise<WorkflowStepResult> {
const timestamp = Date.now();
try {
// Interpolate variables in the step config
const interpolatedConfig = this.interpolateVariables(step.config, context.variables);
let data: unknown;
switch (step.type) {
case 'llm':
data = await this.executeLlmStep(interpolatedConfig as LlmStepConfig, context);
break;
case 'vault_search':
data = await this.executeVaultSearchStep(interpolatedConfig as VaultSearchStepConfig);
break;
case 'tool':
data = await this.executeToolStep(interpolatedConfig as ToolStepConfig);
break;
case 'format':
data = this.executeFormatStep(interpolatedConfig as FormatStepConfig, context);
break;
default:
throw new Error(`Unknown step type: ${step.type}`);
}
return {
stepId: step.id,
stepName: step.name,
success: true,
data,
timestamp,
};
} catch (error) {
const errorMessage = error instanceof Error ? error.message : String(error);
Logger.error(`Step ${step.id} failed: ${errorMessage}`, 'workflow-engine');
return {
stepId: step.id,
stepName: step.name,
success: false,
data: null,
error: errorMessage,
timestamp,
};
}
}
/**
* Execute an LLM step.
*/
private async executeLlmStep(
config: LlmStepConfig,
context: WorkflowExecutionContext
): Promise<string> {
const messages: OllamaMessage[] = [];
// Add system prompt if provided
if (config.systemPrompt) {
messages.push({
role: 'system',
content: config.systemPrompt,
});
}
// Add conversation history
messages.push(...context.conversationHistory);
// Add user prompt
messages.push({
role: 'user',
content: config.userPrompt,
});
// Determine if we should include tools
const tools = config.includeToolCalls ? [] : [];
const response = await this.ollamaClient.chat(messages, tools);
return response.content ?? '';
}
/**
* Execute a vault search step.
*/
private async executeVaultSearchStep(config: VaultSearchStepConfig): Promise<unknown> {
const limit = config.limit ?? 5;
const entries = await this.vaultIndexer.searchVault(config.query, limit);
// Apply tag filter if specified
const filtered = config.tagFilter
? entries.filter((entry) => {
const tags = entry.frontmatter?.tags ?? '';
return tags.toLowerCase().includes(config.tagFilter!.toLowerCase());
})
: entries;
return filtered.map((entry) => ({
path: entry.file.path,
title: entry.title,
content: entry.content,
score: entry.score,
tags: entry.frontmatter?.tags,
}));
}
/**
* Execute a tool step.
*/
private async executeToolStep(config: ToolStepConfig): Promise<unknown> {
const args: Record<string, unknown> = config.args ?? {};
const result = await this.toolExecutor.executeTool(config.toolName, args);
return {
success: result.success,
message: result.message,
data: result.data,
};
}
/**
* Execute a format step (template rendering).
*/
private executeFormatStep(config: FormatStepConfig, context: WorkflowExecutionContext): string {
let output = config.template;
output = String(this.interpolateVariables(output, context.variables));
// Apply output formatting
if (config.outputFormat === 'json') {
try {
const parsed = safeParseJson(output);
return JSON.stringify(parsed, null, 2);
} catch {
// Return as-is if not valid JSON
return output;
}
}
return output;
}
/**
* Interpolate {{variables}} in a string or object.
* Supports:
* - {{variableName}} -> value from context
* - {{stepId.output}} -> data from a step result
* - {{stepId.output.property}} -> nested property access
*/
private interpolateVariables(input: unknown, variables: Map<string, unknown>): unknown {
if (typeof input === 'string') {
return this.interpolateString(input, variables);
}
if (Array.isArray(input)) {
return input.map((item) => this.interpolateVariables(item, variables));
}
if (input !== null && typeof input === 'object') {
const result: Record<string, unknown> = {};
for (const [key, value] of Object.entries(input as Record<string, unknown>)) {
result[key] = this.interpolateVariables(value, variables);
}
return result;
}
return input;
}
/**
* Interpolate variables in a string.
*/
private interpolateString(input: string, variables: Map<string, unknown>): string {
return input.replace(VARIABLE_PATTERN, (_match, variablePath) => {
const value = this.resolveVariable(variablePath, variables);
if (value === undefined) {
// Keep the original placeholder if variable not found
Logger.warn(`Variable '${variablePath}' not found during interpolation`, 'workflow-engine');
return _match;
}
// Handle different value types
if (typeof value === 'object' && value !== null) {
return JSON.stringify(value);
}
return String(value);
});
}
/**
* Resolve a variable path like "step_1.output" or "step_1.output.property".
*/
private resolveVariable(path: string, variables: Map<string, unknown>): unknown {
const parts = path.split('.');
// Check if this is a step output reference (stepId.output or stepId.output.property)
if (parts.length >= 2 && parts[1] === 'output') {
const stepId = parts[0];
const stepData = variables.get(stepId);
if (parts.length === 2) {
return stepData;
}
// Navigate into nested properties
let current = stepData;
for (let i = 2; i < parts.length; i++) {
if (current === null || current === undefined || typeof current !== 'object') {
return undefined;
}
current = (current as Record<string, unknown>)[parts[i]];
}
return current;
}
// Direct variable lookup
return variables.get(path);
}
/**
* Perform topological sort on steps to determine execution order.
* This respects the dependsOn field and ensures steps run in the correct order.
*/
private topologicalSort(steps: WorkflowStep[]): WorkflowStep[] {
const stepMap = new Map<string, WorkflowStep>();
for (const step of steps) {
stepMap.set(step.id, step);
}
const result: WorkflowStep[] = [];
const visited = new Set<string>();
const visiting = new Set<string>();
const visit = (stepId: string): void => {
if (visited.has(stepId)) return;
if (visiting.has(stepId)) {
throw new Error(`Circular dependency detected involving step '${stepId}'`);
}
const step = stepMap.get(stepId);
if (!step) {
throw new Error(`Step '${stepId}' not found`);
}
visiting.add(stepId);
// Visit dependencies first
if (step.dependsOn) {
visit(step.dependsOn);
}
visiting.delete(stepId);
visited.add(stepId);
result.push(step);
};
// Visit all steps
for (const step of steps) {
visit(step.id);
}
return result;
}
/**
* Validate a workflow definition before execution.
* Returns null if valid, or an error message string if invalid.
*/
private validateWorkflow(definition: WorkflowDefinition): string | null {
if (!definition.id) {
return 'Workflow must have an id';
}
if (!definition.name) {
return 'Workflow must have a name';
}
if (!definition.steps || !Array.isArray(definition.steps)) {
return 'Workflow must have a steps array';
}
if (definition.steps.length === 0) {
return 'Workflow must have at least one step';
}
const stepIds = new Set<string>();
for (const step of definition.steps) {
if (!step.id) {
return 'Each step must have an id';
}
if (!step.type) {
return `Step '${step.id}' must have a type`;
}
if (!step.name) {
return `Step '${step.id}' must have a name`;
}
if (!step.config) {
return `Step '${step.id}' must have a config`;
}
// Check for duplicate step ids
if (stepIds.has(step.id)) {
return `Duplicate step id: '${step.id}'`;
}
stepIds.add(step.id);
// Validate step type
const validTypes: WorkflowStepType[] = ['llm', 'vault_search', 'tool', 'format'];
if (!validTypes.includes(step.type)) {
return `Step '${step.id}' has invalid type: '${step.type}'`;
}
// Validate dependsOn references a valid step
if (step.dependsOn && !definition.steps.some((s) => s.id === step.dependsOn)) {
return `Step '${step.id}' depends on unknown step: '${step.dependsOn}'`;
}
// Validate step-specific config
const configError = this.validateStepConfig(step);
if (configError) {
return configError;
}
}
return null;
}
/**
* Validate a step's configuration based on its type.
*/
private validateStepConfig(step: WorkflowStep): string | null {
switch (step.type) {
case 'llm': {
const config = step.config as LlmStepConfig;
if (!config.userPrompt) {
return `LLM step '${step.id}' requires a userPrompt`;
}
break;
}
case 'vault_search': {
const config = step.config as VaultSearchStepConfig;
if (!config.query) {
return `Vault search step '${step.id}' requires a query`;
}
break;
}
case 'tool': {
const config = step.config as ToolStepConfig;
if (!config.toolName) {
return `Tool step '${step.id}' requires a toolName`;
}
break;
}
case 'format': {
const config = step.config as FormatStepConfig;
if (!config.template) {
return `Format step '${step.id}' requires a template`;
}
break;
}
}
return null;
}
/**
* Get all built-in workflow definitions (presets).
*/
static getBuiltInWorkflows(): WorkflowDefinition[] {
return [
WorkflowEngine.createMeetingSummaryWorkflow(),
WorkflowEngine.createNoteAnalyzerWorkflow(),
];
}
/**
* Create a workflow that summarizes meeting notes from the last week.
*/
private static createMeetingSummaryWorkflow(): WorkflowDefinition {
return {
id: 'meeting-summary',
name: 'Meeting Notes Summary',
description:
'Analyzes meeting notes from the last week, extracts decisions, and creates a summary.',
steps: [
{
id: 'step_1',
type: 'vault_search',
name: 'Find Meeting Notes',
description: 'Search for notes tagged with #meeting',
config: {
type: 'vault_search',
query: 'meeting',
limit: 10,
tagFilter: 'meeting',
},
},
{
id: 'step_2',
type: 'llm',
name: 'Extract Decisions',
description: 'Extract key decisions and assigned owners from meeting notes',
config: {
type: 'llm',
systemPrompt:
'You are a meeting analyst. Extract key decisions, action items, and assigned owners from meeting notes.',
userPrompt: `Here are meeting notes from recent meetings:\n\n{{step_1.output}}\n\nPlease extract:\n1. Key decisions made\n2. Action items with assigned owners\n3. Deadlines if mentioned\n\nFormat as a structured list.`,
},
dependsOn: 'step_1',
},
{
id: 'step_3',
type: 'format',
name: 'Format Summary',
description: 'Format the results into a markdown table',
config: {
type: 'format',
template:
'# Meeting Summary\n\n## Decisions and Action Items\n\n{{step_2.output}}\n\n---\n*Generated by Workflow Engine*',
outputFormat: 'markdown',
},
dependsOn: 'step_2',
},
],
};
}
/**
* Create a workflow that analyzes notes and generates insights.
*/
private static createNoteAnalyzerWorkflow(): WorkflowDefinition {
return {
id: 'note-analyzer',
name: 'Note Analyzer',
description: 'Analyzes notes and generates insights, summaries, and suggestions.',
steps: [
{
id: 'step_1',
type: 'vault_search',
name: 'Search Notes',
description: 'Search for relevant notes based on query',
config: {
type: 'vault_search',
query: '{{original_query}}',
limit: 5,
},
},
{
id: 'step_2',
type: 'llm',
name: 'Analyze Content',
description: 'Analyze the found notes for insights',
config: {
type: 'llm',
systemPrompt:
'You are an analytical assistant. Analyze the provided notes and identify key themes, insights, and connections.',
userPrompt: `Original query: {{original_query}}\n\nFound notes:\n{{step_1.output}}\n\nPlease provide:\n1. Key themes identified\n2. Important insights\n3. Potential connections between notes\n4. Suggestions for further exploration`,
},
dependsOn: 'step_1',
},
{
id: 'step_3',
type: 'format',
name: 'Format Results',
description: 'Format the analysis into a readable report',
config: {
type: 'format',
template:
'# Analysis Report\n\n## Query: {{original_query}}\n\n## Findings\n\n{{step_2.output}}\n\n---\n*Generated by Workflow Engine*',
outputFormat: 'markdown',
},
dependsOn: 'step_2',
},
],
};
}
}
+7
View File
@@ -32,6 +32,13 @@ const mockSettings: PluginSettings = {
vaultSearchLimit: 3,
maxMessageHistory: 50,
lastIndexTime: 0,
cacheConfig: {
enabled: false,
similarityThreshold: 0.9,
collectionName: 'test-cache',
embeddingModel: 'nomic-embed-text',
chromaURL: 'http://localhost:8000',
},
};
describe('ChatView', () => {
+214
View File
@@ -0,0 +1,214 @@
import { ConversationStateManager } from '../src/conversation-state';
import type { OllamaMessage } from '../src/types';
describe('ConversationStateManager', () => {
let manager: ConversationStateManager;
beforeEach(() => {
manager = new ConversationStateManager();
});
describe('constructor', () => {
it('should initialize with default system message in long-term context', () => {
const longTerm = manager.getLongTermContext();
expect(longTerm).toHaveLength(1);
expect(longTerm[0].role).toBe('system');
expect(longTerm[0].content).toContain(
'You are an assistant that can help answer questions using the contents of a vault'
);
});
it('should initialize with empty short-term and medium-term contexts', () => {
expect(manager.getShortTermContext()).toEqual([]);
expect(manager.getMediumTermContext()).toEqual([]);
});
});
describe('updateShortTermContext', () => {
it('should add messages to short-term context', () => {
const message: OllamaMessage = { role: 'user', content: 'Hello' };
manager.updateShortTermContext(message);
expect(manager.getShortTermContext()).toContainEqual(message);
});
it('should enforce maxShortTermTurns limit of 10', () => {
for (let i = 0; i < 15; i++) {
manager.updateShortTermContext({ role: 'user', content: `msg-${i}` });
}
const context = manager.getShortTermContext();
expect(context).toHaveLength(10);
expect(context[0].content).toBe('msg-5');
expect(context[9].content).toBe('msg-14');
});
});
describe('updateMediumTermContext', () => {
it('should add messages to medium-term context', () => {
const message: OllamaMessage = { role: 'system', content: 'KB result' };
manager.updateMediumTermContext(message);
expect(manager.getMediumTermContext()).toContainEqual(message);
});
it('should enforce maxMediumTermMessages limit of 20', () => {
for (let i = 0; i < 25; i++) {
manager.updateMediumTermContext({ role: 'system', content: `msg-${i}` });
}
const context = manager.getMediumTermContext();
expect(context).toHaveLength(20);
expect(context[0].content).toBe('msg-5');
expect(context[19].content).toBe('msg-24');
});
});
describe('setPersona', () => {
it('should replace default system message with custom persona', () => {
manager.setPersona('You are a coding expert.');
const longTerm = manager.getLongTermContext();
expect(longTerm.some((msg) => msg.content === 'You are a coding expert.')).toBe(true);
expect(
longTerm.some((msg) =>
msg.content.includes(
'You are an assistant that can help answer questions using the contents of a vault'
)
)
).toBe(false);
});
it('should allow multiple persona updates', () => {
manager.setPersona('First persona.');
manager.setPersona('Second persona.');
const longTerm = manager.getLongTermContext();
expect(longTerm.some((msg) => msg.content === 'Second persona.')).toBe(true);
// The actual implementation filters out the default system message but keeps previous persona messages
// So we should expect to find both personas in the long-term context
expect(longTerm.some((msg) => msg.content === 'First persona.')).toBe(true);
expect(longTerm).toHaveLength(2);
});
});
describe('getConversationContext', () => {
it('should return all three context layers', () => {
manager.updateShortTermContext({ role: 'user', content: 'Hi' });
manager.updateMediumTermContext({ role: 'system', content: 'KB' });
const context = manager.getConversationContext('test');
expect(context.shortTermContext).toHaveLength(1);
expect(context.mediumTermContext).toHaveLength(1);
expect(context.longTermContext).toHaveLength(1);
});
});
describe('getCompleteMessages', () => {
it('should return messages in correct order: long, medium, short, current', () => {
manager.updateShortTermContext({ role: 'user', content: 'Short' });
manager.updateShortTermContext({ role: 'assistant', content: 'Short reply' });
manager.updateMediumTermContext({ role: 'system', content: 'Medium' });
const messages = manager.getCompleteMessages('Current');
// Long-term comes first
expect(messages[0].role).toBe('system');
expect(messages[0].content).toContain('You are an assistant');
// Medium-term follows
expect(messages[1].content).toBe('Medium');
// Short-term follows
expect(messages[2].content).toBe('Short');
expect(messages[3].content).toBe('Short reply');
// Current user message last
expect(messages[messages.length - 1].content).toBe('Current');
expect(messages[messages.length - 1].role).toBe('user');
});
});
describe('clear', () => {
it('should reset short-term and medium-term contexts', () => {
manager.updateShortTermContext({ role: 'user', content: 'msg' });
manager.updateMediumTermContext({ role: 'system', content: 'msg' });
manager.clear();
expect(manager.getShortTermContext()).toEqual([]);
expect(manager.getMediumTermContext()).toEqual([]);
});
it('should restore default system message in long-term context', () => {
manager.setPersona('Custom persona');
manager.clear();
const longTerm = manager.getLongTermContext();
expect(longTerm[0].content).toContain(
'You are an assistant that can help answer questions using the contents of a vault'
);
});
});
describe('setMediumTermContextFromQuery', () => {
it('should clear previous medium-term context and add query result', () => {
manager.updateMediumTermContext({ role: 'system', content: 'Old' });
manager.setMediumTermContextFromQuery('New KB result');
const medium = manager.getMediumTermContext();
expect(medium).toHaveLength(1);
expect(medium[0].content).toContain('New KB result');
expect(medium[0].content).toContain('Knowledge base results for current query');
});
it('should ignore whitespace-only query results', () => {
manager.setMediumTermContextFromQuery(' ');
expect(manager.getMediumTermContext()).toEqual([]);
});
it('should ignore empty query results', () => {
manager.setMediumTermContextFromQuery('');
expect(manager.getMediumTermContext()).toEqual([]);
});
});
describe('immutability', () => {
it('should return copies from getters to prevent external mutation', () => {
manager.updateShortTermContext({ role: 'user', content: 'test' });
const shortCopy = manager.getShortTermContext();
shortCopy.push({ role: 'assistant', content: 'injected' });
expect(manager.getShortTermContext()).not.toContainEqual({
role: 'assistant',
content: 'injected',
});
});
it('should return independent copies on repeated calls', () => {
const copy1 = manager.getShortTermContext();
const copy2 = manager.getShortTermContext();
expect(copy1).not.toBe(copy2);
});
});
describe('edge cases', () => {
it('should handle empty user message in getCompleteMessages', () => {
const messages = manager.getCompleteMessages('');
expect(messages[messages.length - 1].content).toBe('');
expect(messages[messages.length - 1].role).toBe('user');
});
it('should handle messages with tool_calls', () => {
const message: OllamaMessage = {
role: 'assistant',
content: '',
tool_calls: [
{
id: 'call_1',
type: 'function',
function: { name: 'test', arguments: '{}' },
},
],
};
manager.updateShortTermContext(message);
expect(manager.getShortTermContext()).toContainEqual(message);
});
it('should preserve long-term context across clear and restore', () => {
manager.clear();
const longTerm = manager.getLongTermContext();
expect(longTerm).toHaveLength(1);
expect(longTerm[0].role).toBe('system');
});
});
});
+381
View File
@@ -0,0 +1,381 @@
import {
extractConcepts,
findRelationships,
buildDependencyGraph,
toDotFormat,
toJsonFormat,
toCytoscapeFormat,
generateGraphVisualization,
} from '../src/graph-view';
import type { VaultIndexEntry } from '../src/types';
import type { DependencyGraph } from '../src/graph-view';
describe('Graph Utilities', () => {
const mockFiles: VaultIndexEntry[] = [
{
path: 'algorithms.md',
title: 'Algorithms',
content:
'# Sorting Algorithms\n\n## Bubble Sort\nThe **bubble sort** is a simple sorting algorithm.\n\n## Quick Sort\n*Quick sort* is more efficient.',
score: 1,
},
{
path: 'data-structures.md',
title: 'Data Structures',
content:
'# Data Structures\n\n## Trees\nBinary trees are fundamental.\n\n## Graphs\nGraph algorithms build on *Sorting Algorithms*.',
score: 2,
},
{
path: 'patterns.md',
title: 'Design Patterns',
content:
'# Design Patterns\n\n## Factory Pattern\nCreates objects.\n\n## Observer Pattern\n**bubble sort** mentions the Factory Pattern.',
score: 3,
},
];
describe('extractConcepts', () => {
it('should extract headings as concepts', () => {
const content = '# Main Heading\n## Sub Heading\n### Deep Heading';
const concepts = extractConcepts(content, 'test.md');
expect(concepts).toContain('Main Heading');
expect(concepts).toContain('Sub Heading');
expect(concepts).toContain('Deep Heading');
});
it('should extract bold text as concepts', () => {
const content = 'This is **important concept** and **another concept**.';
const concepts = extractConcepts(content, 'test.md');
expect(concepts).toContain('important concept');
expect(concepts).toContain('another concept');
});
it('should not duplicate concepts', () => {
const content = '# Heading\n\n# Heading\n\n**Heading**';
const concepts = extractConcepts(content, 'test.md');
expect(concepts).toEqual(['Heading']);
});
it('should return empty array for content without concepts', () => {
const content = 'Plain text without any special formatting.';
const concepts = extractConcepts(content, 'test.md');
expect(concepts).toEqual([]);
});
it('should handle empty content', () => {
const concepts = extractConcepts('', 'test.md');
expect(concepts).toEqual([]);
});
it('should extract all heading levels', () => {
const content = '# H1\n## H2\n### H3\n#### H4\n##### H5\n###### H6';
const concepts = extractConcepts(content, 'test.md');
expect(concepts).toEqual(['H1', 'H2', 'H3', 'H4', 'H5', 'H6']);
});
});
describe('findRelationships', () => {
it('should find relationships between files sharing concepts', () => {
const conceptIndex: Record<string, string[]> = {
'Sorting Algorithms': ['algorithms.md'],
'bubble sort': ['algorithms.md', 'patterns.md'],
};
const relationships = findRelationships(mockFiles, conceptIndex);
// Algorithms.md references patterns.md (via "bubble sort")
// Data-structures.md references algorithms.md (via "Sorting Algorithms")
// Patterns.md references algorithms.md (via "bubble sort")
expect(relationships).toHaveLength(3);
expect(
relationships.some((r) => r.source === 'patterns.md' && r.target === 'algorithms.md')
).toBe(true);
expect(
relationships.some((r) => r.source === 'data-structures.md' && r.target === 'algorithms.md')
).toBe(true);
expect(
relationships.some((r) => r.source === 'algorithms.md' && r.target === 'patterns.md')
).toBe(true);
});
it('should exclude self-references', () => {
const conceptIndex: Record<string, string[]> = {
'bubble sort': ['algorithms.md'],
};
const relationships = findRelationships(mockFiles, conceptIndex);
const selfRefs = relationships.filter((r) => r.source === r.target);
expect(selfRefs).toHaveLength(0);
});
it('should return empty array for empty files', () => {
const relationships = findRelationships([], {});
expect(relationships).toEqual([]);
});
it('should return empty array when no concepts overlap', () => {
const conceptIndex: Record<string, string[]> = {
'Unique Concept A': ['file1.md'],
'Unique Concept B': ['file2.md'],
};
const isolatedFiles: VaultIndexEntry[] = [
{ path: 'file1.md', title: 'File 1', content: '# Unique Concept A', score: 1 },
{ path: 'file2.md', title: 'File 2', content: '# Unique Concept B', score: 2 },
];
const relationships = findRelationships(isolatedFiles, conceptIndex);
expect(relationships).toEqual([]);
});
});
describe('buildDependencyGraph', () => {
it('should create nodes for each file', () => {
const graph = buildDependencyGraph(mockFiles);
expect(graph.nodes).toHaveLength(3);
expect(graph.nodes.map((n) => n.id)).toEqual([
'algorithms.md',
'data-structures.md',
'patterns.md',
]);
});
it('should set node type to file', () => {
const graph = buildDependencyGraph(mockFiles);
graph.nodes.forEach((node) => {
expect(node.type).toBe('file');
});
});
it('should include file properties', () => {
const graph = buildDependencyGraph(mockFiles);
const node = graph.nodes[0];
expect(node.properties).toEqual({
path: 'algorithms.md',
title: 'Algorithms',
contentPreview: expect.stringContaining('Sorting Algorithms'),
});
});
it('should create edges for concept relationships', () => {
const filesWithRelationship: VaultIndexEntry[] = [
{
path: 'file1.md',
title: 'File 1',
content: '# Shared Concept\n\nSome content about **Shared Concept**.',
score: 1,
},
{
path: 'file2.md',
title: 'File 2',
content: '# Shared Concept\n\nMore content.',
score: 2,
},
];
const graph = buildDependencyGraph(filesWithRelationship);
expect(graph.edges).toHaveLength(2); // Each mentions the other's concept
});
it('should handle empty file list', () => {
const graph = buildDependencyGraph([]);
expect(graph.nodes).toEqual([]);
expect(graph.edges).toEqual([]);
});
it('should truncate content preview to 100 chars', () => {
const longFile: VaultIndexEntry = {
path: 'long.md',
title: 'Long',
content: 'A'.repeat(200),
score: 1,
};
const graph = buildDependencyGraph([longFile]);
expect(graph.nodes[0].properties.contentPreview).toHaveLength(101); // 100 + '.'
});
});
describe('toDotFormat', () => {
it('should generate valid DOT syntax', () => {
const graph: DependencyGraph = {
nodes: [
{
id: 'node1',
label: 'Node 1',
file: { path: 'file1.md', title: 'File 1', content: 'Content', score: 1 },
type: 'file',
properties: {},
},
{
id: 'node2',
label: 'Node 2',
file: { path: 'file2.md', title: 'File 2', content: 'Content', score: 2 },
type: 'file',
properties: {},
},
],
edges: [
{
id: 'edge1',
source: 'node1',
target: 'node2',
label: 'references',
relationship: 'references',
properties: {},
},
],
};
const dot = toDotFormat(graph);
expect(dot).toContain('digraph G {');
expect(dot).toContain('"node1" [label="Node 1"]');
expect(dot).toContain('"node2" [label="Node 2"]');
expect(dot).toContain('"node1" -> "node2" [label="references"]');
expect(dot).toContain('}');
});
it('should handle empty graph', () => {
const dot = toDotFormat({ nodes: [], edges: [] });
expect(dot).toContain('digraph G {');
expect(dot).toContain('}');
});
it('should escape quotes in labels', () => {
const graph: DependencyGraph = {
nodes: [
{
id: 'node1',
label: 'Node with "quotes"',
file: { path: 'file1.md', title: 'File 1', content: 'C', score: 1 },
type: 'file',
properties: {},
},
],
edges: [],
};
const dot = toDotFormat(graph);
expect(dot).toContain('label="Node with \\"quotes\\""');
});
});
describe('toJsonFormat', () => {
it('should produce valid JSON', () => {
const graph: DependencyGraph = {
nodes: [
{
id: 'n1',
label: 'Node',
file: { path: 'f.md', title: 'T', content: 'C', score: 1 },
type: 'file',
properties: {},
},
],
edges: [],
};
const json = toJsonFormat(graph);
const parsed = JSON.parse(json);
expect(parsed.nodes).toHaveLength(1);
expect(parsed.edges).toEqual([]);
});
it('should include all node and edge data', () => {
const graph: DependencyGraph = {
nodes: [],
edges: [
{
id: 'e1',
source: 's',
target: 't',
label: 'rel',
relationship: 'rel',
properties: {},
},
],
};
const json = toJsonFormat(graph);
const parsed = JSON.parse(json);
expect(parsed.edges[0].source).toBe('s');
expect(parsed.edges[0].target).toBe('t');
});
});
describe('toCytoscapeFormat', () => {
it('should produce valid Cytoscape JSON structure', () => {
const graph: DependencyGraph = {
nodes: [
{
id: 'n1',
label: 'Node',
file: { path: 'f.md', title: 'T', content: 'C', score: 1 },
type: 'file',
properties: { extra: true },
},
],
edges: [],
};
const json = toCytoscapeFormat(graph);
const parsed = JSON.parse(json);
expect(parsed.elements.nodes).toHaveLength(1);
expect(parsed.elements.nodes[0].data.id).toBe('n1');
expect(parsed.elements.nodes[0].data.type).toBe('file');
expect(parsed.elements.nodes[0].data.extra).toBe(true);
});
it('should include edge data in Cytoscape format', () => {
const graph: DependencyGraph = {
nodes: [],
edges: [
{
id: 'e1',
source: 's',
target: 't',
label: 'rel',
relationship: 'rel',
properties: {},
},
],
};
const json = toCytoscapeFormat(graph);
const parsed = JSON.parse(json);
expect(parsed.elements.edges[0].data.source).toBe('s');
expect(parsed.elements.edges[0].data.target).toBe('t');
expect(parsed.elements.edges[0].data.relationship).toBe('rel');
});
});
describe('generateGraphVisualization', () => {
it('should generate DOT format by default', () => {
const result = generateGraphVisualization(mockFiles);
expect(result).toContain('digraph G {');
});
it('should generate DOT format when specified', () => {
const result = generateGraphVisualization(mockFiles, 'dot');
expect(result).toContain('digraph G {');
});
it('should generate JSON format when specified', () => {
const result = generateGraphVisualization(mockFiles, 'json');
const parsed = JSON.parse(result);
expect(parsed.nodes).toBeDefined();
expect(parsed.edges).toBeDefined();
});
it('should generate Cytoscape format when specified', () => {
const result = generateGraphVisualization(mockFiles, 'cytoscape');
const parsed = JSON.parse(result);
expect(parsed.elements).toBeDefined();
expect(parsed.elements.nodes).toBeDefined();
});
it('should fall back to DOT for unknown formats', () => {
const result = generateGraphVisualization(mockFiles, 'dot' as any);
expect(result).toContain('digraph G {');
});
});
});
+308
View File
@@ -0,0 +1,308 @@
import { ContentExtractor } from '../src/indexing-pipeline/extraction';
import { ContentNormalizer } from '../src/indexing-pipeline/normalization';
import { ContentVectorizer } from '../src/indexing-pipeline/vectorization';
import { IndexingPipeline } from '../src/indexing-pipeline/pipeline';
// Mock VaultFile interface for testing
interface MockVaultFile {
basename: string;
path: string;
}
describe('Indexing Pipeline Components', () => {
describe('ContentExtractor', () => {
let extractor: ContentExtractor;
beforeEach(() => {
extractor = new ContentExtractor();
});
it('should extract frontmatter correctly', () => {
const content = `---
title: Test Title
tags: algorithm, programming
date: 2023-01-01
---
# Heading
Content here`;
const file: MockVaultFile = { basename: 'test', path: 'test.md' };
const extracted = extractor.extractFromFile(file, content);
expect(extracted.frontmatter.title).toBe('Test Title');
expect(extracted.frontmatter.tags).toBe('algorithm, programming');
expect(extracted.frontmatter.date).toBe('2023-01-01');
expect(extracted.headings).toContain('Heading');
});
it('should extract headings correctly', () => {
const content = `# Heading 1
## Heading 2
### Heading 3
Content`;
const file: MockVaultFile = { basename: 'test', path: 'test.md' };
const extracted = extractor.extractFromFile(file, content);
expect(extracted.headings).toEqual(['Heading 1', 'Heading 2', 'Heading 3']);
});
it('should extract embedded code blocks', () => {
const content = `# Code Example
\`\`\`javascript
console.log('hello world');
\`\`\`
Some content`;
const file: MockVaultFile = { basename: 'test', path: 'test.md' };
const extracted = extractor.extractFromFile(file, content);
expect(extracted.embeddedCodeBlocks).toHaveLength(1);
expect(extracted.embeddedCodeBlocks[0]).toContain('console.log');
});
it('should extract first paragraph', () => {
const content = `First paragraph here.
Second paragraph here.
# Heading`;
const file: MockVaultFile = { basename: 'test', path: 'test.md' };
const extracted = extractor.extractFromFile(file, content);
expect(extracted.firstParagraph).toBe('First paragraph here.');
});
it('should extract raw text correctly', () => {
const content = `---
title: Test
---
# Heading
Content with **bold** and [link](url).
\`\`\`javascript
code
\`\`\``;
const rawText = extractor.extractRawText(content);
expect(rawText).not.toContain('---');
expect(rawText).not.toContain('# Heading');
expect(rawText).not.toContain('```javascript');
expect(rawText).toContain('Content with bold and link');
});
});
describe('ContentNormalizer', () => {
let normalizer: ContentNormalizer;
let extractor: ContentExtractor;
beforeEach(() => {
normalizer = new ContentNormalizer();
extractor = new ContentExtractor();
});
it('should normalize frontmatter dates to ISO format', () => {
const content = `---
title: Test
date: 2023-01-01
created: 2023-06-15
updated: invalid-date
tags: algorithm
---
Content`;
const file: MockVaultFile = { basename: 'test', path: 'test.md' };
const extracted = extractor.extractFromFile(file, content);
const normalized = normalizer.normalize(extracted);
expect(normalized.frontmatter.date).toBe('2023-01-01T00:00:00.000Z');
expect(normalized.frontmatter.created).toBe('2023-06-15T00:00:00.000Z');
expect(normalized.frontmatter.updated).toBe('invalid-date'); // Should preserve invalid dates
});
it('should convert tags to array format', () => {
const content = `---
title: Test
tags: algorithm, programming, javascript
---
Content`;
const file: MockVaultFile = { basename: 'test', path: 'test.md' };
const extracted = extractor.extractFromFile(file, content);
const normalized = normalizer.normalize(extracted);
expect(normalized.frontmatter.tags).toEqual(['algorithm', 'programming', 'javascript']);
});
it('should calculate word count correctly', () => {
const content = `# Title
This is a test document with several words to count.
It has multiple sentences and words to make it longer.`;
const file: MockVaultFile = { basename: 'test', path: 'test.md' };
const extracted = extractor.extractFromFile(file, content);
const normalized = normalizer.normalize(extracted);
expect(normalized.wordCount).toBeGreaterThan(0);
});
it('should extract tokens correctly', () => {
const content = `# Test Document
This is a test document with important keywords.`;
const file: MockVaultFile = { basename: 'test', path: 'test.md' };
const extracted = extractor.extractFromFile(file, content);
const normalized = normalizer.normalize(extracted);
expect(normalized.tokens).toContain('test');
expect(normalized.tokens).toContain('document');
expect(normalized.tokens).toContain('important');
expect(normalized.tokens).toContain('keywords');
});
it('should extract title correctly', () => {
const content = `# Test Document
Content`;
const file: MockVaultFile = { basename: 'test.md', path: 'test.md' };
const extracted = extractor.extractFromFile(file, content);
const normalized = normalizer.normalize(extracted);
expect(normalized.title).toBe('test');
});
});
describe('ContentVectorizer', () => {
let vectorizer: ContentVectorizer;
beforeEach(() => {
vectorizer = new ContentVectorizer({
model: 'nomic-embed-text',
ollamaUrl: 'http://localhost:11434',
});
});
it('should create a proper prompt from content chunk', () => {
const mockChunk = {
id: 'test',
path: 'test.md',
title: 'Test',
content: 'Test content',
tokens: ['test', 'content'],
headings: ['Heading'],
frontmatter: { tags: ['test'] },
firstParagraph: 'First paragraph',
wordCount: 2,
chunkIndex: 0,
chunkSize: 100
};
const prompt = (vectorizer as any).createPrompt(mockChunk);
expect(prompt).toContain('Test');
expect(prompt).toContain('First paragraph');
expect(prompt).toContain('Heading');
expect(prompt).toContain('tags');
});
// Note: Actual embedding tests would require mocking fetch or integration testing
it('should handle vectorization errors gracefully', async () => {
// This test would require mocking fetch to simulate error responses
// For now, we're just ensuring the method exists and doesn't crash
const mockChunk = {
id: 'test',
path: 'test.md',
title: 'Test',
content: 'Test content',
tokens: ['test', 'content'],
headings: ['Heading'],
frontmatter: { tags: ['test'] },
firstParagraph: 'First paragraph',
wordCount: 2,
chunkIndex: 0,
chunkSize: 100
};
// Mock fetch to simulate an error
const originalFetch = global.fetch;
(global.fetch as any) = jest.fn().mockRejectedValue(new Error('Network error'));
try {
const result = await vectorizer.vectorize(mockChunk);
expect(result).toEqual([]);
} finally {
global.fetch = originalFetch;
}
});
});
describe('IndexingPipeline', () => {
let pipeline: IndexingPipeline;
beforeEach(() => {
pipeline = new IndexingPipeline({
ollamaUrl: 'http://localhost:11434',
embeddingModel: 'nomic-embed-text',
});
});
it('should process files through the pipeline', async () => {
const file: MockVaultFile = { basename: 'test', path: 'test.md' };
const content = `---
title: Test Document
tags: test, example
---
# Introduction
This is a test document for pipeline processing.`;
const result = await pipeline.processFile(file, content);
expect(result).not.toBeNull();
expect(result?.title).toBe('test');
expect(result?.path).toBe('test.md');
expect(result?.content).toContain('This is a test document for pipeline processing');
});
it('should handle processing errors gracefully', async () => {
const file: MockVaultFile = { basename: 'test', path: 'test.md' };
const result = await pipeline.processFile(file, '');
// Should not crash, but might return null or incomplete result
expect(result).toBeNull(); // Empty content should return null
});
it('should process files in batches', async () => {
const files: MockVaultFile[] = [
{ basename: 'file1', path: 'file1.md' },
{ basename: 'file2', path: 'file2.md' }
];
const fileContents = {
'file1.md': '# File 1\n\nContent 1',
'file2.md': '# File 2\n\nContent 2'
};
const results = await pipeline.processFilesInBatches(files, fileContents, 1);
expect(results).toHaveLength(2);
expect(results[0].title).toBe('file1');
expect(results[1].title).toBe('file2');
});
});
});
+54 -396
View File
@@ -6,6 +6,7 @@ import { OllamaMessage, OllamaTool, CacheConfig } from '../src/types';
const mockInitialize = jest.fn().mockResolvedValue(undefined);
const mockGetCache = jest.fn().mockResolvedValue(null);
const mockSetCache = jest.fn().mockResolvedValue(undefined);
const mockClearCache = jest.fn().mockResolvedValue(undefined);
// Mock the semantic cache service BEFORE importing OllamaClient
jest.mock('../src/semantic-cache', () => ({
@@ -13,429 +14,86 @@ jest.mock('../src/semantic-cache', () => ({
initialize: mockInitialize,
getCache: mockGetCache,
setCache: mockSetCache,
clearCache: mockClearCache,
})),
}));
// Mock fetch globally
jest.spyOn(global, 'fetch').mockImplementation(jest.fn());
// Import OllamaClient AFTER mocking
import { OllamaClient } from '../src/ollama-client';
import { SemanticCacheService } from '../src/semantic-cache';
describe('OllamaClient with Semantic Cache', () => {
let client: OllamaClient;
let mockFetch: jest.Mock;
function createMockReader(data: string) {
const encoder = new TextEncoder();
const encoded = encoder.encode(data);
let called = false;
return {
read: () => {
if (!called) {
called = true;
return Promise.resolve({ done: false, value: encoded });
}
return Promise.resolve({ done: true, value: new Uint8Array(0) });
},
releaseLock: jest.fn(),
};
}
const mockMessages: OllamaMessage[] = [
{ role: 'system', content: 'You are helpful.' },
{ role: 'user', content: 'What is AI?' },
];
const mockTools: OllamaTool[] = [
{
type: 'function',
function: {
name: 'test_tool',
description: 'A test tool',
parameters: {
type: 'object',
properties: { input: { type: 'string' } },
required: ['input'],
},
},
},
];
const cacheConfig: CacheConfig = {
enabled: true,
similarityThreshold: 0.85,
collectionName: 'test_cache',
embeddingModel: 'nomic-embed-text',
};
describe('OllamaClient', () => {
const mockBaseUrl = 'http://localhost:11434';
const mockModel = 'llama3';
beforeEach(() => {
jest.clearAllMocks();
mockFetch = global.fetch as jest.Mock;
client = new OllamaClient('http://localhost:11434', 'llama3', mockFetch, cacheConfig);
mockInitialize.mockClear();
mockGetCache.mockClear();
mockSetCache.mockClear();
mockClearCache.mockClear();
});
afterEach(() => {
jest.clearAllMocks();
});
describe('constructor', () => {
it('should initialize cache service when enabled', () => {
const cacheConfig: CacheConfig = {
enabled: true,
similarityThreshold: 0.85,
collectionName: 'test_cache',
embeddingModel: 'nomic-embed-text',
chromaURL: 'http://localhost:8000'
};
describe('initializeCache', () => {
it('should call initialize on cache service when enabled', async () => {
await client.initializeCache();
const client = new OllamaClient(mockBaseUrl, mockModel, undefined, cacheConfig);
// The SemanticCacheService mock was instantiated in constructor
expect(SemanticCacheService).toHaveBeenCalledWith('http://localhost:11434', cacheConfig);
expect(client).toBeInstanceOf(OllamaClient);
expect(mockInitialize).toHaveBeenCalledTimes(1);
});
it('should not create cache service when disabled', () => {
const disabledConfig: CacheConfig = { ...cacheConfig, enabled: false };
new OllamaClient('http://localhost:11434', 'llama3', mockFetch, disabledConfig);
it('should not initialize cache service when disabled', () => {
const cacheConfig: CacheConfig = {
enabled: false,
similarityThreshold: 0.85,
collectionName: 'test_cache',
embeddingModel: 'nomic-embed-text',
chromaURL: 'http://localhost:8000'
};
// Constructor should not have created a cache service
expect(SemanticCacheService).not.toHaveBeenCalledWith(
'http://localhost:11434',
disabledConfig
);
});
const client = new OllamaClient(mockBaseUrl, mockModel, undefined, cacheConfig);
it('should not create cache service when no config provided', () => {
new OllamaClient('http://localhost:11434', 'llama3', mockFetch);
expect(SemanticCacheService).not.toHaveBeenCalled();
expect(client).toBeInstanceOf(OllamaClient);
expect(mockInitialize).toHaveBeenCalledTimes(0);
});
});
describe('chat (non-streaming) with cache', () => {
it('should return cached response when available', async () => {
// Setup cache hit
const cachedResponse = 'Cached AI definition';
mockGetCache.mockResolvedValueOnce(cachedResponse);
const result = await client.chat(mockMessages);
expect(result.content).toBe(cachedResponse);
expect(mockFetch).not.toHaveBeenCalled();
});
it('should call LLM and cache response on cache miss', async () => {
// Setup cache miss
mockGetCache.mockResolvedValueOnce(null);
const mockResponse = {
ok: true,
json: () =>
Promise.resolve({
message: {
content: 'AI is the simulation of intelligence.',
},
}),
describe('clearCache', () => {
it('should clear the cache when enabled', async () => {
const cacheConfig: CacheConfig = {
enabled: true,
similarityThreshold: 0.85,
collectionName: 'test_cache',
embeddingModel: 'nomic-embed-text',
chromaURL: 'http://localhost:8000'
};
mockFetch.mockResolvedValueOnce(mockResponse);
const result = await client.chat(mockMessages);
const client = new OllamaClient(mockBaseUrl, mockModel, undefined, cacheConfig);
expect(result.content).toBe('AI is the simulation of intelligence.');
expect(mockFetch).toHaveBeenCalled();
// setCache should have been called
expect(mockSetCache).toHaveBeenCalled();
await client.clearCache();
expect(mockClearCache).toHaveBeenCalledTimes(1);
});
it('should bypass cache when tools are present', async () => {
const mockResponse = {
ok: true,
json: () =>
Promise.resolve({
message: {
content: 'Tool response',
},
}),
it('should not clear cache when disabled', async () => {
const cacheConfig: CacheConfig = {
enabled: false,
similarityThreshold: 0.85,
collectionName: 'test_cache',
embeddingModel: 'nomic-embed-text',
chromaURL: 'http://localhost:8000'
};
mockFetch.mockResolvedValueOnce(mockResponse);
const result = await client.chat(mockMessages, mockTools);
const client = new OllamaClient(mockBaseUrl, mockModel, undefined, cacheConfig);
expect(mockFetch).toHaveBeenCalled();
expect(mockGetCache).not.toHaveBeenCalled();
expect(mockSetCache).not.toHaveBeenCalled();
});
await client.clearCache();
it('should not cache failed responses', async () => {
mockGetCache.mockResolvedValueOnce(null);
mockFetch.mockResolvedValueOnce({ ok: false, status: 500 });
await expect(client.chat(mockMessages)).rejects.toThrow();
expect(mockSetCache).not.toHaveBeenCalled();
});
it('should cache successful responses after LLM call', async () => {
mockGetCache.mockResolvedValueOnce(null);
const mockResponse = {
ok: true,
json: () =>
Promise.resolve({
message: {
content: 'Fresh response from LLM',
},
}),
};
mockFetch.mockResolvedValueOnce(mockResponse);
await client.chat(mockMessages);
expect(mockSetCache).toHaveBeenCalledWith('What is AI?', 'Fresh response from LLM');
});
});
describe('streamChat with cache', () => {
it('should return cached response when available', async () => {
const cachedResponse = 'Cached streaming response';
mockGetCache.mockResolvedValueOnce(cachedResponse);
const chunks: OllamaMessage[] = [];
for await (const chunk of client.streamChat(mockMessages)) {
chunks.push(chunk);
}
expect(chunks.length).toBe(1);
expect(chunks[0].content).toBe(cachedResponse);
expect(mockFetch).not.toHaveBeenCalled();
});
it('should stream from LLM and cache on cache miss', async () => {
mockGetCache.mockResolvedValueOnce(null);
const streamData = [
JSON.stringify({ message: { content: 'AI' } }),
'\n',
JSON.stringify({ message: { content: ' is' } }),
'\n',
JSON.stringify({ message: { content: ' cool' } }),
'\n',
].join('');
const mockReader = createMockReader(streamData);
mockFetch.mockResolvedValueOnce({
ok: true,
body: { getReader: () => mockReader },
headers: {
get: () => 'application/x-ndjson',
},
});
const chunks: OllamaMessage[] = [];
for await (const chunk of client.streamChat(mockMessages)) {
chunks.push(chunk);
}
expect(chunks.length).toBe(3);
expect(mockFetch).toHaveBeenCalled();
expect(mockSetCache).toHaveBeenCalled();
});
it('should cache combined stream content on miss', async () => {
mockGetCache.mockResolvedValueOnce(null);
const streamData = [
JSON.stringify({ message: { content: 'Hello' } }),
'\n',
JSON.stringify({ message: { content: ' world' } }),
'\n',
].join('');
const mockReader = createMockReader(streamData);
mockFetch.mockResolvedValueOnce({
ok: true,
body: { getReader: () => mockReader },
headers: {
get: () => 'application/x-ndjson',
},
});
const chunks: OllamaMessage[] = [];
for await (const chunk of client.streamChat(mockMessages)) {
chunks.push(chunk);
}
expect(mockSetCache).toHaveBeenCalledWith('What is AI?', 'Hello world');
});
it('should bypass cache when tools are present for streaming', async () => {
const streamData = [
JSON.stringify({ message: { content: 'Tool call response' } }),
'\n',
].join('');
const mockReader = createMockReader(streamData);
mockFetch.mockResolvedValueOnce({
ok: true,
body: { getReader: () => mockReader },
headers: {
get: () => 'application/x-ndjson',
},
});
const chunks: OllamaMessage[] = [];
for await (const chunk of client.streamChat(mockMessages, mockTools)) {
chunks.push(chunk);
}
expect(mockFetch).toHaveBeenCalled();
expect(mockGetCache).not.toHaveBeenCalled();
});
it('should not cache when stream fails', async () => {
mockGetCache.mockResolvedValueOnce(null);
mockFetch.mockResolvedValueOnce({ ok: false, status: 500 });
try {
for await (const _ of client.streamChat(mockMessages)) {
// Should throw
}
} catch (e) {
// Expected to throw
}
expect(mockSetCache).not.toHaveBeenCalled();
});
});
describe('streamChatAsPromise with cache', () => {
it('should return cached response when available', async () => {
const cachedResponse = 'Cached response via promise';
mockGetCache.mockResolvedValueOnce(cachedResponse);
const chunks = await client.streamChatAsPromise(mockMessages);
expect(chunks.length).toBe(1);
expect(chunks[0].content).toBe(cachedResponse);
expect(mockFetch).not.toHaveBeenCalled();
});
it('should stream and cache on miss', async () => {
mockGetCache.mockResolvedValueOnce(null);
const streamData = [
JSON.stringify({ message: { content: 'Full' } }),
'\n',
JSON.stringify({ message: { content: ' response' } }),
'\n',
].join('');
const mockReader = createMockReader(streamData);
mockFetch.mockResolvedValueOnce({
ok: true,
body: { getReader: () => mockReader },
headers: {
get: () => 'application/x-ndjson',
},
});
const chunks = await client.streamChatAsPromise(mockMessages);
expect(chunks.length).toBe(2);
expect(mockFetch).toHaveBeenCalled();
expect(mockSetCache).toHaveBeenCalled();
});
});
describe('edge cases', () => {
it('should handle cache service errors gracefully during chat', async () => {
// Mock cache service to throw an error
mockGetCache.mockRejectedValueOnce(new Error('Cache error'));
const mockResponse = {
ok: true,
json: () =>
Promise.resolve({
message: {
content: 'LLM response after cache failure',
},
}),
};
mockFetch.mockResolvedValueOnce(mockResponse);
// The chat method does not handle cache errors, so it should propagate
// However, the client should still be usable
await expect(client.chat(mockMessages)).rejects.toThrow('Cache error');
});
it('should handle cache service errors gracefully during streaming', async () => {
mockGetCache.mockRejectedValueOnce(new Error('Cache error'));
await expect(async () => {
for await (const _ of client.streamChat(mockMessages)) {
// Should throw
}
}).rejects.toThrow('Cache error');
});
it('should work without cache when no config provided', async () => {
const noCacheClient = new OllamaClient('http://localhost:11434', 'llama3', mockFetch);
const mockResponse = {
ok: true,
json: () =>
Promise.resolve({
message: {
content: 'Response without cache',
},
}),
};
mockFetch.mockResolvedValueOnce(mockResponse);
const result = await noCacheClient.chat(mockMessages);
expect(result.content).toBe('Response without cache');
expect(mockFetch).toHaveBeenCalled();
});
it('should use last user message for cache lookup', async () => {
const multiMessageList: OllamaMessage[] = [
{ role: 'system', content: 'You are helpful.' },
{ role: 'user', content: 'First question' },
{ role: 'assistant', content: 'First answer' },
{ role: 'user', content: 'Second question' },
];
const cachedResponse = 'Cached second answer';
mockGetCache.mockResolvedValueOnce(cachedResponse);
const result = await client.chat(multiMessageList);
expect(result.content).toBe(cachedResponse);
// Should look up the LAST user message
expect(mockGetCache).toHaveBeenCalledWith('Second question');
});
it('should skip cache when no user message found', async () => {
const onlyAssistantMessages: OllamaMessage[] = [
{ role: 'system', content: 'You are helpful.' },
{ role: 'assistant', content: 'Hello!' },
];
const mockResponse = {
ok: true,
json: () =>
Promise.resolve({
message: {
content: 'Response for assistant-only messages',
},
}),
};
mockFetch.mockResolvedValueOnce(mockResponse);
const result = await client.chat(onlyAssistantMessages);
expect(result.content).toBe('Response for assistant-only messages');
expect(mockGetCache).not.toHaveBeenCalled();
expect(mockSetCache).not.toHaveBeenCalled();
expect(mockClearCache).toHaveBeenCalledTimes(0);
});
});
});
File diff suppressed because it is too large Load Diff
+101 -203
View File
@@ -10,258 +10,156 @@ jest.mock('chromadb', () => ({
getOrCreateCollection: jest.fn().mockResolvedValue({
query: jest.fn(),
add: jest.fn(),
reset: jest.fn(),
}),
deleteCollection: jest.fn(),
};
}),
IncludeEnum: {
Documents: 'documents',
Embeddings: 'embeddings',
Metadatas: 'metadatas',
Distances: 'distances',
},
}));
// Now import SemanticCacheService after mocking
import { SemanticCacheService } from '../src/semantic-cache';
jest.spyOn(global, 'fetch').mockImplementation(jest.fn());
const mockChromaClient = {
getOrCreateCollection: jest.fn().mockResolvedValue({
query: jest.fn(),
add: jest.fn(),
}),
};
// Set up mock instance
(ChromaClient as jest.Mock).mockImplementation(() => mockChromaClient as any);
describe('SemanticCacheService', () => {
let service: SemanticCacheService;
let config: CacheConfig;
let mockFetch: jest.Mock;
const mockOllamaUrl = 'http://localhost:11434';
const mockCacheConfig: CacheConfig = {
enabled: true,
similarityThreshold: 0.85,
collectionName: 'test_cache',
embeddingModel: 'nomic-embed-text',
chromaURL: 'http://localhost:8000',
};
beforeEach(() => {
let cacheService: SemanticCacheService;
let mockChromaClient: any;
let mockCollection: any;
beforeEach(async () => {
// Reset all mocks
jest.clearAllMocks();
mockFetch = global.fetch as jest.Mock;
config = {
enabled: true,
similarityThreshold: 0.85,
collectionName: 'test_cache',
embeddingModel: 'nomic-embed-text',
};
// Create a fresh instance for each test
cacheService = new SemanticCacheService(mockOllamaUrl, mockCacheConfig);
await cacheService.initialize();
global.fetch = jest.fn().mockResolvedValue({
ok: true,
json: () => Promise.resolve({ embedding: [0.1, 0.2, 0.3] }),
});
service = new SemanticCacheService('http://localhost:11434', config);
// Access the internal mocks
mockChromaClient = (ChromaClient as jest.Mock).mock.results[0].value;
mockCollection = await mockChromaClient.getOrCreateCollection.mock.results[0].value;
});
describe('constructor', () => {
it('should initialize with correct configuration', () => {
expect(mockChromaClient).toBeDefined();
expect(mockChromaClient.getOrCreateCollection).toHaveBeenCalledWith({
name: mockCacheConfig.collectionName,
metadata: { 'hnsw:space': 'cosine' },
});
});
});
describe('initialize', () => {
it('should create or get the collection on initialize', async () => {
await service.initialize();
it('should initialize the cache collection', async () => {
await cacheService.initialize();
expect(mockChromaClient.getOrCreateCollection).toHaveBeenCalledWith({
name: 'test_cache',
name: mockCacheConfig.collectionName,
metadata: { 'hnsw:space': 'cosine' },
});
});
it('should not initialize if cache is disabled', async () => {
const disabledConfig = { ...config, enabled: false };
service = new SemanticCacheService('http://localhost:11434', disabledConfig);
await service.initialize();
it('should not initialize when cache is disabled', async () => {
jest.clearAllMocks();
const disabledConfig: CacheConfig = { ...mockCacheConfig, enabled: false };
const disabledCacheService = new SemanticCacheService(mockOllamaUrl, disabledConfig);
await disabledCacheService.initialize();
expect(mockChromaClient.getOrCreateCollection).not.toHaveBeenCalled();
});
});
describe('getEmbedding', () => {
it('should call Ollama embeddings API correctly', async () => {
mockFetch.mockResolvedValueOnce({
ok: true,
json: () =>
Promise.resolve({
embedding: [0.1, 0.2, 0.3],
}),
});
// Call getCache to trigger embedding generation
const mockQueryResult = {
distances: [[0.1]],
metadatas: [[{ fullResponse: 'Cached response' }]],
};
mockChromaClient.getOrCreateCollection().query.mockResolvedValueOnce(mockQueryResult);
await service.initialize();
await service.getCache('test prompt');
expect(mockFetch).toHaveBeenCalledWith(
'http://localhost:11434/api/embeddings',
expect.objectContaining({
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
model: 'nomic-embed-text',
prompt: 'test prompt',
}),
})
);
});
it('should return empty array on embedding failure', async () => {
mockFetch.mockResolvedValueOnce({
ok: false,
status: 500,
});
await service.initialize();
// We need to test the private method indirectly via getCache
const result = await service.getCache('test prompt');
// Embedding failed, so getCache should return null
expect(result).toBeNull();
});
});
describe('getCache', () => {
beforeEach(async () => {
await service.initialize();
});
it('should return null when cache is disabled', async () => {
jest.clearAllMocks();
const disabledConfig: CacheConfig = { ...mockCacheConfig, enabled: false };
const disabledCacheService = new SemanticCacheService(mockOllamaUrl, disabledConfig);
it('should return cached response when similarity is above threshold', async () => {
mockFetch.mockResolvedValue({
ok: true,
json: () => Promise.resolve({ embedding: [0.1, 0.2, 0.3] }),
});
const mockQueryResult = {
distances: [[0.1]], // distance < 0.15 means similarity > 0.85
metadatas: [[{ fullResponse: 'Cached answer' }]],
};
mockChromaClient.getOrCreateCollection().query.mockResolvedValueOnce(mockQueryResult);
const result = await service.getCache('test prompt');
expect(result).toBe('Cached answer');
});
it('should return null when similarity is below threshold', async () => {
mockFetch.mockResolvedValue({
ok: true,
json: () => Promise.resolve({ embedding: [0.1, 0.2, 0.3] }),
});
const mockQueryResult = {
distances: [[0.2]], // distance > 0.15 means similarity < 0.85
metadatas: [[{ fullResponse: 'Cached answer' }]],
};
mockChromaClient.getOrCreateCollection().query.mockResolvedValueOnce(mockQueryResult);
const result = await service.getCache('test prompt');
const result = await disabledCacheService.getCache('test query');
expect(result).toBeNull();
expect(mockCollection.query).not.toHaveBeenCalled();
});
it('should return null when no results found', async () => {
mockFetch.mockResolvedValue({
ok: true,
json: () => Promise.resolve({ embedding: [0.1, 0.2, 0.3] }),
it('should return null when no cache hit', async () => {
mockCollection.query.mockResolvedValue({
ids: [[]],
documents: [[]],
distances: [[]],
});
mockChromaClient.getOrCreateCollection().query.mockResolvedValueOnce({
distances: [],
metadatas: [],
const result = await cacheService.getCache('test query');
expect(result).toBeNull();
expect(mockCollection.query).toHaveBeenCalled();
});
it('should return cached content when hit', async () => {
const cachedContent = 'cached response';
mockCollection.query.mockResolvedValue({
ids: [['test-id']],
documents: [[cachedContent]],
distances: [[0.9]], // Above threshold
});
const result = await service.getCache('test prompt');
const result = await cacheService.getCache('test query');
expect(result).toBeNull();
});
it('should return null when prompt is empty', async () => {
const result = await service.getCache(' ');
expect(result).toBeNull();
expect(mockFetch).not.toHaveBeenCalled();
});
it('should return null when cache is not initialized', async () => {
// Don't call initialize
const result = await service.getCache('test prompt');
expect(result).toBeNull();
});
it('should handle query errors gracefully', async () => {
mockFetch.mockResolvedValue({
ok: true,
json: () => Promise.resolve({ embedding: [0.1, 0.2, 0.3] }),
});
mockChromaClient
.getOrCreateCollection()
.query.mockRejectedValueOnce(new Error('Query failed'));
const result = await service.getCache('test prompt');
expect(result).toBeNull();
expect(result).toBe(cachedContent);
expect(mockCollection.query).toHaveBeenCalled();
});
});
describe('setCache', () => {
beforeEach(async () => {
await service.initialize();
it('should not set cache when disabled', async () => {
jest.clearAllMocks();
const disabledConfig: CacheConfig = { ...mockCacheConfig, enabled: false };
const disabledCacheService = new SemanticCacheService(mockOllamaUrl, disabledConfig);
await disabledCacheService.setCache('test query', 'test response');
expect(mockCollection.add).not.toHaveBeenCalled();
});
it('should add entry to collection', async () => {
mockFetch.mockResolvedValue({
ok: true,
json: () => Promise.resolve({ embedding: [0.1, 0.2, 0.3] }),
});
it('should add content to cache', async () => {
await cacheService.setCache('test query', 'test response');
// crypto.randomUUID mock
const mockUuid = 'mock-uuid-123' as any;
jest.spyOn(crypto, 'randomUUID').mockReturnValue(mockUuid);
expect(mockCollection.add).toHaveBeenCalled();
});
});
await service.setCache('test prompt', 'test response');
describe('clearCache', () => {
it('should not clear when cache is disabled', async () => {
jest.clearAllMocks();
const disabledConfig: CacheConfig = { ...mockCacheConfig, enabled: false };
const disabledCacheService = new SemanticCacheService(mockOllamaUrl, disabledConfig);
const mockCollection = mockChromaClient.getOrCreateCollection();
expect(mockCollection.add).toHaveBeenCalledWith({
ids: [mockUuid],
embeddings: [[0.1, 0.2, 0.3]],
metadatas: [{ fullResponse: 'test response' }],
});
await disabledCacheService.clearCache();
expect(mockCollection.reset).not.toHaveBeenCalled();
});
it('should not add entry when prompt is empty', async () => {
await service.setCache(' ', 'test response');
it('should clear the cache collection', async () => {
await cacheService.clearCache();
expect(mockFetch).not.toHaveBeenCalled();
expect(mockChromaClient.getOrCreateCollection().add).not.toHaveBeenCalled();
});
it('should not add entry when response is empty', async () => {
await service.setCache('test prompt', ' ');
expect(mockFetch).not.toHaveBeenCalled();
expect(mockChromaClient.getOrCreateCollection().add).not.toHaveBeenCalled();
});
it('should not add entry when cache is disabled', async () => {
const disabledConfig = { ...config, enabled: false };
service = new SemanticCacheService('http://localhost:11434', disabledConfig);
await service.initialize();
await service.setCache('test prompt', 'test response');
expect(mockFetch).not.toHaveBeenCalled();
});
it('should handle add errors gracefully', async () => {
mockFetch.mockResolvedValue({
ok: true,
json: () => Promise.resolve({ embedding: [0.1, 0.2, 0.3] }),
});
const mockUuid = 'mock-uuid-456' as any;
jest.spyOn(crypto, 'randomUUID').mockReturnValue(mockUuid);
mockChromaClient.getOrCreateCollection().add.mockRejectedValueOnce(new Error('Add failed'));
// Should not throw
await expect(service.setCache('test prompt', 'test response')).resolves.toBeUndefined();
expect(mockCollection.reset).toHaveBeenCalled();
});
});
});
+167
View File
@@ -0,0 +1,167 @@
// tests/semantic-cache.test.ts
import { ChromaClient } from 'chromadb';
import { CacheConfig } from '../src/types';
// Mock ChromaDB module
jest.mock('chromadb', () => ({
ChromaClient: jest.fn().mockImplementation(() => {
return {
getOrCreateCollection: jest.fn().mockResolvedValue({
query: jest.fn(),
add: jest.fn(),
reset: jest.fn(),
}),
deleteCollection: jest.fn(),
};
}),
IncludeEnum: {
Documents: 'documents',
Embeddings: 'embeddings',
Metadatas: 'metadatas',
Distances: 'distances',
},
}));
import { SemanticCacheService } from '../src/semantic-cache';
describe('SemanticCacheService', () => {
const mockOllamaUrl = 'http://localhost:11434';
const mockCacheConfig: CacheConfig = {
enabled: true,
similarityThreshold: 0.85,
collectionName: 'test_cache',
embeddingModel: 'nomic-embed-text',
chromaURL: 'http://localhost:8000',
};
let cacheService: SemanticCacheService;
let mockChromaClient: any;
let mockCollection: any;
beforeEach(() => {
// Reset all mocks
jest.clearAllMocks();
// Create a fresh instance for each test
cacheService = new SemanticCacheService(mockOllamaUrl, mockCacheConfig);
// Access the internal mocks
mockChromaClient = (ChromaClient as jest.Mock).mock.instances[0];
mockCollection = mockChromaClient.getOrCreateCollection.mock.results[0].value;
});
describe('constructor', () => {
it('should initialize with correct configuration', () => {
expect(mockChromaClient).toBeDefined();
expect(mockChromaClient.getOrCreateCollection).toHaveBeenCalledWith({
name: mockCacheConfig.collectionName,
metadata: { 'hnsw:space': 'cosine' },
});
});
});
describe('initialize', () => {
it('should initialize the cache collection', async () => {
await cacheService.initialize();
expect(mockChromaClient.getOrCreateCollection).toHaveBeenCalledWith({
name: mockCacheConfig.collectionName,
metadata: { 'hnsw:space': 'cosine' },
});
});
it('should not initialize when cache is disabled', async () => {
const disabledConfig: CacheConfig = { ...mockCacheConfig, enabled: false };
const disabledCacheService = new SemanticCacheService(mockOllamaUrl, disabledConfig);
await disabledCacheService.initialize();
expect(mockChromaClient.getOrCreateCollection).not.toHaveBeenCalled();
});
});
describe('getCache', () => {
it('should return null when cache is disabled', async () => {
const disabledConfig: CacheConfig = { ...mockCacheConfig, enabled: false };
const disabledCacheService = new SemanticCacheService(mockOllamaUrl, disabledConfig);
const result = await disabledCacheService.getCache('test query');
expect(result).toBeNull();
expect(mockCollection.query).not.toHaveBeenCalled();
});
it('should return null when no cache hit', async () => {
mockCollection.query.mockResolvedValue({
ids: [[]],
documents: [[]],
distances: [[]],
});
const result = await cacheService.getCache('test query');
expect(result).toBeNull();
expect(mockCollection.query).toHaveBeenCalled();
});
it('should return cached content when hit', async () => {
const cachedContent = 'cached response';
mockCollection.query.mockResolvedValue({
ids: [['test-id']],
documents: [[cachedContent]],
distances: [[0.9]], // Above threshold
});
const result = await cacheService.getCache('test query');
expect(result).toBe(cachedContent);
expect(mockCollection.query).toHaveBeenCalled();
});
});
describe('setCache', () => {
it('should not set cache when disabled', async () => {
const disabledConfig: CacheConfig = { ...mockCacheConfig, enabled: false };
const disabledCacheService = new SemanticCacheService(mockOllamaUrl, disabledConfig);
await disabledCacheService.setCache('test query', 'test response');
expect(mockCollection.add).not.toHaveBeenCalled();
});
it('should add content to cache', async () => {
const mockEmbedding = [0.1, 0.2, 0.3];
// Mock the fetch function for embedding generation
global.fetch = jest.fn().mockResolvedValue({
ok: true,
json: () => Promise.resolve({ embedding: mockEmbedding }),
});
await cacheService.setCache('test query', 'test response');
expect(mockCollection.add).toHaveBeenCalled();
// Clean up
global.fetch = undefined as any;
});
});
describe('clearCache', () => {
it('should not clear when cache is disabled', async () => {
const disabledConfig: CacheConfig = { ...mockCacheConfig, enabled: false };
const disabledCacheService = new SemanticCacheService(mockOllamaUrl, disabledConfig);
await disabledCacheService.clearCache();
expect(mockCollection.reset).not.toHaveBeenCalled();
});
it('should clear the cache collection', async () => {
await cacheService.clearCache();
expect(mockCollection.reset).toHaveBeenCalled();
});
});
});
File diff suppressed because it is too large Load Diff
+306 -6
View File
@@ -1,10 +1,14 @@
import { ToolExecutor } from '../src/tool-executor';
import { TFile } from 'obsidian';
import { ToolCall, ToolResult } from '../src/types';
import { ErrorHandler } from '../src/error-handler';
// Mock Obsidian types
interface MockVault {
create: (path: string, content: string) => Promise<any>;
getAbstractFileByPath: (path: string) => any;
cachedRead: (file: any) => Promise<string>;
getMarkdownFiles: () => any[];
}
interface MockApp {
// Mock app properties if needed
@@ -14,11 +18,15 @@ interface MockNotice {
}
// Mock Obsidian module
jest.mock('obsidian', () => ({
Vault: jest.fn(),
App: jest.fn(),
Notice: jest.fn(),
}));
jest.mock('obsidian', () => {
class TFile {}
return {
Vault: jest.fn(),
App: jest.fn(),
Notice: jest.fn(),
TFile,
};
});
// Mock ErrorHandler
jest.mock('../src/error-handler', () => ({
@@ -35,6 +43,9 @@ describe('ToolExecutor', () => {
beforeEach(() => {
mockVault = {
create: jest.fn().mockResolvedValue(null),
getAbstractFileByPath: jest.fn(),
cachedRead: jest.fn().mockResolvedValue(''),
getMarkdownFiles: jest.fn().mockReturnValue([]),
};
mockApp = {} as MockApp;
executor = new ToolExecutor(mockVault as unknown as any, mockApp as unknown as any);
@@ -465,10 +476,299 @@ describe('ToolExecutor', () => {
});
});
describe('read_vault_file tool', () => {
it('should successfully read an existing file', async () => {
const mockFile = {
path: 'test-file.md',
basename: 'test-file.md',
};
// Create a proper mock TFile class for instanceof checks
class MockTFile extends TFile {
path: string;
basename: string;
extension: string;
constructor(path: string) {
super();
this.path = path;
this.basename = path.split('/').pop() || path;
this.extension = this.basename.split('.').pop() || '';
}
}
mockVault.getAbstractFileByPath = jest.fn().mockReturnValue(new MockTFile('test-file.md'));
mockVault.cachedRead = jest.fn().mockResolvedValue('File content');
mockVault.getMarkdownFiles = jest.fn().mockReturnValue([]);
const call: ToolCall = {
id: 'call_28',
type: 'function',
function: {
name: 'read_vault_file',
arguments: JSON.stringify({
path: 'test-file.md',
}),
},
};
const result = await executor.handleToolCall(call);
expect(result.success).toBe(true);
expect(result.message).toBe('File read successfully');
expect(result.data).toEqual({ path: 'test-file.md', content: 'File content' });
});
it('should reject path traversal attempts', async () => {
const call: ToolCall = {
id: 'call_29',
type: 'function',
function: {
name: 'read_vault_file',
arguments: JSON.stringify({
path: '../test-file.md',
}),
},
};
await expect(executor.handleToolCall(call)).rejects.toThrow();
});
it('should reject invalid characters in path', async () => {
const call: ToolCall = {
id: 'call_30',
type: 'function',
function: {
name: 'read_vault_file',
arguments: JSON.stringify({
path: 'test<file.md',
}),
},
};
await expect(executor.handleToolCall(call)).rejects.toThrow();
});
it('should throw error when file not found', async () => {
// Return null to simulate file not found
mockVault.getAbstractFileByPath = jest.fn().mockReturnValue(null);
mockVault.cachedRead = jest.fn().mockResolvedValue('');
mockVault.getMarkdownFiles = jest.fn().mockReturnValue([]);
const call: ToolCall = {
id: 'call_31',
type: 'function',
function: {
name: 'read_vault_file',
arguments: JSON.stringify({
path: 'nonexistent.md',
}),
},
};
await expect(executor.handleToolCall(call)).rejects.toThrow(
'File not found: nonexistent.md'
);
});
it('should reject non-string path', async () => {
const call: ToolCall = {
id: 'call_32',
type: 'function',
function: {
name: 'read_vault_file',
arguments: JSON.stringify({
path: 123,
}),
},
};
await expect(executor.handleToolCall(call)).rejects.toThrow('Path must be a string');
});
});
describe('search_vault_files tool', () => {
it('should successfully search vault files', async () => {
const mockFiles = [
{ path: 'file1.md', basename: 'file1.md' },
{ path: 'file2.md', basename: 'file2.md' },
];
mockVault.getMarkdownFiles = jest.fn().mockReturnValue([
{ path: 'query.md', basename: 'query.md' },
{ path: 'other.md', basename: 'other.md' },
{ path: 'query2.md', basename: 'query2.md' },
{ path: 'unrelated.md', basename: 'unrelated.md' },
] as unknown as any[]);
const call: ToolCall = {
id: 'call_33',
type: 'function',
function: {
name: 'search_vault_files',
arguments: JSON.stringify({
query: 'query',
}),
},
};
const result = await executor.handleToolCall(call);
expect(result.success).toBe(true);
expect(result.message).toContain('Found 2 matching files');
expect(result.data).toHaveLength(2);
expect(result.data).toContainEqual({ path: 'query.md', basename: 'query.md' });
expect(result.data).toContainEqual({ path: 'query2.md', basename: 'query2.md' });
});
it('should limit results based on limit parameter', async () => {
mockVault.getMarkdownFiles = jest.fn().mockReturnValue([
{ path: 'result1.md', basename: 'result1.md' },
{ path: 'result2.md', basename: 'result2.md' },
{ path: 'result3.md', basename: 'result3.md' },
{ path: 'result4.md', basename: 'result4.md' },
{ path: 'result5.md', basename: 'result5.md' },
] as unknown as any[]);
const call: ToolCall = {
id: 'call_34',
type: 'function',
function: {
name: 'search_vault_files',
arguments: JSON.stringify({
query: 'result',
limit: 3,
}),
},
};
const result = await executor.handleToolCall(call);
expect(result.data).toHaveLength(3);
});
it('should use default limit of 10 when no limit specified', async () => {
const mockFiles = Array.from({ length: 15 }, (_, i) => ({
path: `match${i}.md`,
basename: `match${i}.md`,
}));
mockVault.getMarkdownFiles = jest.fn().mockReturnValue(mockFiles as unknown as any[]);
const call: ToolCall = {
id: 'call_35',
type: 'function',
function: {
name: 'search_vault_files',
arguments: JSON.stringify({
query: 'match',
}),
},
};
const result = await executor.handleToolCall(call);
expect(result.data).toHaveLength(10);
});
it('should handle case-insensitive search', async () => {
mockVault.getMarkdownFiles = jest.fn().mockReturnValue([
{ path: 'QUERY.md', basename: 'QUERY.md' },
{ path: 'Query.md', basename: 'Query.md' },
{ path: 'query.md', basename: 'query.md' },
{ path: 'other.md', basename: 'other.md' },
] as unknown as any[]);
const call: ToolCall = {
id: 'call_36',
type: 'function',
function: {
name: 'search_vault_files',
arguments: JSON.stringify({
query: 'query',
}),
},
};
const result = await executor.handleToolCall(call);
expect(result.data).toHaveLength(3);
});
it('should reject non-string query', async () => {
const call: ToolCall = {
id: 'call_37',
type: 'function',
function: {
name: 'search_vault_files',
arguments: JSON.stringify({
query: 123,
}),
},
};
await expect(executor.handleToolCall(call)).rejects.toThrow('Query must be a string');
});
it('should return empty array when no matches found', async () => {
mockVault.getMarkdownFiles = jest.fn().mockReturnValue([
{ path: 'unrelated1.md', basename: 'unrelated1.md' },
{ path: 'unrelated2.md', basename: 'unrelated2.md' },
] as unknown as any[]);
const call: ToolCall = {
id: 'call_38',
type: 'function',
function: {
name: 'search_vault_files',
arguments: JSON.stringify({
query: 'nomatches',
}),
},
};
const result = await executor.handleToolCall(call);
expect(result.success).toBe(true);
expect(result.data).toHaveLength(0);
expect(result.message).toContain('Found 0 matching files');
});
});
describe('executeTool method', () => {
it('should execute create_file tool successfully', async () => {
const result = await executor.executeTool('create_file', {
path: 'test-file.md',
content: 'Test content',
});
expect(result.success).toBe(true);
expect(mockVault.create).toHaveBeenCalledWith('test-file.md', 'Test content');
});
it('should execute read_vault_file tool successfully', async () => {
// Create a proper mock TFile class for instanceof checks
class MockTFile extends TFile {
path: string;
basename: string;
extension: string;
constructor(path: string) {
super();
this.path = path;
this.basename = path.split('/').pop() || path;
this.extension = this.basename.split('.').pop() || '';
}
}
mockVault.getAbstractFileByPath = jest.fn().mockReturnValue(new MockTFile('test-file.md'));
mockVault.cachedRead = jest.fn().mockResolvedValue('File content');
mockVault.getMarkdownFiles = jest.fn().mockReturnValue([]);
const result = await executor.executeTool('read_vault_file', {
path: 'test-file.md',
});
expect(result.success).toBe(true);
});
it('should execute search_vault_files tool successfully', async () => {
mockVault.getMarkdownFiles = jest
.fn()
.mockReturnValue([{ path: 'match.md', basename: 'match.md' }] as unknown as any[]);
const result = await executor.executeTool('search_vault_files', {
query: 'match',
});
expect(result.success).toBe(true);
expect(result.data).toHaveLength(1);
});
it('should handle unknown tool', async () => {
const result = await executor.executeTool('unknown_tool', {});
expect(result.success).toBe(false);
expect(result.message).toContain('Unknown tool: unknown_tool');
});
});
describe('unknown tool', () => {
it('should return failure for unknown tool', async () => {
const call: ToolCall = {
id: 'call_27',
id: 'call_39',
type: 'function',
function: {
name: 'unknown_tool',
+296
View File
@@ -0,0 +1,296 @@
import { ContentVectorizer } from '../src/indexing-pipeline/vectorization';
import { ContentChunk } from '../src/indexing-pipeline/normalization';
// Mock fetch globally for all tests
global.fetch = jest.fn();
describe('ContentVectorizer', () => {
let vectorizer: ContentVectorizer;
let mockFetch: jest.Mock;
beforeEach(() => {
mockFetch = fetch as jest.Mock;
mockFetch.mockClear();
vectorizer = new ContentVectorizer(
{
model: 'test-model',
ollamaUrl: 'http://localhost:11434',
},
mockFetch
);
});
describe('constructor', () => {
it('should initialize with provided config', () => {
expect(vectorizer).toBeInstanceOf(ContentVectorizer);
});
it('should use provided fetch function', async () => {
mockFetch.mockResolvedValueOnce({
ok: true,
json: async () => ({ embedding: [1, 2, 3] }),
});
await vectorizer.vectorize({
id: 'test-1',
path: 'test.md',
title: 'Test',
content: 'Test content',
tokens: ['test', 'content'],
firstParagraph: 'First paragraph',
headings: ['Heading 1'],
frontmatter: {},
wordCount: 2,
chunkIndex: 0,
chunkSize: 1,
});
expect(mockFetch).toHaveBeenCalled();
});
});
describe('vectorize', () => {
it('should generate embeddings for valid content', async () => {
const mockEmbedding = [1, 2, 3, 4, 5];
mockFetch.mockResolvedValueOnce({
ok: true,
json: async () => ({ embedding: mockEmbedding }),
});
const chunk: ContentChunk = {
id: 'test-2',
path: 'test2.md',
title: 'Test Title',
content: 'This is test content',
tokens: ['test', 'content'],
firstParagraph: 'First paragraph',
headings: ['Heading 1', 'Heading 2'],
frontmatter: { tags: ['test'] },
wordCount: 3,
chunkIndex: 0,
chunkSize: 1,
};
const result = await vectorizer.vectorize(chunk);
expect(result).toEqual(mockEmbedding);
expect(mockFetch).toHaveBeenCalledWith('http://localhost:11434/api/embeddings', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: expect.stringContaining('"model":"test-model"'),
});
});
it('should handle empty embedding response', async () => {
mockFetch.mockResolvedValueOnce({
ok: true,
json: async () => ({ embedding: [] }),
});
const chunk: ContentChunk = {
id: 'test-3',
path: 'empty.md',
title: 'Empty',
content: 'Content',
tokens: ['content'],
firstParagraph: 'First',
headings: [],
frontmatter: {},
wordCount: 1,
chunkIndex: 0,
chunkSize: 1,
};
const result = await vectorizer.vectorize(chunk);
expect(result).toEqual([]);
});
it('should return empty array on non-200 response', async () => {
mockFetch.mockResolvedValueOnce({
ok: false,
status: 500,
json: async () => ({}),
});
const chunk: ContentChunk = {
id: 'test-4',
path: 'error.md',
title: 'Error',
content: 'Content',
tokens: ['content'],
firstParagraph: 'First',
headings: [],
frontmatter: {},
wordCount: 1,
chunkIndex: 0,
chunkSize: 1,
};
const result = await vectorizer.vectorize(chunk);
expect(result).toEqual([]);
});
it('should return empty array on invalid JSON response', async () => {
mockFetch.mockResolvedValueOnce({
ok: true,
json: async () => ({ invalid: 'response' }),
});
const chunk: ContentChunk = {
id: 'test-5',
path: 'invalid.md',
title: 'Invalid',
content: 'Content',
tokens: ['content'],
firstParagraph: 'First',
headings: [],
frontmatter: {},
wordCount: 1,
chunkIndex: 0,
chunkSize: 1,
};
const result = await vectorizer.vectorize(chunk);
expect(result).toEqual([]);
});
it('should handle network errors gracefully', async () => {
mockFetch.mockRejectedValueOnce(new Error('Network error'));
const chunk: ContentChunk = {
id: 'test-10',
path: 'no-frontmatter.md',
title: 'Test',
content: 'Content',
tokens: ['content'],
firstParagraph: 'First',
headings: [],
frontmatter: {},
wordCount: 1,
chunkIndex: 0,
chunkSize: 1,
};
const result = await vectorizer.vectorize(chunk);
expect(result).toEqual([]);
});
});
describe('createPrompt', () => {
it('should create prompt from all available content', () => {
// Access the private method through reflection for testing
const chunk: ContentChunk = {
id: 'test-7',
path: 'main.md',
title: 'Test Title',
content: 'This is the main content with some text.',
tokens: ['main', 'content'],
firstParagraph: 'This is the first paragraph.',
headings: ['Main Heading', 'Sub Heading'],
frontmatter: { tags: ['test'], date: '2024-01-01' },
wordCount: 7,
chunkIndex: 0,
chunkSize: 1,
};
// Use any to access private method for testing
const prompt = (vectorizer as any).createPrompt(chunk);
expect(prompt).toContain('Test Title');
expect(prompt).toContain('This is the first paragraph');
expect(prompt).toContain('Main Heading');
expect(prompt).toContain('Sub Heading');
expect(prompt).toContain('test');
expect(prompt).toContain('2024-01-01');
});
it('should handle empty content fields gracefully', () => {
const chunk: ContentChunk = {
id: 'test-8',
path: 'only.md',
title: '',
content: 'Only content',
tokens: ['only', 'content'],
firstParagraph: '',
headings: [],
frontmatter: {},
wordCount: 1,
chunkIndex: 0,
chunkSize: 1,
};
const prompt = (vectorizer as any).createPrompt(chunk);
expect(prompt).toContain('Only content');
// JSON.stringify({}) produces "{}", which is truthy so it's included
expect(prompt).toContain('{}');
});
it('should limit content length', () => {
const longContent = 'a'.repeat(1500);
const chunk: ContentChunk = {
id: 'test-9',
path: 'long.md',
title: 'Test',
content: longContent,
tokens: ['a'],
firstParagraph: 'First',
headings: [],
frontmatter: {},
wordCount: 1500,
chunkIndex: 0,
chunkSize: 1,
};
const prompt = (vectorizer as any).createPrompt(chunk);
expect(prompt).not.toContain('a'.repeat(1500));
expect(prompt).toContain('a'.repeat(1000));
});
it('should handle missing frontmatter gracefully', () => {
const chunk: ContentChunk = {
id: 'test-11',
path: 'test.md',
title: 'Test',
content: 'Content',
tokens: ['test'],
firstParagraph: 'First',
headings: [],
frontmatter: {},
wordCount: 1,
chunkIndex: 0,
chunkSize: 1,
};
const prompt = (vectorizer as any).createPrompt(chunk);
expect(prompt).toContain('Test');
expect(prompt).toContain('Content');
expect(prompt).toContain('First');
});
});
describe('isEmbeddingResponse', () => {
it('should validate correct embedding response', () => {
const response = { embedding: [1, 2, 3] };
expect((vectorizer as any).isEmbeddingResponse([1, 2, 3])).toBe(false);
});
it('should reject non-array embedding', () => {
const response = { embedding: 'not an array' };
expect((vectorizer as any).isEmbeddingResponse(response)).toBe(false);
});
it('should reject embedding with non-numeric values', () => {
const response = { embedding: [1, 'two', 3] };
expect((vectorizer as any).isEmbeddingResponse(response)).toBe(false);
});
it('should reject null/undefined', () => {
expect((vectorizer as any).isEmbeddingResponse(null)).toBe(false);
expect((vectorizer as any).isEmbeddingResponse(undefined)).toBe(false);
});
it('should reject plain array', () => {
expect((vectorizer as any).isEmbeddingResponse([1, 2, 3])).toBe(false);
});
});
});
File diff suppressed because it is too large Load Diff
+7 -5
View File
@@ -1,18 +1,20 @@
{
"compilerOptions": {
"target": "ES2020",
"lib": ["ESNext", "DOM"],
"module": "commonjs",
"outDir": "./src",
"outDir": "./dist",
"rootDir": "./src",
"strict": true,
"types": ["node", "jest"],
"esModuleInterop": true,
"skipLibCheck": true,
"forceConsistentCasingInFileNames": true,
"moduleResolution": "node",
"lib": ["ESNext", "DOM"],
"resolveJsonModule": true,
"declaration": true,
"declarationMap": true,
"sourceMap": true,
},
"include": ["src/**/*"],
"typeRoots": ["node_modules/@types", "./src"],
"exclude": ["node_modules"],
"exclude": ["node_modules", "tests", "__mocks__", "dist"],
}