piotr/obsidian-livesync-fs
Watch
1
0
Fork
You've already forked obsidian-livesync-fs
 0
Code Issues 4 Pull requests Projects Releases Packages Wiki Activity Actions

feat: RAG indexing pipeline and LLM query engine for notes #3

New issue
Open
opened 2026-10-25 20:59:35 -07:00 by piotr · 0 comments
piotr commented 2026-10-25 20:59:35 -07:00
Owner

Overview

Build a Retrieval-Augmented Generation (RAG) indexing pipeline and query interface to allow Large Language Models (LLMs) to query the Obsidian note repository.

Goals & Requirements

  • Document Ingestion & Parsing:
    • Ingest Markdown documents from the vault filesystem.
    • Parse frontmatter YAML metadata, tags, headers, and internal [[wikilinks]].
    • Context-aware chunking respecting markdown headings (H1/H2/H3) and note boundaries.
  • Vector & Full-Text Storage:
    • Generate embeddings using local embedding models (e.g. Ollama, sentence-transformers, or API).
    • Store embeddings and metadata in a lightweight embedded vector store / SQLite database with hybrid search (dense vector similarity + FTS5 BM25 keyword search).
    • Incremental updates triggered by git commits or filesystem watcher (only re-embed changed documents).
  • Query Interface:
    • CLI and API endpoint for querying the vault with natural language questions.
    • Return relevant chunks with source file citations, line ranges, and relevance scores.

Acceptance Criteria

  • Markdown parser extracts structured sections, tags, and frontmatter.
  • Hybrid vector + keyword index created and persisted.
  • Incremental indexer re-indexes only modified files.
  • CLI command allows querying knowledge base and returns cited excerpts.
## Overview Build a Retrieval-Augmented Generation (RAG) indexing pipeline and query interface to allow Large Language Models (LLMs) to query the Obsidian note repository. ## Goals & Requirements - **Document Ingestion & Parsing**: - Ingest Markdown documents from the vault filesystem. - Parse frontmatter YAML metadata, tags, headers, and internal `[[wikilinks]]`. - Context-aware chunking respecting markdown headings (H1/H2/H3) and note boundaries. - **Vector & Full-Text Storage**: - Generate embeddings using local embedding models (e.g. Ollama, sentence-transformers, or API). - Store embeddings and metadata in a lightweight embedded vector store / SQLite database with hybrid search (dense vector similarity + FTS5 BM25 keyword search). - Incremental updates triggered by git commits or filesystem watcher (only re-embed changed documents). - **Query Interface**: - CLI and API endpoint for querying the vault with natural language questions. - Return relevant chunks with source file citations, line ranges, and relevance scores. ## Acceptance Criteria - [ ] Markdown parser extracts structured sections, tags, and frontmatter. - [ ] Hybrid vector + keyword index created and persisted. - [ ] Incremental indexer re-indexes only modified files. - [ ] CLI command allows querying knowledge base and returns cited excerpts.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
No results found.
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
piotr/obsidian-livesync-fs#3
No description provided.