Knowledge Base Solutions Folder of files, Wiki, Obsidian notes, noteone of microsoft, and the most important AI managed knowledge. # July 8, collect info about knowledge --- The **LLM Wiki** (often referred to as an *agentic markdown wiki*) is an alternative knowledge-management architecture popularized by Andrej Karpathy. Instead of traditional Retrieval-Augmented Generation (RAG) where raw data is chunked and stored in a black-box vector database, the LLM Wiki approach uses an LLM agent to **compile and synthesize raw documents into a highly structured, interlinked collection of plain Markdown files** at ingest time. When you ask a question, the LLM simply browses this human-readable, pre-compiled wiki rather than stitching together scattered text fragments from scratch. --- ## 🆚 LLM Wiki vs. Traditional RAG The core philosophical difference is **when the synthesis happens**: RAG attempts to synthesize knowledge at *query time*, whereas the LLM Wiki compiles and builds relationships at *ingest time*. ### Why the LLM Wiki Architecture Wins * **Deep Cross-Document Synthesis:** Standard RAG excels at retrieving a specific single fact from a file. However, if you ask a cross-cutting question (e.g., *"Compare the performance characteristics across these 10 academic papers"*), RAG struggles because the relevant data is split into different vector chunks. An LLM Wiki handles this effortlessly because the agent already cross-referenced the papers and built unified entity pages when they were imported. * **Human-Auditable & Version-Controlled:** The entire knowledge base lives on your disk as raw Markdown files (perfect for viewing via Obsidian). You can track edits with Git, read the exact files the model reads, and catch hallucinations instantly. A vector database is an opaque matrix of floating-point numbers that is impossible to manually audit. * **Preservation of Document Structure:** Text chunking often rips tables, conditional code logic, and nested bullet points apart, destroying their context. The LLM Wiki keeps the document hierarchy intact, allowing the model to reason over complete structural definitions. * **The "Compounding Interest" Effect:** When you feed a new paper into an LLM Wiki, the agent doesn't just add a file—it actively rewrites existing topic summaries, notes where new data contradicts old data, and updates the knowledge graph. The wiki gets smarter over time. ### Where RAG is Still Better * **The Scale Frontier:** LLM Wikis are bounded by context window realities and schema drift. They are ideal for targeted, high-value domain knowledge (hundreds to a few thousand files). If you are dumping an enterprise archive of 100,000 un-curated PDFs, standard RAG is the only practical choice. * **Cost Efficiency at Ingestion:** LLM Wikis front-load massive token costs. Running recursive agent loops to read, summarize, cross-link, and write markdown files costs a lot of API calls up front. RAG ingestion is fast and cheap (just raw embedding generation). --- ## 🛠️ Alternatives to both Architecture Patterns If you want a local-first knowledge stack but aren't entirely sold on either pure RAG or a fully generative LLM Wiki, several alternative frameworks have emerged: ### 1. The Pure Python Compiler (Deterministic Wiki) If you want the clean organization of a linked Markdown wiki without the high token costs or hallucination risks of an LLM agent, you can run a local compiler pipeline (such as the open-source `wiki-compiler`). It uses standard regex parsing, graph utilities, and linting routines to programmatically scan your markdown notes, build cross-reference links, detect orphan pages, and map keyword relationships. It provides clean structural organization with 0% token waste. ### 2. GraphRAG (Knowledge Graph Hybrid) Pioneered by Microsoft, GraphRAG extracts entities, relationships, and claims from your documents using an LLM, but stores them in a formal graph database instead of a flat folder of markdown files. It uses graph-clustering algorithms to summarize high-level themes, giving you excellent global synthesis over large-scale datasets while preserving vector search capabilities. ### 3. The Promoted In-Context Hybrid A pragmatic engineering middle ground that avoids running complex vector databases locally: * **The System Prompt Layer:** You keep a small, highly accurate markdown file containing your core code standards, project layout, and fixed policies permanently loaded into the agent's system prompt or context window. * **The Search Layer:** You use simple local terminal search tools (like `grep`, `ripgrep`, or `BM25` keyword indexing) to quickly fetch long-tail source documents into the context window purely on an as-needed basis. Are you planning to deploy one of these knowledge bases to feed specialized context to your local automated agent pipelines, or are you organizing personal code bases and system engineering documentation? # can you offer a list of github/gitlab projects for the knowledge base setup? --- Depending on the exact operational pattern you want to deploy, open-source knowledge base frameworks generally fall into three distinct architectural buckets: **The LLM Wiki compiler**, **Graph-based RAG**, or **Local-First MCP engines**. A curated index of production-ready repositories on GitHub includes the following options for building a knowledge framework: --- ## 1. The LLM Wiki Paradigm (Compilation-at-Ingest) These tools implement Andrej Karpathy's blueprint—turning an agent into an automated maintainer that synthesizes your incoming documents directly into interlinked Markdown archives. * **`nashsu/llm_wiki`** A dedicated cross-platform desktop application that turns raw source folders into an organized, interlinked Markdown knowledge network. It supports automatic folder watching, two-step chain-of-thought ingestion, and surfaces unexpected relationships across your documents via a dedicated "surprise score" panel. * **`lucasastorian/llmwiki`** A local Python API and web workspace designed to run autonomously via scheduled routines. It exposes your document vault as an **MCP (Model Context Protocol)** server. This allows terminal-based or desktop AI agents to read, write, query, and render conceptual connections inside an interactive graph viewer. * **`nvk/llm-wiki`** An advanced multi-agent project structured explicitly as a plugin for developers utilizing local coding interfaces. It focuses on thesis-driven investigation, automated session logging, and explicitly captures high-signal user feedback loops to prune out hallucinations during the compilation stage. --- ## 2. The GraphRAG Paradigm (Knowledge-Graph Systems) If you want to map structural dependencies and high-level macro themes across thousands of entities without relying on flat Markdown files, these tools offer advanced graph-clustering setups. * **`microsoft/graphrag`** The flagship modular framework for extracting structured data from unstructured text using LLMs. It handles complex cross-document global analysis through map-reduce community reports and localized entity extraction, making it ideal for deep engineering deep-dives over uncurated research sets. * **`noworneverev/graphrag-visualizer`** The perfect companion for local testing. It is a web-based UI that processes your indexing artifacts locally in your terminal, parsing `.parquet` output into beautiful, filterable 2D/3D visual dependency maps. --- ## 3. High-Fidelity Local-First RAG (Zero-Cloud Infrastructures) If your goal is to feed code bases or documentation pipelines directly to your terminal agents with zero external server dependencies, these repositories excel at local deployment. * **`lyonzin/knowledge-rag`** A specialized Local RAG engine built directly as an **MCP server**. It requires **zero API keys and zero external servers**, relying entirely on in-process ONNX FastEmbed vectors and a local ChromaDB instance. It includes 20 format parsers—specifically fine-tuned to extract structural context from markdown alongside raw codebases like **Python, C, C++, TypeScript, and JSON**. * **`infiniflow/ragflow`** A heavy-duty open-source RAG engine that merges deep vector retrieval with explicit agent pipeline building. It relies on precise template-based document chunking to preserve the visual hierarchies of intricate documents instead of blindly fracturing strings. --- Are you looking to hook one of these knowledge bases directly into an active terminal assistant profile to assist with your live coding, or are you aiming to index a static collection of engineering reference documents? # how these wiki or graph solutions compare to Obsidian ? The short answer is that **they are not mutually exclusive—they are designed to be used together.** In the modern AI ecosystem, **Obsidian is the preferred frontend user interface and data storage format (`.md`) for LLM Wikis and GraphRAG systems.** As the current engineering mantra goes: *Obsidian is how you view the brain; the AI agent is how you build and maintain it.* --- ## 1. Traditional Obsidian (Human-Curation Layer) By default, standard Obsidian is a passive cabinet for your thoughts. You write markdown files and manually weave connections using `[[bidirectional links]]`. * **The Native Graph Limitation:** Obsidian’s built-in Graph View is strictly structural, not semantic. It only connects two notes if *you* explicitly typed a link between them. If you have two notes about identical engineering concepts but forgot to link them, they will sit at opposite ends of your graph forever. * **The Maintenance Debt:** As your technical files scale into the thousands, human bookkeeping breaks down. You get "orphan notes" and broken links, and you forget what you wrote six months ago. --- ## 2. LLM Wiki & GraphRAG (The Automated Bookkeeper) When you hook an LLM Wiki loop or a GraphRAG engine into an Obsidian vault, you transition from a static file cabinet to an autonomous, self-improving memory layer. Instead of you doing the linking, an AI agent watches your vault: * **Semantic Association:** GraphRAG engines (like *LightRAG*) look at the actual *meaning* of your text. It extracts entities, definitions, and hidden dependencies, then automatically clusters related concepts together even if you never manually linked them. * **Compilation at Ingest:** When you drop a raw server log, an API documentation file, or a code snippet into an inbox directory, the agent reads it, synthesizes it, updates your main index files, and appends metadata frontmatter. * **Cross-Document Synthesis:** If a new paper or system log contradicts an older note, the LLM Wiki doesn't just create a separate file; it actively rewrites your existing concept pages to flag the contradiction and link the source. --- ## 3. Production Integrations (How to Bridge Them) You don't need to choose between Obsidian's local privacy and an AI knowledge graph. Several open-source tools bridge this gap perfectly using local runners like Ollama: ### 🚀 Neural Composer (Obsidian Plugin) This community plugin brings a local **LightRAG server** directly inside your Obsidian client. * It automatically spins up a local engine in the background when you launch Obsidian. * It maps out your vault as a semantic entity graph (visualized in 2D or 3D) and allows you to chat with your second brain using localized context. ### 🚀 Karpathy LLM Wiki Plugin A native Obsidian plugin built explicitly around Andrej Karpathy's "knowledge compiler" design. * Runs completely local-first with **Ollama** or **LM Studio** so your notes never leave your machine. * It processes incremental updates—meaning when you rewrite a single system file, it only targets and updates the specific interconnected wiki pages tied to that code block instead of wasting compute rebuilding your whole vault. ### 🚀 kytmanov/obsidian-llm-wiki-local (CLI Framework) If you prefer terminal-driven pipelines, this Python tool watches your local directories (`olw watch`). * It auto-commits every AI modification using Git-aware tracking (`[olw]`), giving you a perfect undo history if an agent hallucinates. * It preserves human changes. If you manually tweak a generated markdown file inside Obsidian, the compiler detects your hand-edits and locks them so future automated runs won't overwrite your manual input. ### Summary Verdict Standard Obsidian is a blank notebook. An LLM Wiki/GraphRAG setup turns that notebook into a structured database that maintains itself. For systems engineers, initializing a local loop that reads your source files and writes clean markdown directly into an Obsidian vault gives you the ultimate mix: **100% human-readable files that are perfectly structured for local AI agents to read and execute against.** ---