LLM Wiki Ingestion Loop
Capture raw knowledge, let an LLM organize it, and retrieve connected insights
- Difficulty
- Easy
- Time to result
- ~days to results
- Steps
- 6
- Confidence
- 99%
The loop separates effortless capture from structured processing. Articles, posts, and video transcripts enter a RAW folder through a browser clipper or manual entry, preserving the source before the user must decide how it fits. An LLM operating over the Obsidian vault then follows persistent instructions in AGENTS.md: read each source, detect duplicates, create or update source records, synthesize topic and entity pages, and add links between related ideas. The user validates this process manually, then schedules it to run periodically so capture can continue throughout the day without creating an organizational burden. The resulting wiki becomes a curated context layer that the model can search, synthesize, and cite when answering questions or completing work.
Origin
Adapted by Matt Wolf from Andre Karpathy's LLM wiki method and demonstrated on Marketing Against The Grain.
Core principles
- 01Capture useful material before it disappears from attention
- 02Keep unprocessed inputs separate from organized knowledge
- 03Give the LLM explicit, reusable processing instructions
- 04Connect related ideas instead of preserving isolated documents
- 05Automate recurring ingestion after validating the workflow manually
How to run it
- 1
Create the markdown vault
Install Obsidian and create a vault in a known local folder. Use that folder as the durable store for sources, topics, entities, and generated indexes.
Pro tip Choose a folder that Codex or another agentic IDE can open directly as a project.
Watch out Do not place sensitive material in the vault unless you understand which models and connectors can access it.
- 2
Capture into a raw inbox
Send articles, posts, transcripts, and notes into a dedicated RAW folder. Favor one-click clipping so deciding how to organize an item never blocks capture.
Pro tip Use the Obsidian Web Clipper to preserve page content and YouTube transcripts.
Watch out Clipping indiscriminately can create a large archive with little practical value.
- 3
Define the ingestion contract
Write persistent AGENTS.md instructions that tell the LLM how to read, validate, classify, synthesize, and link each source. Include a duplicate-source check before generating files.
Pro tip Specify exact folders and page types so repeated runs produce a stable structure.
Watch out Vague instructions will produce inconsistent categories and links.
- 4
Process and interlink sources
Ask the LLM to process every unhandled file in RAW. It should update source records, topic pages, entity pages, comparisons, and the main index as appropriate.
Pro tip Review several manual runs before allowing unattended processing.
Watch out Do not assume every automatically inferred connection is meaningful or correct.
- 5
Automate the loop
Schedule a recurring prompt that checks for unprocessed RAW files and handles them. Run it at a low-interruption time such as overnight.
Pro tip Make the scheduled prompt conditional so it does nothing when the inbox is empty.
Watch out An unattended agent can propagate a bad instruction across many files.
- 6
Retrieve grounded answers
Ask questions explicitly based on material saved in the wiki. Require the response to identify the internal sources that support its recommendations.
Pro tip Frame requests around a concrete project or decision rather than asking for a generic summary.
Watch out The wiki reflects what you captured, so omissions and source bias will shape its answers.
In the wild
Matt saved seven sources related to answer engine optimization. The system linked them under an AEO topic, then Codex used those sources to recommend how Future Tools could become more citable in ChatGPT and Claude responses, including crawler-access guidance.
→ A scattered set of saved sources became a concise, source-grounded strategy for a live website.
Matt used the Obsidian Web Clipper to capture a Futurepedia video transcript into RAW. He then asked Codex to process the folder according to the vault's AGENTS.md ingestion instructions.
→ The transcript entered the structured wiki without manual filing or cross-linking.
Common mistakes
Organizing every item during capture
Requiring immediate classification adds friction and discourages saving. Capture first into RAW, then let a consistent processing routine organize the material.
Skipping duplicate detection
The same source may be saved more than once, creating redundant pages and distorted topic counts. Validate source URLs before processing.
Automating before reviewing outputs
A weak ingestion prompt can repeatedly create poor links or malformed pages. Test the instructions manually before scheduling unattended runs.
Is it for you?
Best for
It is best for researchers, marketers, founders, and heavy content consumers who repeatedly need to reuse what they learn.
Not ideal for
It is not ideal for people who save little reference material or cannot maintain basic control over sensitive local files.
From the transcript
“This is my inbox. This is where the stuff gets saved.”
“process all the files in the raw folder.”
“If there are any unprocessed files in the raw directory, please process them now.”
From the episode
This AI Second Brain Remembers Everything I Save (Codex)