LLM Wiki Pipeline
Turn raw source files into persistent, AI-readable knowledge.
- Difficulty
- Easy
- Time to result
- ~days to results
- Steps
- 6
- Confidence
- 98%
The LLM Wiki Pipeline separates captured material from structured knowledge. The user drops source files into a raw folder, then an AI assistant reads those sources, extracts useful concepts, and creates or updates focused wiki pages. Those pages become persistent memory that the assistant can consult during later conversations. Instead of repeatedly supplying the same background, the user grows a reusable knowledge base whose value increases with each ingestion cycle. The mechanism is deliberately simple: capture first, process second, preserve the synthesis, and expose it to the assistant whenever work requires relevant context.
Origin
The host attributes the method to Andre Karpathy, whose April post about building an LLM wiki helped popularize AI second brains. Extracted from Marketing Against The Grain.
Core principles
- 01Store source material before attempting synthesis.
- 02Convert scattered information into structured topic pages.
- 03Give the AI persistent access to the resulting wiki.
- 04Let accumulated knowledge improve future conversations.
How to run it
- 1
Create the raw folder
Establish a simple location where unprocessed source material can be collected.
Pro tip Use plain files and a stable folder structure so multiple AI tools can access the material.
Watch out Do not treat the raw folder itself as the finished knowledge base.
- 2
Capture relevant sources
Drop documents, notes, transcripts, and other useful information into the raw folder.
Pro tip Capture material close to when it is encountered to reduce forgotten context.
Watch out indiscriminate capture can bury useful information beneath noise.
- 3
Process the material
Have the AI read the collected sources and identify concepts, facts, decisions, and relationships worth preserving.
Pro tip Give the assistant explicit criteria for what counts as valuable intelligence.
Watch out Without selection criteria, the assistant may produce shallow summaries rather than reusable knowledge.
- 4
Build topic pages
Write the extracted intelligence into singular wiki files organized around durable concepts or projects.
Pro tip Update an existing page when a concept recurs instead of creating unnecessary duplicates.
Watch out Fragmenting one subject across many files weakens retrieval.
- 5
Connect the assistant
Ensure the AI assistant can consult the wiki whenever it answers questions or performs work.
Pro tip Require answers to trace important claims back to their stored sources.
Watch out A wiki that is not available in the working context provides no persistent advantage.
- 6
Repeat the pipeline
Add new sources and refresh the wiki so the assistant becomes more informed over time.
Pro tip Run ingestion on a regular cadence tied to your actual workflow.
Watch out A static wiki eventually becomes stale.
In the wild
The host describes an AI noticing that the concept of an internal AI SDR keeps recurring. It creates a dedicated wiki page and continues storing what the organization learns about using an agent for prospecting and booking meetings.
→ A recurring business concept becomes persistent, cumulative knowledge rather than being rediscovered in separate conversations.
A founder feeds pricing experiments, customer feedback, and meeting notes into the raw folder. The AI consolidates recurring lessons into a pricing-strategy wiki page that it can consult during later planning.
→ Future pricing decisions begin with accumulated evidence instead of a blank context window.
Common mistakes
Calling a raw folder a wiki
A pile of searchable files is storage, not structured intelligence. The sources still need to be processed into coherent topic pages.
Creating duplicate topic pages
Repeatedly creating new files for the same concept fragments the accumulated knowledge and makes retrieval less reliable.
Failing to refresh the wiki
Without recurring ingestion, old conclusions remain in place even when newer evidence should modify them.
Is it for you?
Best for
It is best for people who repeatedly research, decide, or create within a stable set of subjects.
Not ideal for
It is not ideal for users who cannot safely store their source material or who only need isolated one-off answers.
From the transcript
“you would have a simple folder where you could drop sources of information into that folder. The AI would read it all, and it would…”
“The AI would process that information and would create a wiki for itself, an LLM wiki, kind of extrapolating all of the intelligence or knowledge…”
“And anytime he talked to his AI assistant, it has access to that knowledge.”
From the episode
If You Use AI for Work, You Need a Second Brain