Iterative Skill-Building Loop
Build concise AI skills through questioning, testing, and feedback
- Difficulty
- Moderate
- Time to result
- ~weeks to results
- Steps
- 6
- Confidence
- 96%
This loop treats an AI skill as an evolving operating document. Begin with a clear job to be done and ask Codex or Claude Code to interview you about requirements, constraints, examples, and desired outcomes. Let the agent draft the instruction file, then test the skill manually on representative work rather than assuming the generated text is correct. Review the behavior, provide concrete feedback, and update the skill from what the test revealed. A separate editing pass should remove repetitive instructions, vague AI language, and needless complexity while preserving capability. Continue alternating execution and revision so the skill captures actual operating knowledge. The resulting skill accelerates future work but still supports human intervention whenever creative taste, factual judgment, or strategic choice is required.
Origin
On Marketing Against The Grain, Peter Yang opened his podcast-preparation skill and explained that Codex had written it after interviewing him. He also demonstrated a separate skill editor that compresses generated instructions.
Core principles
- 01A skill is a reusable instruction file, not a magical one-shot automation
- 02Let the agent interview you before it drafts the skill
- 03Test behavior on real work instead of trusting the text alone
- 04Human taste remains part of the operating loop
- 05Concise instructions reduce repetition and AI-generated clutter
How to run it
- 1
Define the Job
Describe the recurring outcome you want the skill to produce and where it fits in the broader workflow. Include what a successful result should enable next.
Pro tip Frame the skill around one coherent responsibility.
Watch out A skill covering unrelated jobs will be harder to test and maintain.
- 2
Invite Questions
Ask the agent to identify missing information and interview you before drafting. Answer with constraints, preferences, examples, and failure conditions.
Pro tip Provide examples of both strong and weak outputs when available.
Watch out If the outcome remains vague, additional instructions will not reliably improve it.
- 3
Generate the Initial Skill
Have Codex or Claude Code turn the conversation into a reusable instruction file. Review the result for obvious omissions or unsafe actions.
Pro tip Let the agent handle the initial structure instead of manually formatting everything.
Watch out Generated complexity can disguise unclear or repetitive instructions.
- 4
Test It Manually
Run the skill on a real, representative task and inspect how it gathers context, makes decisions, and produces output. Treat execution as the primary test of the instructions.
Pro tip Observe where you need to intervene, because those interventions reveal missing rules.
Watch out Reading the skill alone cannot prove that it behaves correctly.
- 5
Revise From Feedback
Tell the agent specifically what worked, what failed, and what should change. Update the skill using evidence from the run rather than hypothetical preferences.
Pro tip Ask the agent to incorporate lessons from the current conversation.
Watch out Broad feedback such as 'make it better' produces unstable revisions.
- 6
Compress the Instructions
Run an editing pass that removes repetition, AI clichés, and unnecessary detail while retaining behavioral power. Keep the skill easy for both humans and agents to inspect.
Pro tip Aim for the least text that consistently produces the required behavior.
Watch out Compression should not delete constraints that prevent known failures.
In the wild
Peter asked Codex to create a skill for researching podcast guests. The agent asked questions, generated the text file, and then ran it alongside a thumbnail, title, and copy skill to research prior YouTube appearances and identify interview angles.
→ A previously manual research process became a reusable starting point for podcast preparation.
To control instruction sprawl, Peter created a second skill that reviews every new skill, tries to reduce it to roughly one page, removes repetitive directions, and cuts AI-generated filler.
→ The reusable instructions remain tighter and easier to maintain.
Common mistakes
Never Testing the Skill
A plausible-looking instruction file may still fail in execution, so every skill should be run on representative work.
Allowing Instruction Sprawl
Repeated AI revisions can accumulate redundant rules and vague language that obscure the skill's purpose.
Removing Human Taste
Skills can prepare and transform work, but creative selection and judgment still require deliberate human participation.
Is it for you?
Best for
It is best for recurring work where examples, preferences, and feedback can progressively improve output quality.
Not ideal for
It is not ideal for undefined outcomes or tasks whose success cannot be inspected by a human.
From the transcript
“And and a skill is pretty much just a text file.”
“The process of building your skill is like you have an idea”
“And then the important thing is after you build it, you want to run through the skill manually.”
From the episode
Automate Boring Tasks With Codex & Claude Code in X Minutes