Build-Test-Refine Skill Loop
Turn expert workflows into reusable AI skills through repeated testing and refinement.
- Difficulty
- Moderate
- Time to result
- ~weeks to results
- Steps
- 6
- Confidence
- 98%
The Build-Test-Refine Skill Loop converts a recurring task into a portable AI operating procedure. First, define the outcome and encode the process, constraints, reference material, and quality bar in a skill.md file. Install the skill, invoke it on a representative task, and inspect the resulting work rather than assuming the written instructions are sufficient. Each failure reveals missing context: the agent may need clearer decision rules, better examples, a style guide, proprietary data, or tighter verification requirements. Add that information and run the task again. Repeating this cycle gradually produces a stable system that can deliver useful first drafts or production-ready work across compatible agents. The mechanism is cumulative: every correction improves future executions instead of disappearing inside a single conversation.
Origin
Extracted from Marketing Against The Grain during a demonstration of repeatedly developing and testing an advertising-optimizer skill in Manus.
Core principles
- 01Treat a skill as a repeatable system rather than a conversational prompt.
- 02Teach the agent how to complete a task as explicitly as you would teach a person.
- 03Test skills on real work instead of judging their instructions in isolation.
- 04Use weak outputs as evidence for what context or instructions are missing.
- 05Add proprietary examples and performance data to improve relevance and accuracy.
How to run it
- 1
Choose a Repeatable Outcome
Select a task that occurs often enough to justify systematizing it. Define the deliverable and the standard that would make the output genuinely useful.
Pro tip Start with a narrow task whose output can be evaluated against concrete criteria.
Watch out A vague goal such as “improve my marketing” will produce a vague and difficult-to-test skill.
- 2
Encode the Workflow
Create a skill.md file that explains the process, required inputs, decision rules, tools, and expected output. Include the knowledge an experienced person would need to perform the task consistently.
Pro tip Ask the agent to draft the first version, then replace generic instructions with your own expertise.
Watch out Do not mistake a short prompt for a complete operating system.
- 3
Install and Invoke the Skill
Add the skill to the agent's skill library and call it explicitly by name while assigning a representative task. Keep the test close to the real environment in which the skill will be used.
Pro tip Use the same invocation wording during early tests so changes in output reflect changes to the skill.
Watch out If the agent does not select the skill reliably, name it explicitly in the request.
- 4
Audit the Output
Review the result for unsupported claims, weak reasoning, poor style, missing steps, and unusable recommendations. Treat each defect as diagnostic evidence about the skill's missing context.
Pro tip Record failures as specific requirements rather than merely saying the result is bad.
Watch out A polished presentation can conceal fabricated numbers or faulty assumptions.
- 5
Add Missing Context
Strengthen the skill with research, examples, style guides, historical results, and platform-specific instructions. Supply real internal data whenever the agent would otherwise have to estimate or invent values.
Pro tip Prioritize context that changes decisions, not background material that merely adds length.
Watch out Without real performance data, apparently precise recommendations may still be fabricated.
- 6
Retest Until Usable
Run the revised skill again and compare the new output with the previous result and the target quality bar. Continue until it consistently produces work worth using, testing, or handing to another agent.
Pro tip Preserve strong instructions while changing one major weakness at a time.
Watch out Iteration can consume substantial credits when tasks involve research, premium data, or image generation.
In the wild
A host built a skill to analyze product and competitor data, identify lower-competition advertising channels, recommend placements, and design matching creative. Early outputs were poor, so he repeatedly installed, ran, and revised the skill. He added platform instructions, research on a favorite advertising copywriter, and a detailed HubSpot style guide. The resulting ads became credible first drafts, while the exercise also exposed the need to add historical advertising performance data before trusting the budget math.
→ The skill progressed from weak output to ads close enough to consider running and testing, while clearly identifying the remaining data gap.
Common mistakes
Stopping After the First Draft
The first version of a skill rarely contains enough context to reproduce expert work. Treating it as finished preserves the very inconsistencies the skill is meant to eliminate.
Trusting Invented Numbers
An agent may manufacture plausible figures when historical data is absent. Recommendations should not be trusted until relevant real-world inputs and verification rules are supplied.
Adding Instructions Without Testing
A long skill file is not evidence that the system works. The instructions must be exercised on representative tasks and judged by the resulting output.
Is it for you?
Best for
It is best for recurring knowledge-work tasks where quality depends on detailed process knowledge, examples, or business context.
Not ideal for
It is not ideal for trivial one-off requests where creating and testing a reusable skill would cost more time than completing the task manually.
From the transcript
“In the same way you would teach a human how to do a skill, you kind of teach the AI exactly how to complete a…”
“And every time I do that, Kieran, I basically install the skill and add the skill and I run it and test it. And I…”
“What context is so important that one thing I haven't done is I haven't uploaded any of our historical our ads advertising data. and real…”
From the episode
Meta’s AI Agent is Better Than OpenClaw (Manus AI Demo)