Restrict-and-Refine Prompt Loop
Reduce inconsistent AI output by tightening explicit constraints iteratively
- Difficulty
- Moderate
- Time to result
- ~weeks to results
- Steps
- 6
- Confidence
- 96%
Treat prompt development as an iterative control problem. Run the same task repeatedly and compare where the model changes format, scope, facts, or behavior. Convert each unacceptable deviation into an explicit instruction, boundary, or response schema, then retest. For factual content, require cited sources or another evidence condition rather than trusting fluent output. The goal is not necessarily identical wording; it is variation that remains inside operationally acceptable limits. Because a generative model is designed to produce creative answers, broad intent alone will not guarantee consistency. Stable workflows emerge from testing, observing failure modes, tightening constraints, and repeating the cycle.
Origin
Extracted from Marketing Against The Grain while Kieran described refining a custom GPT used in an AI-generated email course.
Core principles
- 01Broad prompts permit dramatically different valid responses
- 02Generative models contain creativity rather than deterministic retrieval
- 03Consistency improves when acceptable behavior and prohibited deviations are explicit
- 04Prompt quality develops through repeated testing and refinement
- 05Source requirements can constrain high-risk factual outputs
How to run it
- 1
Establish a Baseline
Run the current prompt several times or have several people run it independently.
Pro tip Keep inputs constant so output differences are attributable to the model.
Watch out A single successful run does not demonstrate consistency.
- 2
Classify Deviations
Mark differences in structure, factuality, scope, tone, and workflow behavior.
Pro tip Separate harmless wording variation from variation that breaks the use case.
Watch out Do not overconstrain differences that have no operational consequence.
- 3
Specify the Contract
State required fields, sequence, boundaries, and completion conditions explicitly.
Pro tip Use enumerated rules and examples for recurring failure modes.
Watch out Vague requests such as 'be consistent' do not define acceptable output.
- 4
Add Negative Constraints
Tell the model what it must not omit, invent, reorder, or deviate from.
Pro tip Write each restriction in response to an observed failure.
Watch out A large collection of conflicting constraints can make performance worse.
- 5
Require Evidence Where Needed
For quotes or factual claims, require a source condition that the output must satisfy.
Pro tip Reject unsupported items rather than asking the model to fill every slot.
Watch out A source requirement still needs external verification in high-stakes contexts.
- 6
Retest and Refine
Repeat the prompt across multiple runs and tighten only the remaining consequential deviations.
Pro tip Maintain a small regression set of inputs that previously caused failures.
In the wild
Kieran builds a long custom GPT with documentation and a menu of homework options. The same selection sometimes produces dramatically different responses, so he repeatedly tightens the prompt and tells the model not to deviate from the intended workflow.
→ The assistant becomes more predictable for course participants.
When asking AI for interview quotes, the prompt can require that only quotes with a cited source be returned. Unsupported material is excluded rather than presented as authentic.
→ The restriction lowers the risk of fabricated quotations entering interview preparation.
Common mistakes
Judging One Run
The builder assumes the workflow is stable because one output happened to match expectations.
Leaving Success Undefined
The prompt describes a broad goal without defining the required structure or prohibited deviations.
Trusting Citation-Shaped Text
The user treats the model's mention of a source as proof without checking that the source exists and supports the claim.
Is it for you?
Best for
It is best for repeated AI workflows, custom GPTs, homework assistants, and structured content generation.
Not ideal for
It is not ideal when creative variation is the desired output or when deterministic software should replace a language model entirely.
From the transcript
“And each time you click it, sometimes it will give you like these kind of dramatically different responses for some reason.”
“And so you have to continue to refine the prompt to make it really strict and let it know that it shouldn't deviate away from…”
“So the way you could restrict it on quotes is say, only give me the ones where there's a source sided, right?”
From the episode
Answering Your Burning Questions On AI & Marketing