Adversarial AI Reasoning Checklist
Stress-test an argument by making AI strengthen and attack it.
- Difficulty
- Easy
- Time to result
- ~days to results
- Steps
- 5
- Confidence
- 98%
The Adversarial AI Reasoning Checklist uses a reasoning model to examine an argument from opposing directions. First, state the proposed conclusion, supporting evidence, and intended decision. Ask the model to steelman the argument so that its strongest coherent form is visible. Then request the strongest bullish interpretation before switching sides and asking the model to poke holes in the logic. Convert the criticism into a list of unsupported assumptions, missing evidence, alternative explanations, and failure conditions. Revise the argument or the decision criteria in response, then repeat the challenge against the revised version. This method is especially useful for people who instinctively run ahead to solutions, because it inserts an explicit opposition phase before action without depending on another person being available to disagree.
Origin
Extracted from Marketing Against The Grain as the hosts described using reasoning models to challenge strategic thinking.
Core principles
- 01Strong decisions require examining both the best supporting case and the strongest objections.
- 02AI can supply structured opposition when people naturally rush toward solutions.
- 03Evidence and assumptions should be separated before committing to a conclusion.
- 04The purpose is to improve judgment, not manufacture certainty.
How to run it
- 1
Frame the argument
Write the conclusion, evidence, assumptions, and decision under consideration in explicit terms.
Pro tip Distinguish observed facts from interpretations.
Watch out A vague claim will produce vague criticism.
- 2
Steelman the case
Ask the model to reconstruct the strongest defensible version of the argument.
Pro tip Require it to preserve the actual objective and constraints.
Watch out Do not let the steelman quietly change the original claim.
- 3
Build the bullish case
Ask for the most compelling reasons the argument could be right and the conditions under which it succeeds.
Pro tip Request causal mechanisms rather than motivational language.
Watch out This phase can amplify confirmation bias if used alone.
- 4
Poke holes
Ask the model to attack assumptions, evidence quality, causal logic, and ignored alternatives.
Pro tip Request the objections in order of potential impact.
Watch out Do not dismiss uncomfortable objections merely because they are hypothetical.
- 5
Revise and rerun
Strengthen, narrow, or reject the argument based on the critique, then test the revised version again.
Pro tip Stop when successive rounds cease to reveal decision-relevant weaknesses.
Watch out Avoid endless critique when a cheap, reversible test can resolve the uncertainty.
In the wild
A leader proposes that one customer segment should receive increased investment. The model first builds the strongest case from historical growth and customer evidence, then attacks attribution, selection bias, implementation capacity, and contradictory retention data. The leader revises the hypothesis into a smaller experiment with explicit failure criteria.
→ A broad conviction becomes a testable and risk-bounded decision.
Kieran described asking reasoning models to steelman arguments, provide a bullish case, and poke holes in his thinking because he tends to run ahead toward solutions.
→ The model supplies an explicit challenge phase that his normal problem-solving style can omit.
Common mistakes
Requesting only supportive reasoning
A steelman or bullish case without an attack phase can make an existing bias feel more intellectually justified.
Treating every objection equally
Minor edge cases can distract from the few criticisms capable of changing the decision.
Critiquing without revising
The exercise has little value if identified weaknesses are not converted into better evidence, narrower claims, or new tests.
Is it for you?
Best for
Leaders, strategists, and creators reviewing consequential ideas before committing resources or giving advice.
Not ideal for
Decisions that require unavailable facts or situations where prolonged analysis would cost more than a reversible experiment.
From the transcript
“you can basically ask it to steel man an argument. You can ask it to give you the bully case of that argument.”
“What I ask it to do is like poke holes in my thinking here”
“I run ahead to solve things and don't spend time trying to like poke holes in my own argument.”
From the episode
GROK 3 vs GPT-4: The AI War Just Got Real [First Look]