Model Council Decision Review
Compare multiple models against first-party data before making a decision
- Difficulty
- Easy
- Time to result
- ~days to results
- Steps
- 5
- Confidence
- 98%
Model Council Decision Review sends the same business question and relevant evidence to multiple AI models, then structures their responses into areas of agreement, disagreement, and model-specific insight. The method becomes materially stronger when the models can inspect first-party systems such as a CRM rather than relying only on generic third-party information. A human reviews both the recommendation and its supporting evidence, paying particular attention to concentrated risks, inconsistent assumptions, and distinctive analytical lenses. The output is not a majority vote: consensus offers confidence, disagreement reveals uncertainty, and unique findings expose considerations that another model may have missed. This produces a compact decision brief that is easier to challenge than a single opaque answer.
Origin
Nate Folin described using Perplexity Computer's model council to choose an executive-dinner city from open CRM opportunities on Marketing Against The Grain.
Core principles
- 01Do not trust a single model for consequential work
- 02Ground recommendations in company-specific data
- 03Separate consensus from disagreement and unique findings
- 04Make the reasoning easy for a human to review
How to run it
- 1
Frame the decision
State the specific choice to be made and the practical outcome required. Define any constraints that the models must respect.
Pro tip Ask for a simple review format before analysis begins.
Watch out A vague question will produce conclusions that are difficult to compare.
- 2
Supply first-party evidence
Connect the systems containing the organization's actual customers, opportunities, performance, or operational constraints.
Pro tip Prioritize current CRM and internal data over generic market assumptions.
Watch out Third-party data alone may produce a plausible but irrelevant recommendation.
- 3
Convene multiple models
Have several current models independently analyze the same question and evidence.
Pro tip Use models with different strengths to increase analytical diversity.
Watch out Do not treat repeated wording as independent corroboration if the models share the same unsupported assumption.
- 4
Map agreement and disagreement
Organize the output into consensus findings, disputed findings, and insights unique to individual models.
Pro tip Request the evidence supporting every important conclusion.
Watch out Consensus can still be wrong when all models inherit poor data.
- 5
Review and decide
Challenge the logic, correct weak assumptions, and select or revise the recommendation.
Pro tip Record why the final choice differs from the council when overriding it.
Watch out Do not automate execution until the decision logic has been inspected.
In the wild
A RevOps lead asks several models to inspect open CRM opportunities and recommend a city for an executive dinner. The council identifies common target criteria, flags that Chicago's pipeline is concentrated under one representative, and presents evidence for each candidate city.
→ The team chooses a city using account quality and concentration risk rather than generic market size.
Common mistakes
Using generic data only
A model may recommend a theoretically attractive market that has little relationship to the company's actual pipeline or customers.
Treating consensus as proof
Agreement is a confidence signal, not a substitute for checking evidence and assumptions.
Ignoring unique findings
Model-specific observations can reveal risks or opportunities concealed by the consensus summary.
Is it for you?
Best for
Teams making thoughtful, data-dependent decisions such as choosing markets, audiences, campaigns, or investments.
Not ideal for
It is unnecessary for low-risk mechanical tasks with a single objectively verifiable answer.
From the transcript
“But I also uh like to not trust one model.”
“You can have this in any format you'd like, broken down by where the models agree, disagree, and unique findings for each of those.”
“I think that's super important because if you just go third-party data or what what is the data say, it's it's less relevant to your…”
From the episode
How 1 Human + AI Replaced a 15-Person RevOps Team