Example-to-Rubric Expertise Encoder
Turn repeated expert edits into an on-demand feedback tool.
- Difficulty
- Easy
- Time to result
- ~days to results
- Steps
- 7
- Confidence
- 99%
The Example-to-Rubric Expertise Encoder extracts standards that an expert applies intuitively but struggles to explain. The expert first collects paired examples containing an original draft and their revision. An AI system compares the columns and identifies editing patterns that recur across examples. Those patterns are converted into a small set of criteria, each with explicit descriptions of passing and failing work. The criteria then become instructions for a narrowly scoped feedback tool, such as an executive-email editor. Team members submit work, receive a diagnosis and suggested improvements, revise it, and retain accountability for the final result. This scales access to expert thinking while encouraging repeated practice instead of outsourcing the work itself.
Origin
Hillary Gridley created an Executive Editor after repeatedly revising team emails intended to secure executive approval, then used paired drafts and revisions to extract her standards.
Core principles
- 01Use expert revisions as evidence of tacit standards.
- 02Let AI identify repeated patterns across before-and-after examples.
- 03Translate patterns into explicit pass-fail criteria.
- 04Build tools around one narrow purpose.
- 05Use feedback to improve judgment rather than replace it.
How to run it
- 1
Choose one narrow job
Select a recurring deliverable with a clear audience and outcome. Define the tool's purpose precisely rather than attempting to clone the expert generally.
- 2
Collect paired examples
Assemble original drafts beside the expert's revisions. Use enough representative examples to expose repeated judgment patterns.
- 3
Extract recurring patterns
Ask the AI to compare the original and revised columns and identify edits the expert makes repeatedly. Review its observations for accuracy.
- 4
Create criteria
Turn the validated patterns into a concise rubric. Define in plain English what passing and failing each criterion looks like.
- 5
Encode the feedback process
Write instructions that evaluate work against every criterion, report pass or fail, explain weaknesses, and suggest revisions. Install them in a focused GPT, skill, agent, or equivalent tool.
- 6
Validate on held-out work
Test the tool using examples that were not part of the extraction set. Compare its evaluation with the expert's own judgment and refine the rubric.
- 7
Use feedback iteratively
Have team members run a draft through the tool, revise it, and repeat before requesting human review. Position the process as development rather than punishment.
In the wild
Gridley placed team members' draft executive emails in one column and her revisions in another. ChatGPT identified recurring changes such as leading with the message, making the request actionable, using the right tone, and replacing ambiguity with clarity. She converted those patterns into pass-fail criteria and installed the resulting instructions in a custom GPT that reviewed new emails and suggested improvements.
→ Team members gained on-demand access to Gridley's communication standards without requiring her to edit every internal email.
Common mistakes
Building a generic expert clone
Uploading miscellaneous documents about an expert produces an unfocused assistant rather than a reliable tool for one outcome.
Skipping held-out validation
A rubric can accurately describe training examples while failing on new or borderline work.
Treating feedback as approval
Passing an automated rubric does not remove the author's responsibility to check context, judgment, and consequences.
Is it for you?
Best for
It is best for managers who repeatedly make similar corrections to recurring team deliverables.
Not ideal for
It is not ideal for subjective work where the expert has no stable standard or representative examples.
From the transcript
“My favorite one is like giving it examples.”
“Like what are what are you noticing about the the edits that I make over and over and over again? And can you help me…”
“And then you have a rubric, and you can run anything against a rubric.”
From the episode
If Your Team Is Producing AI Slop, Here's How To Fix it