Five-Awareness AI Agent Scorecard
Evaluate AI agents across compute, data, coordination, safety, and user value
- Difficulty
- Moderate
- Time to result
- ~weeks to results
- Steps
- 5
- Confidence
- 96%
The scorecard evaluates an AI agent through five connected forms of awareness. Compute awareness asks whether the agent understands and limits the resources it consumes. Data awareness checks whether it can locate and use the appropriate model and information source. Agent awareness examines coordination with humans and other automated agents. Safety awareness requires sandboxing, output verification, and an honest estimate of result quality. User awareness closes the loop by measuring whether people use the output and whether it creates value in the real world. A weakness in any dimension can prevent an impressive demonstration from becoming dependable production software. Teams apply the scorecard repeatedly as the agent evolves, converting broad concerns about readiness into concrete tests and improvement priorities.
Origin
Extracted from Marketing Against The Grain while Kipp Bodnar summarized Sequoia Capital's assessment of the capabilities AI agents still need to reach their full potential.
Core principles
- 01Agent capability alone does not guarantee useful deployment
- 02Resource consumption must be visible and controlled
- 03Agents need access to the right models and data
- 04Reliable agents coordinate with humans and other agents
- 05Outputs require safety checks and quality controls
- 06Real-world user feedback should improve future performance
How to run it
- 1
Assess Compute Awareness
Measure the resources consumed by each agent run and determine whether the agent can select an appropriately efficient path. Establish cost and latency limits for representative tasks.
Pro tip Compare resource consumption with the measurable value of the completed task.
Watch out A capable agent can still be commercially unusable if its compute costs are unpredictable.
- 2
Assess Data Awareness
Test whether the agent finds the correct model and data source for the requested job. Include ambiguous tasks where several sources appear relevant.
Pro tip Record which sources and models were selected so failures can be diagnosed.
Watch out Fluent output can conceal that the agent used irrelevant or incomplete data.
- 3
Assess Agent Awareness
Evaluate how the agent delegates work, communicates with other agents, and hands decisions back to humans. Check whether responsibilities and stopping conditions remain clear.
Pro tip Use explicit task contracts between every human or automated participant.
Watch out Poor coordination can compound small errors across an agent chain.
- 4
Assess Safety Awareness
Sandbox risky actions and require the agent to check its own outputs before release. Compare its confidence with independently measured quality.
Pro tip Separate output generation from approval for consequential actions.
Watch out An agent may express irrational confidence even when its answer is wrong.
- 5
Assess User Awareness
Measure whether users adopt the output and whether it improves the intended business or personal outcome. Feed this performance evidence into subsequent development.
Pro tip Track downstream behavior rather than relying only on user satisfaction surveys.
Watch out Technically impressive output is not valuable if nobody uses it.
In the wild
A marketing team tests an agent that researches an audience, drafts a campaign, and recommends a budget. They measure run cost, inspect its research sources, review handoffs between research and writing agents, require human approval before publishing, and track campaign conversion data. Weak source selection emerges as the primary failure, so the team improves data retrieval before expanding autonomy.
→ The team isolates a specific readiness gap and improves reliability without prematurely deploying a fully autonomous system.
A support organization scores an autonomous resolution agent across all five dimensions. The agent is inexpensive and uses the correct documentation, but it cannot recognize when a human specialist should take over. The company adds escalation rules and tests them against difficult historical cases.
→ Agent coordination improves, reducing unsafe automated resolutions while preserving efficiency.
Common mistakes
Judging Only Answer Quality
A good answer in a demonstration says nothing about compute efficiency, coordination, safety, or sustained user value. Evaluate all five dimensions.
Treating Confidence as Verification
An agent's confident language is not evidence that its output is correct. Use independent checks and calibrated quality measurements.
Ignoring Downstream Outcomes
Teams may optimize agent output while failing to measure whether users adopt it or achieve better results. Close the loop with real performance data.
Is it for you?
Best for
It is best for teams evaluating agent prototypes, autonomous workflows, and AI-enabled products before broader deployment.
Not ideal for
It is not ideal for assessing simple deterministic automation that does not reason, coordinate, or generate open-ended outputs.
From the transcript
“to reach their full potential AI agents have to become compute aware”
“Needs to be agent aware, which is like, basically how can it communicate with other agents, both human and robots to actually have the right…”
“And then they're saying it needs to be user aware, and I interpret user aware as like, is there a good feedback between the user…”
From the episode
ChatGPT Releases Code Interpreter: Your Personal Data Scientist (#115)