Best-Fit Model Routing
Route each task to the model best suited to its requirements.
- Difficulty
- Advanced
- Time to result
- ~months to results
- Steps
- 5
- Confidence
- 91%
Best-fit model routing treats models as interchangeable specialists rather than selecting one universal provider. Each incoming task is classified by requirements such as capability, privacy, latency, cost, context length, and acceptable risk. A routing layer then sends it to the model that performs best under those constraints, with fallbacks when confidence or availability is inadequate. Performance is monitored continuously because the best choice can change as models, prices, and workloads evolve. The architecture reduces dependence on one vendor, creates room for open-source models, and lets organizations use expensive frontier systems only where their advantage matters. Its output is a governed routing policy rather than an informal collection of model calls.
Origin
Extracted from Marketing Against The Grain during a discussion of Apple using its own model for some tasks and ChatGPT for better-fitting use cases.
Core principles
- 01No single model is best for every task.
- 02Task requirements should determine model selection.
- 03Routing preserves access to specialized strengths.
- 04Open models can increase control and deployment flexibility.
- 05Model choice should remain replaceable as capabilities change.
How to run it
- 1
Classify the workload
Group requests by the capabilities, sensitivity, latency, context, and reliability they require.
Pro tip Use real production task categories rather than broad labels such as writing or analysis.
Watch out Poor classification makes sophisticated routing ineffective.
- 2
Benchmark candidate models
Evaluate models on representative examples for quality, cost, latency, privacy, and failure behavior.
Pro tip Include difficult and ambiguous cases, not only average requests.
Watch out Vendor benchmarks may not reflect your workload.
- 3
Define routing rules
Translate benchmark results and operational constraints into explicit task-to-model policies.
Pro tip Keep rules inspectable before adopting learned routing.
Watch out Opaque routing makes regressions difficult to diagnose.
- 4
Add fallbacks and escalation
Specify what happens when a model is unavailable, produces low-confidence output, or violates a constraint.
Pro tip Escalate only the uncertain portion to a more capable model.
Watch out A fallback that shares the same failure mode provides little resilience.
- 5
Monitor and recalibrate
Track outcomes and periodically rerun evaluations as models, prices, and user behavior change.
Pro tip Version routing policies alongside evaluation results.
Watch out A once-optimal route can become expensive or inferior quickly.
In the wild
A device uses Apple's own model for use cases suited to local or platform-specific capabilities and calls ChatGPT when that model is a better fit. The user experiences one assistant while routing happens underneath.
→ Different models contribute their strengths behind a unified interface.
An enterprise sends routine private classifications to a locally hosted open model, directs difficult reasoning tasks to a frontier model, and escalates only low-confidence cases.
→ The system balances privacy, cost, and quality without relying on one model.
Common mistakes
Choosing one model for convenience
Operational simplicity can conceal major differences in task quality, cost, privacy, and latency.
Routing without workload evaluations
Model reputation is not a substitute for performance on representative production inputs.
Leaving routes permanently fixed
Rapid model and pricing changes require periodic reevaluation of routing decisions.
Is it for you?
Best for
It is best for products and enterprises running diverse AI tasks with different quality, privacy, latency, or cost requirements.
Not ideal for
It is not ideal for very small deployments where routing complexity exceeds the value of specialization.
From the transcript
“when ChatGPT has better use cases, when the use case fit better fits ChatGPT, it will use ChatGPT. When it fits Apple, it will fit…”
“I do think in the future, companies will actually have software that will allow them to switch between models for the best use case because…”
From the episode
Apple Intelligence Can Outsmart 99% Of Marketers (WWDC Recap)