Generalist-to-Specialist Model Routing
Route each AI task to the smallest model suited to its domain
- Difficulty
- Advanced
- Time to result
- ~months to results
- Steps
- 6
- Confidence
- 88%
Treat AI models like levels in a professional referral system. A general model receives broad or unfamiliar requests because it can operate across many domains, while specialist models handle recurring tasks that demand narrow knowledge, predictable behavior, lower cost, or better latency. The system first classifies each request, then routes it to the least costly model capable of meeting its quality threshold. Representative evaluations verify that specialist efficiency does not sacrifice required accuracy. Because hosted models can change without transparent notice, each route also needs behavioral monitoring, version awareness, and a tested fallback. The output is an adaptable model portfolio rather than a product whose entire experience depends on one provider and one opaque model configuration.
Origin
Ekenstam compared general AI models to general physicians who refer patients to specialists, arguing that domain-specific models can perform narrow product tasks without the overhead of the largest cloud models. Extracted from Marketing Against The Grain.
Core principles
- 01Use general models for broad and ambiguous requests
- 02Use specialist models for narrow domain-dependent work
- 03Match model capacity to the task rather than defaulting to maximum scale
- 04Keep model choices replaceable as quality and economics change
- 05Protect critical workflows from uncontrolled upstream changes
How to run it
- 1
Inventory AI tasks
List the product's model-dependent tasks and separate broad, irregular requests from narrow, repeated workflows.
Pro tip Describe each task with real inputs and expected outputs.
Watch out Do not treat all text-generation calls as equivalent.
- 2
Set route requirements
Define minimum quality, acceptable latency, privacy constraints, and cost limits for each task class.
Pro tip Use task-specific acceptance tests rather than one global quality score.
Watch out Optimizing cost without a quality floor can silently damage the product.
- 3
Assign generalist routes
Send broad requests requiring wide world knowledge or flexible interpretation to an appropriate general model.
Pro tip Reserve the largest models for tasks that demonstrate a measurable benefit from their breadth.
Watch out A powerful general model is not automatically the most accurate choice for a narrow domain.
- 4
Develop specialist routes
Use smaller, fine-tuned, or domain-constrained models for stable tasks with specialized terminology and outputs.
Pro tip Start with the highest-volume narrow workflow, where specialization can create the clearest economic gain.
Watch out A specialist trained on weak or outdated examples will reproduce those weaknesses consistently.
- 5
Benchmark the portfolio
Evaluate every candidate route on representative cases and compare quality, latency, and cost.
Pro tip Include difficult and adversarial cases, not only routine examples.
Watch out Do not approve a route based on a handful of impressive demonstrations.
- 6
Monitor and reroute
Track production behavior over time and switch to a fallback when a provider or model falls below the route's acceptance threshold.
Pro tip Store model identifiers and evaluation results alongside operational metrics.
Watch out Without observability, upstream model degradation may appear to be an unexplained product failure.
In the wild
The discussion presents Shopify's Sidekick as the kind of product that benefits from a model optimized for Shopify's own specialized product actions. A domain model can focus on those workflows at lower overhead than a giant model designed to answer almost anything.
→ A specialized assistant can provide more controlled product behavior and economics.
An online marketplace sends open-ended merchant questions to a general model but routes refund classification, catalog normalization, and policy checks to smaller specialist models. Each route has regression tests and automatically falls back when its weekly score drops below the approved threshold.
→ The marketplace lowers inference costs while maintaining measurable quality on critical workflows.
Common mistakes
Using the biggest model everywhere
Broad capability adds cost and overhead that narrow workflows may not need. Select models against task-specific evidence.
Specializing without evaluation data
A narrow model may appear efficient while failing uncommon but important cases. Benchmark it on a representative test set before routing production traffic.
Ignoring upstream drift
Hosted model behavior can change even when the application code does not. Continuous evaluation and fallbacks are necessary for dependable routing.
Is it for you?
Best for
It is best for teams operating multiple AI workflows with distinct domain, quality, latency, or cost requirements.
Not ideal for
It is not ideal for an early prototype whose limited usage cannot yet justify model evaluation and routing infrastructure.
From the transcript
“we can see the general doctors as the general models”
“they're very capable of taking a broad set of jobs, right?”
“we're gonna see kind of these specialists, people with scalpels and the scalpels make really nice fine cuts”
From the episode
Tech Expert Reveals How AI Could Destroy Your Startup (#147)