Many Small Models Strategy
Match specialized AI models to individual use cases instead of forcing one model
- Difficulty
- Advanced
- Time to result
- ~months to results
- Steps
- 6
- Confidence
- 90%
The Many Small Models Strategy replaces a one-model-for-everything assumption with a portfolio of task-specific models. The organization decomposes its AI workload into distinct use cases, establishes requirements for each, and benchmarks suitable open or closed models against representative data. A smaller model can then be selected, prompted, or fine-tuned for the job it performs best, while other tasks use different models. This plug-and-play architecture can improve control, privacy, cost efficiency, and specialization without requiring every component to match the broad capabilities of a frontier model. The strategy produces a governed model portfolio in which each deployment has a defined purpose, performance threshold, security boundary, and replacement path.
Origin
Extracted from Marketing Against the Grain during a prediction that open-source models would become a major part of company AI adoption.
Core principles
- 01No single model needs to solve every problem
- 02Task fit can matter more than benchmark leadership
- 03Specialization can improve quality and efficiency
- 04Open models provide greater control over tuning and deployment
- 05Security and operational safety remain selection criteria
How to run it
- 1
Decompose the Work
Separate the business process into specific tasks such as classification, extraction, drafting, translation, or support responses.
Pro tip Choose boundaries that allow each task to be evaluated independently.
Watch out Do not divide the workflow so finely that orchestration becomes unmanageable.
- 2
Define Per-Task Requirements
Set quality, latency, cost, privacy, hosting, and safety requirements for every task.
Pro tip Use actual production examples to establish the evaluation set.
Watch out A model with strong public benchmarks may still fail on company-specific data.
- 3
Match and Benchmark Models
Shortlist models suited to each task and test them against the same representative cases.
Pro tip Include smaller open models when the task is narrow and repeatable.
Watch out Do not select solely on maximum model size or general benchmark rank.
- 4
Customize Where Needed
Improve the selected model with prompting, retrieval, or fine-tuning according to the performance gap.
Pro tip Try the least expensive customization method first.
Watch out Fine-tuning cannot compensate for poor task definition or low-quality data.
- 5
Deploy with Controls
Place each model behind monitoring, access controls, versioning, and a fallback path.
Pro tip Track quality and cost by task rather than only at the platform level.
Watch out Plug-and-play deployment must not bypass security review.
- 6
Replace Models as the Market Evolves
Periodically benchmark alternatives and swap components when another model offers a meaningful improvement.
Pro tip Preserve stable interfaces so models can be replaced without redesigning the workflow.
Watch out Frequent replacement without measurable benefit creates operational churn.
In the wild
A company uses one compact model to classify inbound leads, a locally hosted open model to summarize sensitive customer notes, and a stronger managed model to draft high-value campaign content. Each model is measured against task-specific quality, privacy, and cost requirements.
→ The company gains stronger task fit and security without paying frontier-model prices for every request.
Common mistakes
Using One Model for Everything
A general model can be unnecessarily expensive or poorly suited to narrow, repetitive work.
Fine-Tuning Before Benchmarking
Teams can spend time customizing a model when better prompting or a better-matched base model would suffice.
Ignoring Portfolio Complexity
Multiple models require consistent interfaces, monitoring, security controls, and ownership to remain manageable.
Is it for you?
Best for
Organizations integrating AI across several specialized workflows with different quality, cost, privacy, or latency requirements.
Not ideal for
Small teams with one simple use case that a managed general-purpose model already handles well.
From the transcript
“the best path may be not one model but many small models”
“these open source models where you can easily Plug and Play to do the use case you want to do and find true them to…”
“easily pick them easily fine-tune them use them securely safely within your business”
From the episode
10 AI Predictions That Will Completely Change Life In 2024 (#186)