Best-Model Plug-and-Play Architecture
Route each task to the model best suited to perform it.
- Difficulty
- Advanced
- Time to result
- ~months to results
- Steps
- 5
- Confidence
- 90%
This architecture treats language models as interchangeable execution components rather than making one provider the entire application. The team first decomposes a business workflow into discrete tasks, such as extraction, generation, analysis, or coding. It then defines the quality, cost, latency, privacy, and reliability requirements for each task and evaluates eligible models against representative data. A routing layer selects the best acceptable model for each piece of work while the surrounding application preserves the customer experience, business rules, and integrations. Fallbacks maintain continuity when a provider is unavailable or degrades. Because model capabilities and prices change quickly, routes are reviewed regularly. The result is a differentiated application assembled from complementary models rather than a thin interface around one API.
Origin
Extracted from Marketing Against the Grain as the hosts discussed how applications can combine OpenAI, Anthropic, Meta, and other model APIs.
Core principles
- 01Different models can excel at different kinds of work.
- 02Applications should separate workflow logic from model selection.
- 03A multi-model layer reduces dependence on one provider.
- 04Routing decisions should reflect quality, cost, latency, privacy, and reliability.
How to run it
- 1
Decompose the workflow
Split the use case into distinct units of work with clear inputs and expected outputs.
Pro tip Keep tasks independently testable wherever possible.
Watch out Routing an entire complex workflow as one task hides important performance differences.
- 2
Define task requirements
Set thresholds for accuracy, latency, cost, context length, privacy, and output format.
Pro tip Weight requirements according to business impact.
Watch out A model with the highest general benchmark score may not be best for the specific task.
- 3
Evaluate candidate models
Test eligible models on representative and difficult examples from the intended workflow.
Pro tip Use the same rubric and dataset for every candidate.
Watch out Do not rely exclusively on provider claims or generic benchmarks.
- 4
Implement routing and fallbacks
Send each task to the best acceptable model and define secondary routes for failures or regressions.
Pro tip Keep provider-specific code behind adapters.
Watch out Fallback outputs must satisfy the same security and validation requirements.
- 5
Monitor and re-route
Track production quality, cost, latency, and errors, then update routes when the evidence changes.
Pro tip Schedule regular re-evaluations because model performance changes rapidly.
Watch out An unattended routing policy can preserve yesterday's winner long after it becomes inferior.
In the wild
A marketing platform uses one model to classify customer research, another to draft copy, and a third to validate structured output. Its routing layer evaluates each stage independently and retains a fallback provider for critical tasks.
→ The platform improves task-level performance and reduces dependence on any single model vendor.
Common mistakes
Picking one winner for every task
Model strengths vary, so a single-provider default can sacrifice quality or economics in parts of the workflow.
Adding providers without evaluation
More integrations create complexity unless each provider has a measured role and clear selection rule.
Forgetting operational fallbacks
A router without tested fallback behavior can turn one provider outage into a complete application failure.
Is it for you?
Best for
It is best for organizations with varied AI workloads and enough engineering capacity to manage multiple providers.
Not ideal for
It is not ideal for simple applications where multi-provider complexity exceeds the likely benefit.
From the transcript
“people are going to build use cases and apps and they're going to Plug and Play into these models and pick the best model for…”
“how do I build really unique use cases by partnering with all of the best large language models”
“there is a role for companies to adopt a lot of the different apis open Ai and Tropics some of the meta apis seamless and…”
From the episode
OpenAI’s GPT 4 Pro Will Run Your Business Better Than You… (#152)