Multimodal-From-the-Ground-Up Design
Train across every information format from the beginning
- Difficulty
- Expert
- Time to result
- ~months to results
- Steps
- 4
- Confidence
- 98%
Multimodal-from-the-ground-up design treats different information formats as foundational training inputs rather than features attached after a model has been built. The team gathers text, video, audio, images, and other relevant formats, uses them during pre-training, and reinforces the same cross-format abilities during fine-tuning. The resulting system can connect information between modalities—for example, interpreting a sketch, watching a physical game, reading scientific charts, and responding through code or language. The mechanism matters because merely bolting visual support onto a text model may permit image recognition without producing fluent cross-modal reasoning. Starting with modality breadth makes the model’s internal capabilities better aligned with real environments, where people naturally communicate through combinations of words, voices, gestures, diagrams, and moving images.
Origin
Extracted from Marketing Against The Grain
Core principles
- 01Design for multiple modalities at the foundation
- 02Train and fine-tune with the same modality breadth
- 03Treat text, images, audio, and video as complementary inputs
- 04Let modality integration create capabilities unavailable to text-only systems
How to run it
- 1
Map the modality surface
List the formats users naturally provide and the formats the system must interpret or generate. Include combinations such as spoken instructions paired with live video.
Pro tip Prioritize modality combinations that unlock a complete user task.
Watch out Do not call a system multimodal merely because it accepts an image attachment.
- 2
Build foundational training coverage
Represent the selected modalities during pre-training so cross-format understanding develops inside the core model.
Pro tip Use examples where information must be connected across modalities.
Watch out Heavily imbalanced data can leave one modality nominally supported but practically weak.
- 3
Fine-tune multimodal behavior
Fine-tune on realistic tasks requiring the model to interpret, reason, and respond across formats.
Pro tip Include tasks with ambiguity, changing visual states, and corrective feedback.
Watch out Single-modality fine-tuning can erode the integrated behavior established earlier.
- 4
Evaluate integrated use cases
Test whether the system can combine modalities in real time rather than passing isolated benchmark questions.
Pro tip Use open-ended tasks such as creating a lesson from physical objects and spoken preferences.
Watch out Polished demonstrations may conceal latency, brittleness, or unavailable production features.
In the wild
A learner asks the AI to create a geography game using a physical map and emojis. The system generates clues, watches the learner point to a country, and determines whether the physical response is correct without requiring a typed answer.
→ The same system creates and administers an interactive lesson across language, images, and live visual input.
The system watches a ball or paper move beneath physical cups and identifies the final location. Success depends on maintaining visual state over time rather than analyzing one static image.
→ The model demonstrates temporal visual reasoning over a real-world interaction.
Common mistakes
Adding modalities after the fact
Attaching image or audio tools to a text-first system does not necessarily produce integrated cross-modal reasoning.
Testing modalities in isolation
Separate text and image benchmarks can pass even when the system cannot connect an instruction to a changing visual scene.
Is it for you?
Best for
Teams building AI products that must reason across text, images, audio, video, diagrams, or physical interactions.
Not ideal for
Simple applications whose inputs and outputs are reliably limited to one structured format.
From the transcript
“Right, it means they pre-trained data on different modalities from the start, right? So text, video, audio, images, all the different formats of information was…”
“And then it was fine-tuned with multimodal data as well.”
From the episode
Google Launches Gemini AI (And It’s Better Than GPT-4)