27 February 2025

BREAKING: Claude 3.7 & Claude Code Just Dropped! (Massive AI Upgrade)

2Frameworks

10Insights

Frameworks in this episode

Strategy5 steps

Distribution and Proprietary Data Moat

Evaluate AI products by who they can reach and what unique data they can use.

Read →

Productivity5 steps

Task-Based AI Model Portfolio

Assign each AI model a job, then test challengers against the incumbent.

Read →

Insights & moments

The myth-busts, hot takes, explainers, and tools worth keeping.

Myth Buster· 1

Myth Buster14:00

Published AI Benchmarks May Reward Training-Set Familiarity

The hosts question whether benchmark performance reflects genuine generalization when benchmark material may have appeared in model training data. They point to private, previously unseen questions as a more revealing way to compare how models handle genuinely new problems.

Benchmark questions may overlap with training data.
Performance can fall when models face net-new questions.
Independent evaluators can maintain private test sets.
Generalization matters more than memorized benchmark familiarity.

“how much of them doing well on the benchmarks, is because they have the benchmark stuff and their training data.”
Kieran Flanagan · 15:00

“There is a big, actual gap between the performance in a benchmark where they've had that data and the training set, versus when you ask…”
Kieran Flanagan · 15:30

#benchmarks#evaluation#training-data#generalization

Hot Take· 3

Hot Take00:30

Claude 3.7 Wins on Coding Taste, Not Just Benchmarks

The hosts argue that Claude's coding appeal cannot be explained fully by standardized scores. Developers describe it as producing tasteful, usable code, with one cited test generating a polished 26-file Next.js template in a single attempt.

Claude 3.7 is positioned as a major coding upgrade.
Developers may prefer a model even when another wins selected benchmarks.
Taste includes design judgment and code quality that benchmarks struggle to measure.
A cited Next.js template required no edits or back-and-forth.

“It's like having a world class dev with exceptional taste.”
Kipp Bodnar · 02:30

“you can't measure taste in these benchmarks.”
Kipp Bodnar · 02:30

#claude#coding#benchmarks#taste

Hot Take15:30

AI Could Counter Humanity's Bias Toward Linear Causality

A Claude response prompts the hosts to consider whether humans systematically miss complex network causality because their minds favor linear, nearby causes. They suggest AI could complement this limitation by identifying interacting causes and opportunities across medicine, economics, and other complex systems.

Humans are strong at identifying linear and proximate causes.
Complex systems often involve network causality.
This cognitive bias may create systematic blind spots.
AI may help people identify relationships their unaided reasoning misses.

“Humans excel at identifying linear and proximate causes, but struggle with complex network causality.”
Kipp Bodnar · 15:30

“We're good at linear thinking. We're not good network thinkers.”
Kipp Bodnar · 16:00

#causality#systems-thinking#cognition#ai

Hot Take17:30

AI Models May Develop Recognizably Different Attitudes

Kieran observes that Grok's descriptions of competing models contain a noticeable point of view, potentially inherited from discourse on X. The segment suggests that models may increasingly differ not only in capability but also in tone, cultural priors, and attitudes toward one another.

Grok characterized OpenAI products in unusually opinionated terms.
Its X-heavy data environment may influence those attitudes.
Model personality can shape how information is framed.
Users may choose models partly for their worldview or conversational feel.

“I do wonder, like, is it just biased? Because the training set in Twitter is like, pro el and Auntie Sam”
Kieran Flanagan · 18:00

“these models are all gonna have like, different attitudes towards each other.”
Kieran Flanagan · 18:00

#model-personality#grok#bias#ai-culture

Explainer· 2

Explainer03:00

Claude 3.7 Combines Quick Answers With Extended Reasoning

Earlier AI products made users choose between a fast general model and a separate reasoning model. Claude 3.7 is presented as the first Claude release able to shift between those modes within one model, anticipating the broader move toward unified model routing.

Previous interfaces exposed separate quick-answer and reasoning models.
Users had to predict which model matched each question.
Claude 3.7 can switch between internal reasoning and faster responses.
Unified routing may eventually hide model selection from users.

“the first model that that is able to switch between chain of thought, like internal chain of thought, where it does reasoning and then just,…”
Kieran Flanagan · 03:00

“You have to kind of choose the model for what question you think you had.”
Kieran Flanagan · 03:30

#claude#reasoning#model-routing

Explainer16:30

Grok's Loose Guardrails Raise Red-Team Questions

The episode explains that red teams probe new models for jailbreaks and unexpected behavior before release. The hosts discuss examples in which Grok reportedly exposed a system prompt or ignored guardrails, wondering whether competitive pressure reduced the depth of pre-release safety testing.

Red teams attempt to uncover jailbreaks and unsafe model behavior.
System-prompt disclosure is presented as one observed weakness.
Users reportedly prompted Grok to move outside its guardrails.
Fast competitive launches can create tension with safety testing.

“these teams are trying to figure out ways that people could jailbreak it, which means you could make it work in unexpected ways.”
Kieran Flanagan · 16:30

“you can actually have grok expose the system think a system prompt.”
Kieran Flanagan · 17:00

#grok#red-teaming#jailbreaks#ai-safety

Story· 2

Story06:00

A Non-Coder Builds the Exact Landing Page He Wants

Kieran recounts using the new Claude model to build a complete landing page rather than first creating a wireframe. The experience let him turn an idea into a concrete artifact that a developer could implement in HubSpot, reconnecting him with an abandoned ambition to build software.

Kieran originally wanted to become a developer but gave up coding.
AI tools made building enjoyable and accessible again.
Claude replaced a separate wireframing step with a functional page.
The finished artifact communicated requirements directly to a developer.

“my original want in life was to be a builder.”
Kieran Flanagan · 06:00

“I just built the full version of the exact web page I want.”
Kieran Flanagan · 07:00

#non-coders#prototyping#landing-pages#claude

Story12:00

Gemini Uses Maps Reviews to Estimate a Restaurant Wait

Kipp describes asking Gemini about the likely wait at a popular New York restaurant directly from its Maps listing. By drawing on review data and the listing context, Gemini supplied an estimate that would previously have been difficult to determine.

Gemini is integrated into Google Maps listings.
Users can converse about a specific place without rebuilding the context.
Review data can support practical predictions such as likely wait time.
The example illustrates AI becoming useful inside existing products.

“you can pick any location, like I was looking at a restaurant that I wanted to go to, and then I can interact with Jim…”
Kipp Bodnar · 12:30

“how long is the wait time like, if I go at this time, how long do you think I'll have to wait”
Kipp Bodnar · 12:30

#gemini#google-maps#restaurants#reviews

Tool· 1

Tool04:00

Claude Code Brings an AI Agent Into the Command Line

Claude Code is described as a beta coding tool that connects to repositories, explains unfamiliar projects, and builds alongside developers. The hosts see it as Anthropic wrapping its model in an AI-first coding experience, directionally similar to the opportunity demonstrated by Cursor.

Claude Code operates through a command-line interface.
It can connect to code repositories.
It can explain code in projects built by other people.
A beta GitHub integration accompanied the release.
Anthropic reportedly saw strong internal adoption among its engineers.

“you can connect it to your repositories, and it can just tell you about your code.”
Kipp Bodnar · 05:00

“it can give you all those insights it can build with you.”
Kipp Bodnar · 05:30

#claude-code#developer-tools#github#cursor

Takeaway· 1

Takeaway08:30

Why Pokémon Was a Better Claude Demo Than Another Academic Benchmark

Anthropic demonstrated Claude 3.7's improvement by comparing how well versions 3.5 and 3.7 played Pokémon. The hosts praise the demonstration as approachable show-don't-tell marketing that makes an abstract capability upgrade legible to ordinary users.

Anthropic benchmarked Claude versions by having them play Pokémon.
The demonstration translated technical progress into a familiar activity.
Relatable proof can communicate more effectively than physics benchmarks.
Fun demonstrations can attract communities while creating value before a sale.

“this is the perfect way, a perfect example of showing and not telling.”
Kipp Bodnar · 09:00

“The marketing for this somewhat writes itself.”
Kieran Flanagan · 09:30

#marketing#pokemon#product-demo#anthropic