Sitemap Change Intelligence Monitor
Watch sitemap additions to detect competitor and industry updates without RSS
- Difficulty
- Easy
- Time to result
- ~days to results
- Steps
- 6
- Confidence
- 98%
The monitor substitutes sitemap comparison for a missing RSS feed. It begins with a curated list of important websites and records the URLs currently present in each sitemap. A scheduled agent periodically fetches those sitemaps, compares the latest URL sets with the stored baseline, and alerts only when a new page appears. The new URL is then classified so routine changes, such as terms-of-service pages, can be separated from blog posts, launches, documentation, or other strategically relevant material. For higher-value monitoring, the system summarizes the page and recommends whether the user should respond, investigate, or update existing work. This turns a basic change signal into an actionable competitive or industry intelligence loop while keeping alert noise manageable.
Origin
Extracted from Marketing Against The Grain from Matt Wolf's monitoring of OpenAI, Anthropic, Google DeepMind, and Meta AI websites.
Core principles
- 01A sitemap can act as a change feed when RSS is unavailable
- 02New URLs are useful detection signals even before content is classified
- 03Fast alerts improve the value of time-sensitive information
- 04Noise should be separated from substantive updates
- 05Monitoring should lead to recommendations, not only summaries
How to run it
- 1
Choose monitored sites
List the organizations whose new pages could affect your work. Keep the list focused enough that every alert has a plausible use.
Pro tip Prioritize competitors, critical platforms, and authoritative industry sources.
Watch out Monitoring too many low-value sites creates alert fatigue.
- 2
Locate the sitemap
Find each site's sitemap or sitemap index and confirm that it exposes meaningful page URLs. Record the relevant sitemap endpoints.
Pro tip Check robots.txt when the sitemap location is not obvious.
Watch out Some sitemaps omit important sections or update slowly.
- 3
Create a baseline
Save the current set of URLs and the time it was observed. This baseline allows future runs to isolate additions rather than rediscovering the whole site.
Pro tip Normalize URLs before comparison to avoid false changes from formatting differences.
Watch out A partial first fetch can make old pages appear newly published.
- 4
Schedule URL diffs
Fetch each sitemap on a recurring schedule and compare it with the stored baseline. Retain newly added URLs and update the baseline after a successful run.
Pro tip Match the interval to how quickly the source publishes and how time-sensitive the information is.
Watch out Do not update the baseline after a failed or obviously incomplete fetch.
- 5
Classify new pages
Identify whether each URL represents a blog post, product page, policy update, documentation change, or irrelevant maintenance page. Suppress categories that repeatedly create noise.
Pro tip Use URL patterns and page metadata before invoking a more expensive model.
Watch out A terms update may still be strategically important even if it is not a blog post.
- 6
Turn alerts into decisions
Summarize relevant additions and state whether they warrant action. Link the recommendation to an existing project, risk, competitor, or content opportunity.
Pro tip Send immediate alerts only for high-priority sites and batch the rest into a digest.
Watch out Fast detection has little value if nobody owns the follow-up.
In the wild
Matt monitors sitemap additions from Anthropic, OpenAI, Google DeepMind, and Meta's AI blog. Although occasional additions are policy pages, he reports that most alerts correspond to new blog posts and arrive close to publication time.
→ He receives early notice of AI industry news even when the publishers provide no RSS feed.
A marketing team could baseline a competitor's product and blog sitemaps, detect newly added URLs, classify launches or positioning changes, and ask an agent whether the development calls for a response.
→ Website changes become actionable competitive intelligence instead of an occasional manual discovery.
Common mistakes
Alerting on every sitemap event
Sitemaps contain policy, utility, and maintenance pages as well as meaningful content. Classify additions and suppress recurring noise.
Treating the sitemap as complete
A sitemap may omit pages or update late. Use it as a useful signal, not guaranteed proof that every site change has been detected.
Stopping at notification
An alert without relevance assessment or an owner creates awareness but not advantage. Connect important changes to a recommended response.
Is it for you?
Best for
It is best for marketers, analysts, and creators who need early awareness of competitor, platform, or industry announcements.
Not ideal for
It is not ideal for websites with incomplete, unstable, or heavily delayed sitemaps.
From the transcript
“I actually set up an automation that looks at the website's site map and alerts me whenever there's a new page added to the website's…”
“I get like almost instant notifications whenever Anthropic, OpenAI, Google, DeepMind, Meta's AI blog, any of these websites actually add a new page to their…”
“create a little agent that's constantly watching the site maps of these websites for a new page added to the site map.”
From the episode
This AI Second Brain Remembers Everything I Save (Codex)