Understanding advanced feed crawling in Letterhead

Advanced feed crawling is an optional per-feed setting that enriches the metadata Letterhead collects from your RSS feeds. Most feeds don’t need it — standard RSS metadata imports just fine — so only enable it when a feed is missing key details like images, excerpts, or author names.

How it works

• Disabled (default) — Letterhead uses only the data the RSS feed provides: title, description, image.
• Enabled — Letterhead’s crawler also visits the webpage each feed item links to and reads Open Graph tags and other structured data from the page, supplementing or updating what the feed provided.

The richer data comes at a cost: crawling every linked page takes longer and generates more bot traffic to your site.

When to enable it

Turn it on for a feed only if you hit one of these:

• The feed lacks images, excerpts, or author names that the webpages do have.
• The webpage data is more current or complete than the feed data.
• You run multiple brands or domains through one feed and need distinct publisher names per site.

When to leave it off

• The feed already includes everything you need.
• You ingest a high volume of content and want faster imports.
• Your site restricts bots (login walls, paywalls, crawler blocks) — Letterhead can’t crawl content behind authentication, including private sites and social platforms.
• You need to keep crawl volume down to avoid rate-limit errors on your server.

Turn it on for a feed

1. Open Curations and go to the RSS Feeds tab.
2. Edit an existing feed, or add a new one.
3. Check Enable advanced feed crawling and save.

For crawling to succeed, your site must allow Letterhead’s crawler: safelist *.tryletterhead.com, letterhead.email, and letterhead.ai, and accept the LetterheadCurationBot user agent (Mozilla/5.0 (compatible; LetterheadCurationBot/1.0; +http://tryletterhead.com/bots)).

Best practice: test on one or two feeds first to see whether metadata actually improves, and keep your feeds healthy with the W3C Feed Validator. If images still aren’t coming through, see ensuring your images appear in the RSS block.