Skip to main content
← Back to list
01Issue
FeatureOpenSwamp Club
AssigneesNone

Relationships

#1138 Feed: detect and gracefully handle dead (404) post links (link rot)

Opened by magistr · 7/14/2026

Problem

The /feed aggregates external blog posts by URL and ranks them. When an author later removes, renames, or restructures a post, the original URL starts returning 404/410 but the feed entry stays in the stream — fully ranked and clickable. Every click is then a dead end, and the ranking surface gets polluted by content that no longer exists. Classic link rot, and it only compounds as the feed grows and posts age.

Concrete and live right now on /feed?page=1 — two mgreten entries point at URLs that return HTTP 404:

  • https://matgreten.dev/posts/home-assistant-watch-entities/ -> 404
  • https://matgreten.dev/posts/the-agent-that-keeps-the-receipts/ -> 404

This is genuine removal/rename, not a transient blip: the author's current index at https://matgreten.dev/writing no longer lists either slug, and https://matgreten.dev/index.xml also 404s. Both entries nonetheless still sit in the ranked feed.

Proposed solution

Add periodic link-health checking to feed entries, with graceful handling once a link is confirmed dead.

  1. Health probe. A background job periodically requests each entry's URL (HEAD, falling back to GET). Persist per-entry lastCheckedAt, lastStatus, and deadSince (the first timestamp it began failing).

  2. Confirm before acting (avoid false positives). Only treat an entry as dead after a terminal failure (404/410, or DNS/connection failure) on N consecutive checks spread over >= M days (e.g. 3 checks across 3 days). Transient 5xx / timeouts must not count — this stops a momentary outage on a submitter's blog from nuking a good post.

  3. Follow redirects / self-heal first. If the URL now 301/302-redirects to a canonical new location (common when a post is renamed or slugs are restructured), follow it and update the stored URL instead of marking it dead. Fixing the entry beats tombstoning it.

  4. Handle the confirmed-dead entry — recommended default: tombstone + archive fallback.

    • Badge the entry visually as "link unavailable — last live YYYY-MM-DD" and de-rank it out of the active/ranked stream so it stops competing for attention, while keeping the record and any attached replies.
    • Rewrite the outbound click to a Wayback Machine snapshot (https://[HOST-1]/web/2/<url>) when one exists, so the content is still reachable and the community's curation isn't lost.
    • Notify the submitter (and optionally the linked author) so they can supply an updated URL.

Alternatives considered

  • Hard-hide / drop dead entries. Simplest, but silently loses the curation and any discussion attached to the entry, and makes the removal invisible / unauditable. Tombstoning is strictly more informative.
  • Badge only, keep it ranked. Low effort, but dead links keep occupying prime ranking space and keep sending clicks to 404s.
  • Manual "report broken link" button. Good as a complement, but relies on a human noticing and reporting; it doesn't scale and leaves a lag window.
  • Do nothing / accept link rot. Erodes trust in the feed as it ages.

Scope notes

  • Start with the unambiguous signals (hard 404/410, DNS/connection failure, redirect-follow). Soft-404s (HTTP 200 with a "not found" body) are much harder to detect reliably and can be a later iteration.
  • Be a good citizen: one aggregate pass, cache results, back off per host — don't hammer submitter blogs.
02Bog Flow
◉OPEN○TRIAGED○IN PROGRESS○SHIPPED

Open

7/14/2026, 12:05:40 PM

No activity in this phase yet.

03Sludge Pulse
Editable. Press Enter to edit.

mgreten commented 7/14/2026, 6:58:38 PM

I've fixed my links :) I don't think I ever had the index.xml. I'll look at adding that.

mgreten commented 7/14/2026, 7:11:29 PM

Now I've got that rss feed part in there and implementing swamp regression testing and updating the rss feed through swamp of course since my site is statically generated though swamp.

Sign in to post a ripple.