Research Collector
Research data collector — gathers intelligence from Hacker News, Lobste.rs, arXiv, SRE Weekly, IFIN Discourse, RedMonk, and The AI Daily Brief for daily AI/tech intelligence briefings. Produces typed resources for downstream workflows.
Usage
swamp model create @webframp/research-collector research-collector
# Gather all sources
swamp model method run research-collector gather
# Build a compact digest from the latest brief
swamp model method run research-collector digestMethods
- gather — Gather data from HN front page, Lobste.rs hottest, arXiv, SRE Weekly, IFIN Discourse security topics, RedMonk, and The AI Daily Brief.
- digest — Build a compact digest from the latest brief: top items per source, cross-source topic clusters, and a delta against the previous digest. Run gather first.
2026.09.15.1
Changed: Bump zod 4.4.3 → 4.6.5
2026.08.28.1
Changed: Normalized the extension license to Apache-2.0 and corrected the copyright holder to "Sean Escriva". Extensions that previously shipped an MIT LICENSE.md are now Apache-2.0, consistent with the repository root and every other extension. No code or behavioral changes.
Upgrade note: License text only. No API, schema, or runtime behavior changed.
2026.08.26.2
Fixed: Restored inline npm:zod@4.4.3 import specifiers so the registry
quality scorer can resolve dependencies and score the extension. An earlier
release used a bare "zod" import-map specifier, which published but scored as
unscored.
Changed: Retained explicit compilerOptions.strict in deno.json. No
behavioral or schema changes.
Global Arguments
| Argument | Type | Description |
|---|---|---|
| hnCount | number | Number of Hacker News front-page stories to fetch |
| lobstersCount | number | Number of Lobste.rs hottest stories to fetch |
| sreCount | number | Number of SRE Weekly issues to fetch |
| ifinCount | number | Number of IFIN Discourse topics to fetch |
| redmonkCount | number | Number of RedMonk blog posts to fetch |
| arxivCount | number | Number of arXiv paper entries to fetch |
| aiDailyBriefDays | number | Number of recent The AI Daily Brief editions to gather (one per day) |
Resources
2026.08.28.1
Changed: Normalized the extension license to Apache-2.0 and corrected the copyright holder to "Sean Escriva". Extensions that previously shipped an MIT LICENSE.md are now Apache-2.0, consistent with the repository root and every other extension. No code or behavioral changes.
Upgrade note: License text only. No API, schema, or runtime behavior changed.
2026.08.26.2
Fixed: Restored inline npm:zod@4.4.3 import specifiers so the registry
quality scorer can resolve dependencies and score the extension. An earlier
release used a bare "zod" import-map specifier, which published but scored as
unscored.
Changed: Retained explicit compilerOptions.strict in deno.json. No
behavioral or schema changes.
2026.08.26.2
Fixed: Restored inline npm:zod@4.4.3 import specifiers so the registry
quality scorer can resolve dependencies and score the extension. An earlier
release used a bare "zod" import-map specifier, which published but scored as
unscored.
Changed: Retained explicit compilerOptions.strict in deno.json. No
behavioral or schema changes.
2026.08.24.1
Added: Output metadata attributes for observability.
durationMs: Method execution duration in milliseconds.collectedBy: Extension name that produced the data.fetchedAt: ISO 8601 timestamp when data was fetched (added to resources that previously lacked it).
2026.08.23.1
Changed: Documentation only — no code changes. Documented the silent
per-source degrade behavior in gatherAll — all 7 scraped sources fail
independently to an empty array with only an info-level log — and flagged
the regex-based AI Daily Brief HTML scraping as the most fragile source.
2026.08.21.1
Changed: When a source (HN, Lobste.rs, SRE Weekly, IFIN Discourse, RedMonk,
arXiv, or The AI Daily Brief) fails to gather, gather now logs the source
name and the underlying error message before falling back to an empty result
for that source. Previously the failure was discarded silently, so a brief
with hnFrontPage.stories: [] gave no clue whether HN genuinely had nothing
new or the fetch failed. gather still returns partial data on a single
source failure — that behavior is unchanged.
Upgrade note: No schema changes. Existing instances need no migration.
Add Swamp Club RSS feed as a research source.
Add System Design RSS feed as a research source.
Add Latent.Space RSS feed as a research source.
Add Simon Willison's Weblog Atom feed as the simonWillison research source.
2026.07.23.1
Added: New digest method builds a compact daily digest from the latest
research brief: top items per source (normalized 0-100 prominence score),
cross-source topic clusters (keywords appearing in 2+ sources), and a delta
against the previous digest (new vs carried items). Writes a new digest
resource (24h lifetime) for downstream journal/briefing workflows.
Changed: The model now reads prior resources via the readResource context
method. digest requires a gather run first — it throws if no brief exists.
Upgrade note: No changes to the research brief schema or existing global
args. Existing instances upgrade in place (no attribute migration needed). The
new digest resource is additive and safe to ignore for workflows that only
use gather.
Modified 1 models
2026.07.21.1
Added: The AI Daily Brief as a seventh gather source. gather now fetches
recent editions from theaidailybrief.com, keeps written takeaway "nuggets" with
headings, bodies, and anchors, and drops video embeds so only written analysis
survives. Controlled by a new aiDailyBriefDays global arg (1-14, default 3).
Fixed: The test module had unescaped quotes in three XML fixture string literals that stopped Deno from parsing the file, so the AI Daily Brief coverage test never ran. Switched those fixtures to template literals.
Changed: gather's output resource now carries an aiDailyBrief object
with editions alongside the existing source arrays. Existing sources and
resource shapes are unchanged.
Add The AI Daily Brief as a seventh source: scrapes daily written editions from aidailybrief.ai, keeps written analysis nuggets, discards video/audio embeds. New global arg aiDailyBriefDays (1-14, default 3).
2026.07.18.1
Added: An upgrades array entry (no-op) to research_collector.ts for proper typeVersion tracking on existing instances. No schema or behavior changes.
- Has README or module doc2/2earned
- README has a code example1/1earned
- README is substantive1/1earned
- Most symbols documented1/1earned
- No slow types (deprecated)1/1earned
- Dependencies pass trust audit2/2earned
- Has description1/1earned
- Platform support declared (or universal)2/2earned
- License declared1/1earned
- Verified public repository2/2earned