Skip to main content
← back to extensions

Web Crawl

@twonines/web-crawlv2026.07.20.1· 12d agoMODELSWORKFLOWSREPORTS
01README

Automated web reading pipeline — harvests articles from configurable sources (Hacker News, Lobsters, RSS/Atom feeds), evaluates them with an AI agent, learns user interest over time via feedback, and produces a curated HTML report with ranked recommendations. Designed for agent-driven reading workflows where an AI reads on the user's behalf and surfaces what matters.

02Models2
@twonines/web-crawl/harvesterv2026.07.20.1harvester.ts

Global Arguments

ArgumentTypeDescription
sourcesarrayList of sources to harvest from. Each source has a type (hackernews, lobsters, rss) and type-specific config.
fn harvest(maxTotal: number)
Fetch articles from all configured sources, deduplicate against previously seen URLs, and store fresh candidates.
ArgumentTypeDescription
maxTotalnumberMaximum total candidates to return across all sources.

Resources

candidates(7d)— Fresh article candidates harvested from configured sources.
seen(30d)— Set of previously seen article URLs for deduplication across runs.
@twonines/web-crawl/evaluatorv2026.07.20.1evaluator.ts

Global Arguments

ArgumentTypeDescription
harvesterModelIdstringModel ID or name of the harvester instance to read candidates from. Empty = find any @twonines/web-crawl/harvester.
maxArticlesnumberMaximum articles to evaluate per run.
fn evaluate(harvestId?: string, assessments: array)
Read the latest harvest candidates, fetch their content, and produce assessments with scores and reactions. The calling agent should provide genuine engagement — not just summaries.
ArgumentTypeDescription
harvestId?stringSpecific harvest run to evaluate. If omitted, uses the most recent.
assessmentsarrayThe agent's assessments of the articles. The agent reads the content (provided via context or fetched) and fills these in.
fn fetch_content(urls: array)
Fetch readable text from a list of URLs. Utility method for agents that want to read article content before producing assessments.
ArgumentTypeDescription
urlsarrayURLs to fetch content from.
fn feedback(signals: array)
Record user feedback on articles to update learned preferences. Call after a user reads or skips recommended articles.
ArgumentTypeDescription
signalsarrayBatch of feedback signals.
fn generate_report(runId?: string, title: string, outputPath?: string)
Generate an HTML reading list report from the most recent assessments. Writes the HTML to a file artifact.
ArgumentTypeDescription
runId?stringSpecific evaluation run to report on. If omitted, uses the most recent.
titlestringTitle for the HTML report page.
outputPath?stringLocal file path to also write the HTML to (in addition to the swamp file artifact).

Resources

assessments(30d)— Evaluated articles with scores, reactions, and recommendations.
preferences(infinite)— Learned user interest preferences from feedback signals.

Files

report(text/html)— HTML reading list report.
03Workflows1
@twonines/web-crawl-run7c3e8f1a-4d2b-4e9f-b8a1-f3c2d5e6a7b9

Complete web crawl pipeline: harvest fresh articles from configured sources, then evaluate them (agent-driven assessment), then generate the curated HTML report. The evaluate step requires an AI agent to run — it reads the harvested candidates, fetches their content, and produces opinionated assessments. This workflow is intended to be triggered by an agent session or scheduled with agent involvement.

harvestFetch fresh article candidates from all configured sources.
1.fetch-articlesweb-crawl-harvester.harvest— Run the harvester to pull from HN, Lobsters, and RSS feeds. Instance 'web-crawl-harvester' must be pre-created with globalArguments.sources configured.
evaluateRead and assess harvested articles.
1.assess-articlesweb-crawl-evaluator.evaluate— Agent evaluates the harvested candidates and produces scored assessments. Instance 'web-crawl-evaluator' must be pre-created.
04Reports1
@twonines/web-crawl-reportworkflow
report.ts

Curated HTML reading list with ranked recommendations and engagement notes. Renders assessments from the web-crawl evaluator into a styled page.

webcrawlcurationreading
05Previous Versions1
2026.07.18.1
06Stats
A
100 / 100
Downloads
0
Archive size
29.3 KB
  • Has README or module doc2/2earned
  • README has a code example1/1earned
  • README is substantive1/1earned
  • Most symbols documented1/1earned
  • No slow types (deprecated)1/1earned
  • Dependencies pass trust audit2/2earned
  • Has description1/1earned
  • Platform support declared (or universal)2/2earned
  • License declared1/1earned
  • Verified public repository2/2earned
Repository
https://codeberg.org/twonines/swamp-extensions
07Platforms
08Labels
#web#crawl#rss#reading#curation#report#ai