Anydoc Ingest
Document-to-knowledge ingestion pipeline powered by Firecrawl's anydoc. Converts Word, PowerPoint, Excel, OpenDocument, RTF, EPUB, CSV, and PDF files into structured markdown, then materialises provenance-aware knowledge entries compatible with @stateless/sourced-kb.
Designed as the intake layer for sourced-kb: point it at a directory of
documents, run ingest, and each file becomes a versioned KB entry with
full provenance (source path, conversion timestamp, format detected,
content hash). Run scan first to preview what will be processed.
Quick Start
swamp extension pull @webframp/anydoc-ingest
swamp model create @webframp/anydoc-ingest doc-ingest \
--global-arg 'documentsDir=/path/to/docs'
# Preview available documents
swamp model method run doc-ingest scan
# Convert and write KB entries
swamp model method run doc-ingest ingest
# Check conversion stats
swamp model method run doc-ingest statusMethods
- scan — Discover supported documents in the configured directory; reports file count, formats, and total size without converting.
- ingest — Convert each document to markdown via anydoc and write a
provenance-aware
documentresource per file. Idempotent: re-running skips files whose content hash hasn't changed. - status — Summarise ingestion state: total documents, last run, formats processed, errors encountered.
Complementing @stateless/sourced-kb
Each document resource carries the same shape that sourced-kb's apply
method expects as a declared entry: id, kind, facets, claims with asOf/source
provenance. Pipe ingest output into a sourced-kb instance's declaration
file to version and track document-derived knowledge alongside other sources.
2026.08.21.2
Changed: documentsDir is now validated as an absolute path at model
creation time, matching what the docstring always required — a relative path
previously slipped through and could produce confusing relative-path output
or unexpected directory resolution deep inside a scan/ingest run instead of
failing immediately with a clear message.
A Deno.stat failure on documentsDir other than "not found" (e.g. a
permissions error) now raises an error naming the directory and the
underlying cause, instead of propagating the bare Deno error with no
indication of which directory failed.
Global Arguments
| Argument | Type | Description |
|---|---|---|
| documentsDir | string | Absolute path to the directory containing documents to ingest |
| recursive | boolean | Whether to scan subdirectories recursively |
| maxFileSizeMb | number | Maximum file size in MB to attempt conversion |
| includePatterns | array | Glob patterns to include (empty = all supported formats) |
| excludePatterns | array | Glob patterns to exclude |
| Argument | Type | Description |
|---|---|---|
| force | boolean | Force re-conversion even if content hash matches |
Resources
2026.08.21.1
Changed: Added descriptions to the previously undocumented fields in the
scan, document, and status resource schemas (e.g. contentHash,
markdownLength, provenance, byFormat, errors). No behavioral change.
2026.08.12.2
Fixed: ingest no longer aborts the whole run when a file is deleted or
becomes inaccessible between directory enumeration and Deno.stat. Previously
the stat call could throw NotFound or PermissionDenied inside walkDir,
bypassing the per-document try/catch and dropping the status writeResource
so nothing about the run was recorded. Missing entries are now silently
skipped in walkDir and will resurface on the next run if they exist.
Fixed: The MAX_INGEST_FILES cap now counts documents that fail in the
outer catch (hashFile errors, writeResource failures) alongside successes
and skips. A directory of 10,000+ unreadable files could previously iterate
without bound because every doc landed in errors and neither ingested
nor skipped incremented, so the cap never triggered.
Added: status resource now carries totalConverted alongside
totalIngested. totalIngested counts every document resource written
this run — including failed conversions that left markdown: "" and set the
error field. totalConverted counts only the documents that produced
non-empty markdown. Trend dashboards should switch to totalConverted for
the success signal.
Upgrade note: totalConverted is a new field on the status resource
with a Zod default of 0. Existing stored status resources written by
2026.08.12.1 will read back cleanly. When the status method re-writes an
older resource, it seeds totalConverted from the historical totalIngested
so timelines don't show a synthetic drop to zero.
2026.08.12.2
Fixed: ingest no longer aborts the whole run when a file is deleted or
becomes inaccessible between directory enumeration and Deno.stat. Previously
the stat call could throw NotFound or PermissionDenied inside walkDir,
bypassing the per-document try/catch and dropping the status writeResource
so nothing about the run was recorded. Missing entries are now silently
skipped in walkDir and will resurface on the next run if they exist.
Fixed: The MAX_INGEST_FILES cap now counts documents that fail in the
outer catch (hashFile errors, writeResource failures) alongside successes
and skips. A directory of 10,000+ unreadable files could previously iterate
without bound because every doc landed in errors and neither ingested
nor skipped incremented, so the cap never triggered.
Added: status resource now carries totalConverted alongside
totalIngested. totalIngested counts every document resource written
this run — including failed conversions that left markdown: "" and set the
error field. totalConverted counts only the documents that produced
non-empty markdown. Trend dashboards should switch to totalConverted for
the success signal.
Upgrade note: totalConverted is a new field on the status resource
with a Zod default of 0. Existing stored status resources written by
2026.08.12.1 will read back cleanly. When the status method re-writes an
older resource, it seeds totalConverted from the historical totalIngested
so timelines don't show a synthetic drop to zero.
- Has README or module doc2/2earned
- README has a code example1/1earned
- README is substantive1/1earned
- Most symbols documented1/1earned
- No slow types (deprecated)1/1earned
- Dependencies pass trust audit2/2earned
- Has description1/1earned
- Platform support declared (or universal)2/2earned
- License declared1/1earned
- Verified public repository2/2earned