Anydoc Ingest
Document-to-knowledge ingestion pipeline powered by Firecrawl's anydoc. Converts Word, PowerPoint, Excel, OpenDocument, RTF, EPUB, CSV, and PDF files into structured markdown, then materialises provenance-aware knowledge entries compatible with @stateless/sourced-kb.
Designed as the intake layer for sourced-kb: point it at a directory of
documents, run ingest, and each file becomes a versioned KB entry with
full provenance (source path, conversion timestamp, format detected,
content hash). Run scan first to preview what will be processed.
Quick Start
swamp extension pull @webframp/anydoc-ingest
swamp model create @webframp/anydoc-ingest doc-ingest \
--global-arg 'documentsDir=/path/to/docs'
# Preview available documents
swamp model method run doc-ingest scan
# Convert and write KB entries
swamp model method run doc-ingest ingest
# Check conversion stats
swamp model method run doc-ingest statusMethods
- scan — Discover supported documents in the configured directory; reports file count, formats, and total size without converting.
- ingest — Convert each document to markdown via anydoc and write a
provenance-aware
documentresource per file. Idempotent: re-running skips files whose content hash hasn't changed. - status — Summarise ingestion state: total documents, last run, formats processed, errors encountered.
Complementing @stateless/sourced-kb
Each document resource carries the same shape that sourced-kb's apply
method expects as a declared entry: id, kind, facets, claims with asOf/source
provenance. Pipe ingest output into a sourced-kb instance's declaration
file to version and track document-derived knowledge alongside other sources.
2026.09.18.1
Upgrade note: Normalized npm:zod dependency version to 4.6.5 across the
repo. No behavioral changes in this extension.
Global Arguments
| Argument | Type | Description |
|---|---|---|
| documentsDir | string | Absolute path to the directory containing documents to ingest |
| recursive | boolean | Whether to scan subdirectories recursively |
| maxFileSizeMb | number | Maximum file size in MB to attempt conversion |
| includePatterns | array | Glob patterns to include (empty = all supported formats) |
| excludePatterns | array | Glob patterns to exclude |
| Argument | Type | Description |
|---|---|---|
| force | boolean | Force re-conversion even if content hash matches |
Resources
2026.09.15.1
Changed: Bump zod 4.4.3 → 4.6.5
2026.08.28.1
Changed: Normalized the extension license to Apache-2.0 and corrected the copyright holder to "Sean Escriva". Extensions that previously shipped an MIT LICENSE.md are now Apache-2.0, consistent with the repository root and every other extension. No code or behavioral changes.
Upgrade note: License text only. No API, schema, or runtime behavior changed.
2026.08.26.3
Fixed: Restored inline npm:zod@4.4.3 import specifiers so the registry
quality scorer can resolve dependencies and score the extension. An earlier
release used a bare "zod" import-map specifier, which published but scored as
unscored.
Changed: Retained explicit compilerOptions.strict in deno.json. No
behavioral or schema changes.
2026.08.28.1
Changed: Normalized the extension license to Apache-2.0 and corrected the copyright holder to "Sean Escriva". Extensions that previously shipped an MIT LICENSE.md are now Apache-2.0, consistent with the repository root and every other extension. No code or behavioral changes.
Upgrade note: License text only. No API, schema, or runtime behavior changed.
2026.08.26.3
Fixed: Restored inline npm:zod@4.4.3 import specifiers so the registry
quality scorer can resolve dependencies and score the extension. An earlier
release used a bare "zod" import-map specifier, which published but scored as
unscored.
Changed: Retained explicit compilerOptions.strict in deno.json. No
behavioral or schema changes.
2026.08.26.3
Fixed: Restored inline npm:zod@4.4.3 import specifiers so the registry
quality scorer can resolve dependencies and score the extension. An earlier
release used a bare "zod" import-map specifier, which published but scored as
unscored.
Changed: Retained explicit compilerOptions.strict in deno.json. No
behavioral or schema changes.
2026.08.26.1
Changed: Normalized deno.json configuration for repo-wide consistency:
added explicit compilerOptions.strict and migrated zod dependency to the
import map (bare "zod" specifier instead of inline npm:zod@4.4.3). No
behavioral changes — runtime resolution is identical.
2026.08.24.2
Added: Output metadata attributes for observability.
durationMs: Method execution duration in milliseconds.collectedBy: Extension name that produced the data.fetchedAt: ISO 8601 timestamp when data was fetched (added to resources that previously lacked it).
2026.08.24.1
Fixed source/manifest version mismatch. Added Troubleshooting section documenting scan/ingest file caps, per-document non-fatal failures, 120s timeout, absolute path requirement, idempotency with error re-attempt, npx dependency, and silent permission-error skipping.
2026.08.21.2
Changed: documentsDir is now validated as an absolute path at model
creation time, matching what the docstring always required — a relative path
previously slipped through and could produce confusing relative-path output
or unexpected directory resolution deep inside a scan/ingest run instead of
failing immediately with a clear message.
A Deno.stat failure on documentsDir other than "not found" (e.g. a
permissions error) now raises an error naming the directory and the
underlying cause, instead of propagating the bare Deno error with no
indication of which directory failed.
2026.08.21.1
Changed: Added descriptions to the previously undocumented fields in the
scan, document, and status resource schemas (e.g. contentHash,
markdownLength, provenance, byFormat, errors). No behavioral change.
2026.08.12.2
Fixed: ingest no longer aborts the whole run when a file is deleted or
becomes inaccessible between directory enumeration and Deno.stat. Previously
the stat call could throw NotFound or PermissionDenied inside walkDir,
bypassing the per-document try/catch and dropping the status writeResource
so nothing about the run was recorded. Missing entries are now silently
skipped in walkDir and will resurface on the next run if they exist.
Fixed: The MAX_INGEST_FILES cap now counts documents that fail in the
outer catch (hashFile errors, writeResource failures) alongside successes
and skips. A directory of 10,000+ unreadable files could previously iterate
without bound because every doc landed in errors and neither ingested
nor skipped incremented, so the cap never triggered.
Added: status resource now carries totalConverted alongside
totalIngested. totalIngested counts every document resource written
this run — including failed conversions that left markdown: "" and set the
error field. totalConverted counts only the documents that produced
non-empty markdown. Trend dashboards should switch to totalConverted for
the success signal.
Upgrade note: totalConverted is a new field on the status resource
with a Zod default of 0. Existing stored status resources written by
2026.08.12.1 will read back cleanly. When the status method re-writes an
older resource, it seeds totalConverted from the historical totalIngested
so timelines don't show a synthetic drop to zero.
2026.08.12.2
Fixed: ingest no longer aborts the whole run when a file is deleted or
becomes inaccessible between directory enumeration and Deno.stat. Previously
the stat call could throw NotFound or PermissionDenied inside walkDir,
bypassing the per-document try/catch and dropping the status writeResource
so nothing about the run was recorded. Missing entries are now silently
skipped in walkDir and will resurface on the next run if they exist.
Fixed: The MAX_INGEST_FILES cap now counts documents that fail in the
outer catch (hashFile errors, writeResource failures) alongside successes
and skips. A directory of 10,000+ unreadable files could previously iterate
without bound because every doc landed in errors and neither ingested
nor skipped incremented, so the cap never triggered.
Added: status resource now carries totalConverted alongside
totalIngested. totalIngested counts every document resource written
this run — including failed conversions that left markdown: "" and set the
error field. totalConverted counts only the documents that produced
non-empty markdown. Trend dashboards should switch to totalConverted for
the success signal.
Upgrade note: totalConverted is a new field on the status resource
with a Zod default of 0. Existing stored status resources written by
2026.08.12.1 will read back cleanly. When the status method re-writes an
older resource, it seeds totalConverted from the historical totalIngested
so timelines don't show a synthetic drop to zero.
- Has README or module doc2/2earned
- README has a code example1/1earned
- README is substantive1/1earned
- Most symbols documented1/1earned
- No slow types (deprecated)1/1earned
- Dependencies pass trust audit2/2earned
- Has description1/1earned
- Platform support declared (or universal)2/2earned
- License declared1/1earned
- Verified public repository2/2earned