Implementer Benchmark
@mgreten/implementer-benchmarkv2026.08.19.2
01README
Runs five deterministic, bounded synthetic JavaScript tasks to compare coding implementers with isolated hidden verification and durable evidence.
02Release Notes
Add runActorSuite for bounded atomic edit-and-test tasks with separate contract, hardening, and certified scoring.
03Models
@mgreten/implementer-benchmarkv2026.08.19.2implementer_benchmark.ts
Global Arguments
| Argument | Type | Description |
|---|---|---|
| workspaceRoot | string | |
| swampPath | string | |
| nodePath | string | |
| bubblewrapPath | string | |
| agentModelName | string |
fn runSuite(suiteId?: string, taskIds: array, candidates: array, wallTimeoutMs: number, maxPacketBytes: number, maxOutputBytes: number, maxChangedLines: number)
Serially run identical bounded context packets through candidate models and certify their returned file contents
| Argument | Type | Description |
|---|---|---|
| suiteId? | string | |
| taskIds | array | |
| candidates | array | |
| wallTimeoutMs | number | |
| maxPacketBytes | number | |
| maxOutputBytes | number | |
| maxChangedLines | number |
fn runActorSuite(suiteId?: string, taskIds: array, candidates: array, wallTimeoutMs: number, maxPacketBytes: number, maxOutputBytes: number, maxChangedLines: number)
Serially run bounded atomic tasks through tool-using implementation agents, then score contract, hardening, and certified outcomes
| Argument | Type | Description |
|---|---|---|
| suiteId? | string | |
| taskIds | array | |
| candidates | array | |
| wallTimeoutMs | number | |
| maxPacketBytes | number | |
| maxOutputBytes | number | |
| maxChangedLines | number |
Resources
caseResult(365d)— Deterministic result for one candidate and synthetic task
suiteSummary(365d)— Aggregate synthetic implementer benchmark suite result
Files
evidence(application/json)— Bounded diff and test evidence for one benchmark case
04Previous Versions
2026.08.19.1
Initial release of the bounded synthetic coding implementer benchmark with hidden verification and explicit effort-level metadata.
05Stats
A
100 / 100
Downloads
0
Archive size
53.0 KB
- Has README or module doc2/2earned
- README has a code example1/1earned
- README is substantive1/1earned
- Most symbols documented1/1earned
- No slow types (deprecated)1/1earned
- Dependencies pass trust audit2/2earned
- Has description1/1earned
- Platform support declared (or universal)2/2earned
- License declared1/1earned
- Verified public repository2/2earned
06Platforms