Skip to main content

Jev Reliability

@vcjdeboer/jev-reliabilityv2026.09.19.2· 17d agoMODELSREPORTS
01README

Is this Jev question safe to build on? A pre-flight check for TypeSafe System One questions. Runs a fully crossed item x framing x repeat grid against the API and reports what you need settled before putting a model's number behind an if: repeatability (does an identical request give an identical answer), framing sensitivity (does rewording move the answer, and by how much more than plain noise), resolution (can the question tell your items apart, or does the scale saturate into unrankable ties), threshold stability (does a fixed cut-off flap on identical input), and answerability (is it confidently rating things that have nothing to rate). Works on all three primitives — score for rating, noul for gates, choice for routers. The headline is a decision flip rate: if you reran this, how often would the decision your code acts on come out differently? The report adds a nested variance decomposition that separates repeat noise from framing sensitivity from real differences between items, fitted by a Gibbs/slice sampler in TypeScript so it needs no Stan, R or Python, and it withholds the posterior when the convergence gate fails. It does not take your paraphrases on trust: it hashes every request body and refuses to count a perturbation that produced a byte-identical request.

02Models1
@vcjdeboer/jev-reliabilityv2026.09.19.2jev_reliability.ts

Global Arguments

ArgumentTypeDescription
apiKey?stringTypeSafe API key. Prefer a vault reference:
modelstringSystem One model. Pin an exact version for a reproducible study — an
baseUrlstring
timeoutMsnumber
maxRetriesnumber
fn check()
Is this question safe to build on? Give a question and some items;
fn replicate()
Low-level escape hatch: run an explicit item x framing x repeat

Resources

replicate(infinite)— Raw nested replicate grid — one row per (item, framing, repeat).
03Reports1
@vcjdeboer/jev-reliability-reportmodel
jev_reliability_report.ts

Pre-flight verdict for a Jev question. Reads every replicate-* grid

jevtypesafereliabilityvalidation
04Stats
A
100 / 100
Downloads
30
Archive size
36.0 KB
  • Has README or module doc2/2earned
  • README has a code example1/1earned
  • README is substantive1/1earned
  • Most symbols documented1/1earned
  • No slow types (deprecated)1/1earned
  • Dependencies pass trust audit2/2earned
  • Has description1/1earned
  • Platform support declared (or universal)2/2earned
  • License declared1/1earned
  • Verified public repository2/2earned
05Platforms
06Labels