#3212 Hold a signal that arrives before its wait opens

Problem

Part 4 of 6 of keyed signal waits. A signaler that holds a key can send before the run reaches its wait: the event it reports can come first, or the steps ahead of the wait can be slow. With swamp-club#3210 that signal is refused, and the sender has to retry without knowing when a wait will open.

Proposal

- name: identify
  allowFailure: true
  task:
    type: wait_for_signal
    key: kitchen-verdict
    acceptEarly: 10 # seconds a signal may arrive before the wait opens
    timeout: 30
    schema:
      type: object
      required: [verdict]
      properties:
        verdict: { type: string, enum: [known, intruder] }

• acceptEarly is an optional field of a keyed wait_for_signal task: the number of seconds a signal may precede the wait, with a bounded maximum. Without it an early signal is refused as in swamp-club#3210.
• With no open wait under a key whose step sets acceptEarly, a signal by key is stored as the key's pending signal: a write-once record in the control-plane store holding the payload, the receipt and the time it was received. A key holds one pending signal. A second early signal while one is pending and within its age limit is refused, with the stored receipt.
• An early payload is checked on arrival against the schema in the workflow definition, because no run exists yet, and under the payload limits of any signal. A wait that sets acceptEarly may therefore not read inputs.* in its schema, and workflow validate refuses one that does. The payload is checked again against the captured schema when a wait takes it, as an outcome is today.
• A step that opens its wait claims the key (swamp-club#3209), then looks for a pending signal. One received no more than acceptEarly seconds ago that the captured schema allows becomes the wait's outcome, created as any outcome is, and the step succeeds at once with the payload and receipt. The run does not suspend on it. A pending signal past its age limit, or one the schema refuses, is ignored and the wait opens as usual.
• Every reader that settles a wait applies a pending signal before it settles the wait as timed out: outcomeAt in src/domain/workflows/signal_wait_cleanup.ts, which workflow signal, workflow waits, workflow resume and serve's sweep all use.
• Pending signals are numbered per claim and never overwritten or deleted to be replaced, as claims are.
• Pending signals are removed with the run that took them. One that no wait took is removed by the wait-record sweep once it is past its age limit and a grace period.
• A pending signal is plaintext in the store, like an outcome, and must not carry secrets.
• The signal's result says the signal is pending, with the receipt and the time until which it is held.
• New term for the design doc: a key's pending signal. It has an age limit, acceptEarly, which is separate from a wait's deadline.

Protocol to settle before code

This issue carries the races of the series. Each needs its interleavings written out and reviewed before implementation:

1. A signal and a wait arriving at the same moment. The signal writes its pending record and then reads the claim; the step claims the key and then reads the pending record. At least one sees the other, and any later reader finishes the job.
2. A pending signal against a timeout. A wait at its deadline with a pending signal received before that deadline succeeds; it does not time out.
3. Two early signals at once. Exactly one is stored.
4. Two steps opening at once with one pending signal. Only the step that wins the claim may take it.

Open decisions

1. A second early signal while one is pending: refuse it (proposed), or let the newest win. With a refusal an early known followed by an early intruder keeps known.
2. The maximum for acceptEarly, and whether swamp serve may lower it as --max-signal-wait-timeout lowers a wait's timeout.
3. Clock skew. The received time is stamped by one host and its age judged by another. Deadlines have no allowance for skew today; a limit of a few seconds makes the question sharper.
4. Whether a pending signal that passes its age limit unused is audited or dropped silently.
5. Mixed builds. A build from before this issue times out a wait that has a pending signal, and never takes one on opening. Either an upgrade rule in the docs (proposed) or a guard.

Acceptance criteria

• A signal sent before the run reaches its wait, within the limit, is the step's output as soon as the step opens its wait. The run never suspends on that step, and the outputs have the shape of any signalled step's.
• A pending signal older than the limit is ignored: the wait opens and takes a later signal.
• An early payload the schema refuses is refused on arrival and nothing is stored.
• A keyed wait without acceptEarly refuses an early signal exactly as in swamp-club#3210.
• A signal and a wait arriving together: the signal is applied and never lost, in whichever order the writes land.
• A wait at its deadline with a pending signal received before it succeeds.
• Pending records are removed with the run.

Tests

• A property test for the settling decision as a pure function over the claim, the pending signal, the outcome and the time.
• Integration tests in integration/workflow_signal_wait_test.ts with a fake clock and ordered creates, never sleeps, one per interleaving listed above.
• Unit tests beside each changed file, including the validation rule for inputs.* in the schema.

Depends on

swamp-club#3209 and swamp-club#3210.

References

swamp-club#3093 (write-once outcomes), swamp-club#3109 (settling by whoever looks), the Timing out and Signalling paragraphs of the Wait for Signal section in design/primitives/workflows.md.