Relationships
#4102 Remote worker queryData exceeds the fixed 30 s RPC limit at ~2k record versions after @swamp/s3-datastore 2026.10.24.2 (follow-up to #3361)
Opened by vcjdeboer · 10/29/2026· Shipped 11/2/2026
A follow-up to #3361 with measurements, on swamp 20261027.021509.0-sha.f96841b5, one swamp serve with an S3 datastore (synced, format 2) and one remote worker. Two analysis workflow runs execute at the same time.
What changed. We moved @swamp/s3-datastore from 2026.10.01.1 to 2026.10.24.2 (the datastore half of #3941/#3860) and changed nothing else. Since then, one of the two runs fails at a step whose worker queryData hits the fixed 30 s RPC limit (DEFAULT_CALL_TIMEOUT_MS, src/domain/remote/rpc_channel.ts:114; not configurable). This happened in 2 of 2 sittings. The sitting before the move, on the same swamp build, passed.
Measured, from serve's own Handler slow: capability.queryData lines (5 s or more) and the workflow run history:
| before (2026.10.01.1) | after (2026.10.24.2), sitting 1 | after, sitting 2 | |
|---|---|---|---|
| record versions in the datastore | about 3,000 | 1,387 | 2,040 |
full metadata listing (data query 'version >= 0') |
about 5 s | 3.7 s | 4.7 s |
| poller invalidations ("N file(s) updated, invalidating catalog") | 37 | 32 | 15 |
| slow queryData: count, median, max | 4, 6.1 s, 7.7 s | 6, 12.8 s, 42.6 s | 5, 7.9 s, 45.0 s |
| mean time of one workflow run (7 model-method steps) | 224 s | 370 s | 427 s |
The datastore got smaller and the poller invalidated less often, yet the slowest query went from 7.7 s to 45 s. Every step also became 65-90 % slower, including steps that make almost no data queries. A size-driven rebuild does not explain this, since a full listing takes under 5 s.
What we read in the source (not run). In 2026.10.24.2, dirty marks made while a push is in flight survive that push (clearHandledMarks, _lib/s3_cache_sync.ts). With two runs writing at once, this plausibly leaves localDirty set after most pushes, so the next sync takes the slow path (index re-read, walk, compare-and-swap commit) under the sync gate. Separately, a pull or step-lock sync still invalidates the whole catalog (catalog_store.ts invalidate()), and a worker query waits for a full backfillAsync. We suspect a query's rebuild is waiting behind that sync work, but we have not proven it.
Asks:
- Is a 40 s+ catalog wait at about 2k versions expected with 2026.10.24.2 and concurrent writers? If so, can the per-step sync after #3860 take the fast path when only this process's own marks are pending?
- Make the worker's queryData timeout configurable, or let a worker query be answered from the last complete catalog while a rebuild runs (#3361's scoped invalidation would remove most rebuilds).
- We can rerun a sitting with
--log-level debugand send the[s3-sync]and catalog timing lines if that helps.
Shipped
Click a lifecycle step above to view its details.
zack commented 10/30/2026, 8:32:26 PM
We couldn't reproduce a sync slowdown from 2026.10.24.2: with two concurrent runs at about 2k versions, sync got faster and surviving marks didn't affect the pull fast path. Could you rerun a sitting with SWAMP_S3_SYNC_TRACE=1 and --log-level debug, and tell us whether you use a namespace or hydrationStrategy: lazy, and how much CPU the instance has? Please include the [s3-sync] lines and the Handler slow lines around a slow query. The 30 s limit and the full rebuild are confirmed in core, and we'll track those separately.
vcjdeboer commented 10/31/2026, 12:40:06 PM
Thanks, zack. Here is the traced rerun and the answers in one go.
Setup.
- No namespace;
hydrationStrategyis not set, so swamp's default applies. - The instance is an AWS t4g.large: 2 vCPU arm64 (Graviton), 8 GiB, burstable with unlimited CPU credits.
swamp serve, one Docker worker (capped at 2 GiB) and a small proxy share it. - swamp serve runs 20261030.160426.0-sha.a2f815f3 with
@swamp/s3-datastore2026.10.24.2 (unchanged since the report), with one remote worker.
The sitting (2026-10-31, 10:25–12:16 UTC). We set SWAMP_S3_SYNC_TRACE=1 and SWAMP_LOG_LEVEL=debug on serve (the env equivalent of --log-level debug).
1. The 30 s timeout did not reproduce. The concurrent scenario passed again. This is the second sitting in a row on a2f815f3 where it did not reproduce, with the same datastore version that failed twice on f96841b5. So, like you, we cannot reproduce it now, and its cause is still unknown to us. In the traced window (09:59–11:52 UTC):
| value | |
|---|---|
Handler slow: capability.queryData (≥ 5 s) |
8 lines, 5.1–7.9 s, max 7.9 s (it was 42.6 s and 45.0 s in the failing sittings) |
[s3-sync] lines |
2,838 |
| journald suppressed or rate-limited lines | none |
2. Sync phase timings from the trace (all [s3-sync] lines in the window):
| phase | n | mean | max | ≥ 1 s |
|---|---|---|---|---|
pullChanged.pullIndex |
305 | 2.9 s | 16.9 s | 238 |
pullChanged.walk |
305 | 2.4 s | 19.0 s | 170 |
pullChanged.remoteListing |
305 | 1.9 s | 13.6 s | 181 |
preparePush.walk |
98 | 1.6 s | 17.6 s | 43 |
commitPush |
97 | 0.9 s | 15.9 s | 8 |
pullChanged.fastpath |
524 | 0.4 s | 6.7 s | 69 |
- Fast paths:
pullChanged.fastpathhit 218 and missed 306 times.pushChanged.fastpathhit 1 and missed 52 times, andpreparePush.fastpathmissed all 98. - When: the slowest pull phases (13–19 s) are spread over the whole first hour, which was sequential, non-concurrent load. They are not concentrated in the concurrent scenario.
3. Debug-level observation. In the 23 minutes of the robustness scenarios, serve logged 13,274 Query predicate skipped row "<table>"/"<name>" debug lines against 268 handler calls. Mostly the rows were from leases, plus some from pending. It looks as if each worker query walks every catalog row. We are not claiming that is slow, just that it is how the volume looks.
4. A separate finding: swamp serve hits the V8 heap limit. This happened three times in this sitting, and also on 2026-10-30 on both f96841b5 and a2f815f3:
| serve started | Fatal JavaScript out of memory: Ineffective mark-compacts near heap limit |
uptime |
|---|---|---|
| 10:25:13 | 11:22:34 | 57 min |
| 11:22:54 | 11:44:03 | 21 min |
| 11:49:59 (a deliberate restart) | 11:52:19 | 2.3 min |
- The kernel killed nothing, and the box has 8 GiB. serve's cgroup memory was 3.1 GB afterwards.
- Per #636 we're not asking for a bigger heap. We'd like to find what grows.
- I'll file this as its own issue with the journal lines: #4224.
The lines you asked for. For each of the 8 slow queryData calls, here are the [s3-sync] lines in the 60 s before it and the Handler slow line itself. Times are UTC. The host prefix and request ids are removed; nothing else is changed. The complete log for the whole window follows in the next ripples (parts 1-4): every [s3-sync], Handler slow, Handler error and heap-limit line, 2,888 lines.
--- slow query 1: 10:11:38 UTC, [s3-sync] lines in the 60 s before it ---
10:10:38 [s3-sync] pullChanged.walk 4192ms toPull=1
10:10:39 [s3-sync] pullChanged.download 70ms pulled=1
10:10:39 [s3-sync] pullChanged.fastpath 19ms miss
10:10:40 [s3-sync] pullChanged.pullIndex 1179ms
10:10:40 [s3-sync] pullChanged.remoteListing 63ms keys=291
10:10:41 [s3-sync] pullChanged.walk 176ms toPull=0
10:10:41 [s3-sync] pullChanged.download 0ms pulled=0
10:10:41 [s3-sync] preparePush.fastpath 0ms miss
10:10:41 [s3-sync] preparePush.shardAssembly 244ms scoped=16
10:10:41 [s3-sync] preparePush.walk 802ms toPush=28 toDelete=0
10:10:42 [s3-sync] preparePush.upload 148ms pushed=22
10:10:42 [s3-sync] preparePush.delete 0ms deleted=0
10:10:42 [s3-sync] commitPush.shards 327ms dirty=11
10:10:42 [s3-sync] commitPush.localIndex 132ms
10:10:42 [s3-sync] commitPush 482ms shard-first changes=22
10:10:43 [s3-sync] pullChanged.fastpath 25ms miss
10:10:43 [s3-sync] pullChanged.pullIndex 53ms
10:10:46 [s3-sync] pullChanged.remoteListing 2657ms keys=35171
10:10:50 [s3-sync] pullChanged.walk 4058ms toPull=1
10:10:50 [s3-sync] pullChanged.download 40ms pulled=1
10:10:50 [s3-sync] pullChanged.fastpath 23ms miss
10:10:51 [s3-sync] pullChanged.pullIndex 1212ms
10:10:53 [s3-sync] pullChanged.remoteListing 1243ms keys=8622
10:10:54 [s3-sync] pullChanged.walk 1109ms toPull=3
10:10:54 [s3-sync] pullChanged.download 36ms pulled=3
10:10:54 [s3-sync] preparePush.fastpath 0ms miss
10:10:54 [s3-sync] preparePush.shardAssembly 235ms scoped=17
10:10:55 [s3-sync] preparePush.walk 937ms toPush=33 toDelete=0
10:10:55 [s3-sync] preparePush.upload 146ms pushed=24
10:10:55 [s3-sync] preparePush.delete 0ms deleted=0
10:10:56 [s3-sync] commitPush.shards 304ms dirty=8
10:10:56 [s3-sync] commitPush.localIndex 138ms
10:10:56 [s3-sync] commitPush 475ms shard-first changes=24
10:10:56 [s3-sync] pullChanged.fastpath 24ms miss
10:10:56 [s3-sync] pullChanged.pullIndex 64ms
10:10:59 [s3-sync] pullChanged.remoteListing 2533ms keys=35192
10:11:05 [s3-sync] pullChanged.walk 6327ms toPull=1
10:11:05 [s3-sync] pullChanged.download 40ms pulled=1
10:11:08 [s3-sync] pullChanged.fastpath 26ms miss
10:11:09 [s3-sync] pullChanged.pullIndex 1173ms
10:11:09 [s3-sync] pullChanged.remoteListing 62ms keys=291
10:11:09 [s3-sync] pullChanged.walk 144ms toPull=0
10:11:09 [s3-sync] pullChanged.download 0ms pulled=0
10:11:10 [s3-sync] pullChanged.fastpath 20ms miss
10:11:11 [s3-sync] pullChanged.pullIndex 1124ms
10:11:13 [s3-sync] pullChanged.remoteListing 1322ms keys=8640
10:11:14 [s3-sync] pullChanged.walk 1074ms toPull=1
10:11:14 [s3-sync] pullChanged.download 24ms pulled=1
10:11:21 [s3-sync] preparePush.fastpath 0ms miss
10:11:21 [s3-sync] preparePush.shardAssembly 300ms scoped=17
10:11:22 [s3-sync] preparePush.walk 995ms toPush=34 toDelete=0
10:11:22 [s3-sync] preparePush.upload 160ms pushed=25
10:11:22 [s3-sync] preparePush.delete 0ms deleted=0
10:11:22 [s3-sync] commitPush.shards 322ms dirty=9
10:11:23 [s3-sync] commitPush.localIndex 127ms
10:11:23 [s3-sync] commitPush 483ms shard-first changes=25
10:11:23 [s3-sync] pullChanged.fastpath 27ms miss
10:11:23 [s3-sync] pullChanged.pullIndex 62ms
10:11:26 [s3-sync] pullChanged.remoteListing 2947ms keys=35213
10:11:30 [s3-sync] pullChanged.walk 4276ms toPull=1
10:11:30 [s3-sync] pullChanged.download 32ms pulled=1
10:11:38 Handler slow: "capability.queryData" for worker "lab-worker-2" took 5523ms
10:11:38 [s3-sync] preparePush.fastpath 0ms miss
10:11:38 [s3-sync] preparePush.shardAssembly 273ms scoped=17
--- slow query 2: 10:30:48 UTC, [s3-sync] lines in the 60 s before it ---
10:29:49 [s3-sync] pullChanged.walk 4124ms toPull=1
10:29:49 [s3-sync] pullChanged.download 30ms pulled=1
10:29:50 [s3-sync] pullChanged.fastpath 26ms miss
10:29:51 [s3-sync] pullChanged.pullIndex 1207ms
10:29:52 [s3-sync] pullChanged.remoteListing 1016ms keys=8798
10:29:53 [s3-sync] pullChanged.walk 1104ms toPull=3
10:29:53 [s3-sync] pullChanged.download 29ms pulled=3
10:29:53 [s3-sync] preparePush.fastpath 0ms miss
10:29:53 [s3-sync] preparePush.shardAssembly 216ms scoped=14
10:29:54 [s3-sync] preparePush.walk 825ms toPush=25 toDelete=0
10:29:54 [s3-sync] preparePush.upload 113ms pushed=19
10:29:54 [s3-sync] preparePush.delete 0ms deleted=0
10:29:55 [s3-sync] commitPush.shards 362ms dirty=8
10:29:55 [s3-sync] commitPush.localIndex 122ms
10:29:55 [s3-sync] commitPush 518ms shard-first changes=19
10:29:56 [s3-sync] pullChanged.fastpath 30ms miss
10:29:56 [s3-sync] pullChanged.pullIndex 50ms
10:29:59 [s3-sync] pullChanged.remoteListing 2912ms keys=35396
10:30:04 [s3-sync] pullChanged.walk 4778ms toPull=1
10:30:04 [s3-sync] pullChanged.download 37ms pulled=1
10:30:06 [s3-sync] preparePush.fastpath 0ms miss
10:30:06 [s3-sync] preparePush.shardAssembly 262ms scoped=20
10:30:07 [s3-sync] preparePush.walk 967ms toPush=40 toDelete=0
10:30:07 [s3-sync] preparePush.upload 169ms pushed=28
10:30:07 [s3-sync] preparePush.delete 0ms deleted=0
10:30:07 [s3-sync] commitPush.shards 335ms dirty=9
10:30:08 [s3-sync] commitPush.localIndex 136ms
10:30:08 [s3-sync] commitPush 506ms shard-first changes=28
10:30:11 [s3-sync] pullChanged.fastpath 31ms miss
10:30:11 [s3-sync] pullChanged.pullIndex 65ms
10:30:14 [s3-sync] pullChanged.remoteListing 2917ms keys=35420
10:30:18 [s3-sync] pullChanged.walk 4211ms toPull=1
10:30:18 [s3-sync] pullChanged.download 40ms pulled=1
10:30:19 [s3-sync] pullChanged.fastpath 20ms miss
10:30:20 [s3-sync] pullChanged.pullIndex 1201ms
10:30:20 [s3-sync] pullChanged.remoteListing 62ms keys=291
10:30:20 [s3-sync] pullChanged.walk 134ms toPull=0
10:30:20 [s3-sync] pullChanged.download 0ms pulled=0
10:30:21 [s3-sync] pullChanged.fastpath 20ms miss
10:30:22 [s3-sync] pullChanged.pullIndex 1097ms
10:30:23 [s3-sync] pullChanged.remoteListing 1068ms keys=8832
10:30:24 [s3-sync] pullChanged.walk 900ms toPull=1
10:30:24 [s3-sync] pullChanged.download 33ms pulled=1
10:30:27 [s3-sync] preparePush.fastpath 0ms miss
10:30:28 [s3-sync] preparePush.shardAssembly 234ms scoped=17
10:30:28 [s3-sync] preparePush.walk 900ms toPush=34 toDelete=0
10:30:28 [s3-sync] preparePush.upload 152ms pushed=25
10:30:28 [s3-sync] preparePush.delete 0ms deleted=0
10:30:29 [s3-sync] commitPush.shards 335ms dirty=9
10:30:29 [s3-sync] commitPush.localIndex 124ms
10:30:29 [s3-sync] commitPush 479ms shard-first changes=25
10:30:30 [s3-sync] pullChanged.fastpath 25ms miss
10:30:30 [s3-sync] pullChanged.pullIndex 55ms
10:30:33 [s3-sync] pullChanged.remoteListing 2915ms keys=35441
10:30:37 [s3-sync] pullChanged.walk 4197ms toPull=1
10:30:37 [s3-sync] pullChanged.download 37ms pulled=1
10:30:42 [s3-sync] pullChanged.fastpath 76ms miss
10:30:44 [s3-sync] pullChanged.pullIndex 1148ms
10:30:44 [s3-sync] pullChanged.remoteListing 51ms keys=291
10:30:44 [s3-sync] pullChanged.walk 152ms toPull=0
10:30:44 [s3-sync] pullChanged.download 0ms pulled=0
10:30:44 [s3-sync] pullChanged.fastpath 23ms miss
10:30:45 [s3-sync] pullChanged.pullIndex 1138ms
10:30:47 [s3-sync] pullChanged.remoteListing 1251ms keys=8850
10:30:48 Handler slow: "capability.queryData" for worker "lab-worker-2" took 7946ms
--- slow query 3: 10:32:55 UTC, [s3-sync] lines in the 60 s before it ---
10:31:57 [s3-sync] pullChanged.remoteListing 2992ms keys=35544
10:32:01 [s3-sync] pullChanged.walk 4371ms toPull=1
10:32:01 [s3-sync] pullChanged.download 47ms pulled=1
10:32:12 [s3-sync] pullChanged.fastpath 41ms miss
10:32:13 [s3-sync] pullChanged.pullIndex 1158ms
10:32:14 [s3-sync] pullChanged.remoteListing 52ms keys=291
10:32:14 [s3-sync] pullChanged.walk 149ms toPull=0
10:32:14 [s3-sync] pullChanged.download 0ms pulled=0
10:32:14 [s3-sync] pullChanged.fastpath 23ms miss
10:32:15 [s3-sync] pullChanged.pullIndex 1109ms
10:32:17 [s3-sync] pullChanged.remoteListing 1002ms keys=8938
10:32:20 [s3-sync] pullChanged.walk 3298ms toPull=1
10:32:20 [s3-sync] pullChanged.download 40ms pulled=1
10:32:20 [s3-sync] preparePush.fastpath 0ms miss
10:32:20 [s3-sync] preparePush.shardAssembly 226ms scoped=17
10:32:21 [s3-sync] preparePush.walk 880ms toPush=33 toDelete=0
10:32:22 [s3-sync] preparePush.upload 646ms pushed=24
10:32:22 [s3-sync] preparePush.delete 0ms deleted=0
10:32:22 [s3-sync] commitPush.shards 317ms dirty=8
10:32:22 [s3-sync] commitPush.localIndex 126ms
10:32:22 [s3-sync] commitPush 466ms shard-first changes=24
10:32:23 [s3-sync] pullChanged.fastpath 27ms miss
10:32:23 [s3-sync] pullChanged.pullIndex 71ms
10:32:26 [s3-sync] pullChanged.remoteListing 2925ms keys=35565
10:32:30 [s3-sync] pullChanged.walk 4233ms toPull=1
10:32:30 [s3-sync] pullChanged.download 33ms pulled=1
10:32:36 [s3-sync] preparePush.fastpath 0ms miss
10:32:37 [s3-sync] preparePush.shardAssembly 256ms scoped=35
10:32:38 [s3-sync] preparePush.walk 1153ms toPush=70 toDelete=0
10:32:38 [s3-sync] preparePush.upload 235ms pushed=43
10:32:38 [s3-sync] preparePush.delete 0ms deleted=0
10:32:38 [s3-sync] commitPush.shards 335ms dirty=9
10:32:38 [s3-sync] commitPush.localIndex 136ms
10:32:38 [s3-sync] commitPush 501ms shard-first changes=43
10:32:39 [s3-sync] pullChanged.fastpath 30ms miss
10:32:39 [s3-sync] pullChanged.pullIndex 60ms
10:32:42 [s3-sync] pullChanged.remoteListing 2470ms keys=35604
10:32:46 [s3-sync] pullChanged.walk 4483ms toPull=1
10:32:46 [s3-sync] pullChanged.download 50ms pulled=1
10:32:47 [s3-sync] pullChanged.fastpath 28ms miss
10:32:48 [s3-sync] pullChanged.pullIndex 1175ms
10:32:49 [s3-sync] pullChanged.remoteListing 57ms keys=291
10:32:49 [s3-sync] pullChanged.walk 165ms toPull=0
10:32:49 [s3-sync] pullChanged.download 0ms pulled=0
10:32:49 [s3-sync] pullChanged.fastpath 23ms miss
10:32:53 [s3-sync] pullChanged.pullIndex 3227ms
10:32:54 [s3-sync] pullChanged.remoteListing 1149ms keys=8992
10:32:55 Handler slow: "capability.queryData" for worker "lab-worker-2" took 7149ms
--- slow query 4: 10:38:03 UTC, [s3-sync] lines in the 60 s before it ---
10:37:11 [s3-sync] pullChanged.fastpath 39ms miss
10:37:11 [s3-sync] pullChanged.pullIndex 52ms
10:37:16 [s3-sync] pullChanged.remoteListing 5162ms keys=35880
10:37:20 [s3-sync] pullChanged.walk 4208ms toPull=1
10:37:20 [s3-sync] pullChanged.download 38ms pulled=1
10:37:21 [s3-sync] pullChanged.fastpath 23ms miss
10:37:22 [s3-sync] pullChanged.pullIndex 1241ms
10:37:23 [s3-sync] pullChanged.remoteListing 65ms keys=291
10:37:23 [s3-sync] pullChanged.walk 149ms toPull=0
10:37:23 [s3-sync] pullChanged.download 0ms pulled=0
10:37:23 [s3-sync] pullChanged.fastpath 34ms miss
10:37:24 [s3-sync] pullChanged.pullIndex 1156ms
10:37:26 [s3-sync] pullChanged.remoteListing 1149ms keys=9229
10:37:27 [s3-sync] pullChanged.walk 1090ms toPull=1
10:37:27 [s3-sync] pullChanged.download 26ms pulled=1
10:37:32 [s3-sync] preparePush.fastpath 0ms miss
10:37:32 [s3-sync] preparePush.shardAssembly 224ms scoped=17
10:37:33 [s3-sync] preparePush.walk 881ms toPush=34 toDelete=0
10:37:33 [s3-sync] preparePush.upload 150ms pushed=25
10:37:33 [s3-sync] preparePush.delete 0ms deleted=0
10:37:33 [s3-sync] commitPush.shards 328ms dirty=9
10:37:33 [s3-sync] commitPush.localIndex 142ms
10:37:33 [s3-sync] commitPush 526ms shard-first changes=25
10:37:44 [s3-sync] pullChanged.fastpath 53ms miss
10:37:48 [s3-sync] pullChanged.pullIndex 3464ms
10:37:48 [s3-sync] pullChanged.remoteListing 68ms keys=291
10:37:48 [s3-sync] pullChanged.walk 162ms toPull=0
10:37:48 [s3-sync] pullChanged.download 0ms pulled=0
10:37:48 [s3-sync] pullChanged.fastpath 22ms miss
10:37:48 [s3-sync] pullChanged.pullIndex 63ms
10:37:51 [s3-sync] pullChanged.remoteListing 2621ms keys=35901
10:37:55 [s3-sync] pullChanged.walk 4211ms toPull=1
10:37:55 [s3-sync] pullChanged.download 41ms pulled=1
10:37:56 [s3-sync] pullChanged.fastpath 25ms miss
10:37:57 [s3-sync] pullChanged.pullIndex 1191ms
10:37:58 [s3-sync] pullChanged.remoteListing 1273ms keys=9247
10:38:00 [s3-sync] pullChanged.walk 1632ms toPull=1
10:38:00 [s3-sync] pullChanged.download 34ms pulled=1
10:38:03 Handler slow: "capability.queryData" for worker "lab-worker-2" took 6018ms
--- slow query 5: 10:39:55 UTC, [s3-sync] lines in the 60 s before it ---
10:39:00 [s3-sync] pullChanged.walk 6443ms toPull=1
10:39:00 [s3-sync] pullChanged.download 55ms pulled=1
10:39:01 [s3-sync] pullChanged.fastpath 27ms miss
10:39:02 [s3-sync] pullChanged.pullIndex 1193ms
10:39:03 [s3-sync] pullChanged.remoteListing 1339ms keys=9292
10:39:05 [s3-sync] pullChanged.walk 1604ms toPull=1
10:39:05 [s3-sync] pullChanged.download 25ms pulled=1
10:39:15 [s3-sync] pullChanged.fastpath 63ms miss
10:39:16 [s3-sync] pullChanged.pullIndex 1202ms
10:39:16 [s3-sync] pullChanged.remoteListing 62ms keys=291
10:39:16 [s3-sync] pullChanged.walk 140ms toPull=0
10:39:16 [s3-sync] pullChanged.download 0ms pulled=0
10:39:17 [s3-sync] pullChanged.fastpath 31ms miss
10:39:18 [s3-sync] pullChanged.pullIndex 1174ms
10:39:19 [s3-sync] pullChanged.remoteListing 1100ms keys=9292
10:39:20 [s3-sync] pullChanged.walk 978ms toPull=0
10:39:20 [s3-sync] pullChanged.download 0ms pulled=0
10:39:26 [s3-sync] preparePush.fastpath 0ms miss
10:39:26 [s3-sync] preparePush.shardAssembly 333ms scoped=17
10:39:27 [s3-sync] preparePush.walk 1035ms toPush=34 toDelete=0
10:39:27 [s3-sync] preparePush.upload 167ms pushed=25
10:39:27 [s3-sync] preparePush.delete 0ms deleted=0
10:39:29 [s3-sync] commitPush.shards 2344ms dirty=9
10:39:29 [s3-sync] commitPush.localIndex 140ms
10:39:29 [s3-sync] commitPush 2515ms shard-first changes=25
10:39:41 [s3-sync] pullChanged.fastpath 40ms miss
10:39:41 [s3-sync] pullChanged.pullIndex 61ms
10:39:44 [s3-sync] pullChanged.remoteListing 2528ms keys=35973
10:39:48 [s3-sync] pullChanged.walk 4327ms toPull=1
10:39:48 [s3-sync] pullChanged.download 50ms pulled=1
10:39:49 [s3-sync] pullChanged.fastpath 27ms miss
10:39:50 [s3-sync] pullChanged.pullIndex 1259ms
10:39:50 [s3-sync] pullChanged.remoteListing 47ms keys=291
10:39:50 [s3-sync] pullChanged.walk 177ms toPull=0
10:39:50 [s3-sync] pullChanged.download 0ms pulled=0
10:39:51 [s3-sync] pullChanged.fastpath 35ms miss
10:39:52 [s3-sync] pullChanged.pullIndex 1145ms
10:39:53 [s3-sync] pullChanged.remoteListing 1106ms keys=9310
10:39:55 Handler slow: "capability.queryData" for worker "lab-worker-2" took 5076ms
10:39:55 [s3-sync] pullChanged.walk 2226ms toPull=1
10:39:55 [s3-sync] pullChanged.download 24ms pulled=1
--- slow query 6: 10:41:04 UTC, [s3-sync] lines in the 60 s before it ---
10:40:04 [s3-sync] preparePush.upload 865ms pushed=43
10:40:04 [s3-sync] preparePush.delete 0ms deleted=0
10:40:04 [s3-sync] commitPush.shards 345ms dirty=9
10:40:04 [s3-sync] commitPush.localIndex 131ms
10:40:04 [s3-sync] commitPush 502ms shard-first changes=43
10:40:15 [s3-sync] pullChanged.fastpath 47ms miss
10:40:17 [s3-sync] pullChanged.pullIndex 1282ms
10:40:17 [s3-sync] pullChanged.remoteListing 43ms keys=291
10:40:17 [s3-sync] pullChanged.walk 150ms toPull=0
10:40:17 [s3-sync] pullChanged.download 0ms pulled=0
10:40:17 [s3-sync] pullChanged.fastpath 27ms miss
10:40:17 [s3-sync] pullChanged.pullIndex 59ms
10:40:19 [s3-sync] pullChanged.remoteListing 2289ms keys=36012
10:40:24 [s3-sync] pullChanged.walk 4190ms toPull=1
10:40:24 [s3-sync] pullChanged.download 33ms pulled=1
10:40:24 [s3-sync] pullChanged.fastpath 33ms miss
10:40:26 [s3-sync] pullChanged.pullIndex 1179ms
10:40:27 [s3-sync] pullChanged.remoteListing 1372ms keys=9346
10:40:28 [s3-sync] pullChanged.walk 1170ms toPull=3
10:40:28 [s3-sync] pullChanged.download 31ms pulled=3
10:40:28 [s3-sync] preparePush.fastpath 0ms miss
10:40:29 [s3-sync] preparePush.shardAssembly 231ms scoped=20
10:40:29 [s3-sync] preparePush.walk 945ms toPush=39 toDelete=0
10:40:30 [s3-sync] preparePush.upload 179ms pushed=27
10:40:30 [s3-sync] preparePush.delete 0ms deleted=0
10:40:30 [s3-sync] commitPush.shards 339ms dirty=8
10:40:30 [s3-sync] commitPush.localIndex 164ms
10:40:30 [s3-sync] commitPush 536ms shard-first changes=27
10:40:45 [s3-sync] pullChanged.fastpath 103ms miss
10:40:47 [s3-sync] pullChanged.pullIndex 1221ms
10:40:47 [s3-sync] pullChanged.remoteListing 70ms keys=291
10:40:47 [s3-sync] pullChanged.walk 151ms toPull=0
10:40:47 [s3-sync] pullChanged.download 0ms pulled=0
10:40:47 [s3-sync] pullChanged.fastpath 30ms miss
10:40:47 [s3-sync] pullChanged.pullIndex 62ms
10:40:50 [s3-sync] pullChanged.remoteListing 2474ms keys=36036
10:40:56 [s3-sync] pullChanged.walk 6120ms toPull=1
10:40:56 [s3-sync] pullChanged.download 42ms pulled=1
10:40:56 [s3-sync] pullChanged.fastpath 23ms miss
10:40:58 [s3-sync] pullChanged.pullIndex 1225ms
10:40:59 [s3-sync] pullChanged.remoteListing 1109ms keys=9367
10:41:01 [s3-sync] pullChanged.walk 1668ms toPull=1
10:41:01 [s3-sync] pullChanged.download 28ms pulled=1
10:41:04 Handler slow: "capability.queryData" for worker "lab-worker-2" took 5542ms
--- slow query 7: 10:42:22 UTC, [s3-sync] lines in the 60 s before it ---
10:41:45 [s3-sync] preparePush.fastpath 0ms miss
10:41:45 [s3-sync] preparePush.shardAssembly 292ms scoped=17
10:41:46 [s3-sync] preparePush.walk 990ms toPush=34 toDelete=0
10:41:46 [s3-sync] preparePush.upload 154ms pushed=25
10:41:46 [s3-sync] preparePush.delete 0ms deleted=0
10:41:47 [s3-sync] commitPush.shards 329ms dirty=9
10:41:47 [s3-sync] commitPush.localIndex 134ms
10:41:47 [s3-sync] commitPush 495ms shard-first changes=25
10:41:49 [s3-sync] pullChanged.fastpath 1750ms miss
10:41:59 [s3-sync] pullChanged.pullIndex 9783ms
10:41:59 [s3-sync] pullChanged.remoteListing 80ms keys=291
10:42:00 [s3-sync] pullChanged.walk 260ms toPull=0
10:42:00 [s3-sync] pullChanged.download 0ms pulled=0
10:42:00 [s3-sync] pullChanged.fastpath 25ms miss
10:42:00 [s3-sync] pullChanged.pullIndex 66ms
10:42:03 [s3-sync] pullChanged.remoteListing 2533ms keys=36057
10:42:07 [s3-sync] pullChanged.walk 4288ms toPull=1
10:42:07 [s3-sync] pullChanged.download 33ms pulled=1
10:42:07 [s3-sync] pullChanged.fastpath 19ms miss
10:42:09 [s3-sync] pullChanged.pullIndex 1234ms
10:42:10 [s3-sync] pullChanged.remoteListing 1358ms keys=9385
10:42:12 [s3-sync] pullChanged.walk 1722ms toPull=1
10:42:12 [s3-sync] pullChanged.download 31ms pulled=1
10:42:16 [s3-sync] pullChanged.fastpath 78ms miss
10:42:17 [s3-sync] pullChanged.pullIndex 1138ms
10:42:18 [s3-sync] pullChanged.remoteListing 56ms keys=291
10:42:18 [s3-sync] pullChanged.walk 175ms toPull=0
10:42:18 [s3-sync] pullChanged.download 0ms pulled=0
10:42:18 [s3-sync] pullChanged.fastpath 29ms miss
10:42:19 [s3-sync] pullChanged.pullIndex 1124ms
10:42:21 [s3-sync] pullChanged.remoteListing 1202ms keys=9385
10:42:22 Handler slow: "capability.queryData" for worker "lab-worker-2" took 7859ms
--- slow query 8: 11:44:43 UTC, [s3-sync] lines in the 60 s before it ---
11:43:45 [s3-sync] preparePush.fastpath 0ms miss
11:43:45 [s3-sync] preparePush.shardAssembly 288ms scoped=17
11:43:46 [s3-sync] preparePush.walk 1018ms toPush=34 toDelete=0
11:43:46 [s3-sync] preparePush.upload 150ms pushed=25
11:43:46 [s3-sync] preparePush.delete 0ms deleted=0
11:43:46 [s3-sync] commitPush.shards 325ms dirty=9
11:43:46 [s3-sync] commitPush.localIndex 137ms
11:43:46 [s3-sync] commitPush 499ms shard-first changes=25
11:43:52 [s3-sync] pullChanged.fastpath 31ms miss
11:44:17 [s3-sync] pullChanged.fastpath 40ms miss
11:44:19 [s3-sync] pullChanged.pullIndex 1541ms
11:44:22 [s3-sync] pullChanged.remoteListing 3034ms keys=37503
11:44:25 [s3-sync] pullChanged.walk 3436ms toPull=1
11:44:25 [s3-sync] pullChanged.download 32ms pulled=1
11:44:43 Handler slow: "capability.queryData" for worker "lab-worker-2" took 5722msThanks for tracking the 30 s limit and the full rebuild in core. Could you link those issues here?
vcjdeboer commented 10/31/2026, 12:40:31 PM
Full [s3-sync] / Handler slow / Handler error / heap-limit log of the traced sitting, part 1 of 4 (11:22:34-10:31:50 UTC):
11:22:34 # Fatal JavaScript out of memory: Ineffective mark-compacts near heap limit
11:44:03 # Fatal JavaScript out of memory: Ineffective mark-compacts near heap limit
11:52:19 # Fatal JavaScript out of memory: Ineffective mark-compacts near heap limit
10:01:19 [s3-sync] pullChanged.fastpath 38ms hit
10:01:23 [s3-sync] pullChanged.fastpath 28ms hit
10:01:27 [s3-sync] pushChanged.fastpath 0ms miss
10:01:28 [s3-sync] pushChanged.shardAssembly 404ms scoped=17
10:01:29 [s3-sync] pushChanged.walk 723ms toPush=7 toDelete=19
10:01:29 [s3-sync] pushChanged.upload 69ms pushed=7
10:01:29 [s3-sync] pushChanged.delete 74ms deleted=19
10:01:29 [s3-sync] pushChanged.v2writeback 530ms
10:01:30 [s3-sync] pullChanged.fastpath 27ms hit
10:01:30 [s3-sync] pushChanged.fastpath 0ms miss
10:01:30 [s3-sync] pushChanged.shardAssembly 166ms scoped=13
10:01:30 [s3-sync] pushChanged.walk 4ms toPush=0 toDelete=19
10:01:30 [s3-sync] pushChanged.upload 1ms pushed=0
10:01:30 [s3-sync] pushChanged.delete 42ms deleted=19
10:01:30 [s3-sync] pushChanged.v2writeback 369ms
10:01:50 [s3-sync] pullChanged.fastpath 53ms miss
10:01:52 [s3-sync] pullChanged.pullIndex 1337ms
10:01:52 [s3-sync] pullChanged.remoteListing 78ms keys=291
10:01:52 [s3-sync] pullChanged.walk 152ms toPull=0
10:01:52 [s3-sync] pullChanged.download 0ms pulled=0
10:01:52 [s3-sync] pullChanged.fastpath 26ms miss
10:01:54 [s3-sync] pullChanged.pullIndex 1306ms
10:01:55 [s3-sync] pullChanged.remoteListing 1370ms keys=8423
10:01:56 [s3-sync] pullChanged.walk 1027ms toPull=0
10:01:56 [s3-sync] pullChanged.download 0ms pulled=0
10:01:56 [s3-sync] pullChanged.fastpath 31ms miss
10:01:59 [s3-sync] pullChanged.pullIndex 3013ms
10:02:03 [s3-sync] pullChanged.remoteListing 2909ms keys=34932
10:02:07 [s3-sync] pullChanged.walk 4244ms toPull=0
10:02:07 [s3-sync] pullChanged.download 0ms pulled=0
10:02:07 [s3-sync] pushChanged.fastpath 0ms miss
10:02:07 [s3-sync] pushChanged.shardAssembly 144ms scoped=5
10:02:07 [s3-sync] pushChanged.walk 14ms toPush=10 toDelete=0
10:02:07 [s3-sync] pushChanged.upload 79ms pushed=10
10:02:07 [s3-sync] pushChanged.delete 0ms deleted=0
10:02:08 [s3-sync] pushChanged.v2writeback 294ms
10:02:16 [s3-sync] pullChanged.fastpath 32ms hit
10:02:24 [s3-sync] pullChanged.fastpath 3930ms hit
10:02:24 [s3-sync] pushChanged.fastpath 0ms miss
10:02:25 [s3-sync] pushChanged.shardAssembly 173ms scoped=13
10:02:25 [s3-sync] pushChanged.walk 34ms toPush=0 toDelete=19
10:02:25 [s3-sync] pushChanged.upload 0ms pushed=0
10:02:25 [s3-sync] pushChanged.delete 111ms deleted=19
10:02:25 [s3-sync] pushChanged.v2writeback 516ms
10:02:47 [s3-sync] pullChanged.fastpath 49ms miss
10:02:48 [s3-sync] pullChanged.pullIndex 1430ms
10:02:49 [s3-sync] pullChanged.remoteListing 70ms keys=291
10:02:49 [s3-sync] pullChanged.walk 146ms toPull=0
10:02:49 [s3-sync] pullChanged.download 0ms pulled=0
10:02:49 [s3-sync] pullChanged.fastpath 30ms miss
10:02:50 [s3-sync] pullChanged.pullIndex 1149ms
10:02:52 [s3-sync] pullChanged.remoteListing 1306ms keys=8423
10:02:52 [s3-sync] pullChanged.walk 876ms toPull=0
10:02:52 [s3-sync] pullChanged.download 0ms pulled=0
10:02:54 [s3-sync] pullChanged.fastpath 29ms miss
10:02:55 [s3-sync] pullChanged.pullIndex 1057ms
10:03:00 [s3-sync] pullChanged.remoteListing 3002ms keys=34932
10:03:04 [s3-sync] pullChanged.walk 4260ms toPull=0
10:03:04 [s3-sync] pullChanged.download 0ms pulled=0
10:03:04 [s3-sync] pushChanged.fastpath 0ms miss
10:03:04 [s3-sync] pushChanged.shardAssembly 153ms scoped=5
10:03:04 [s3-sync] pushChanged.walk 8ms toPush=10 toDelete=0
10:03:04 [s3-sync] pushChanged.upload 92ms pushed=10
10:03:04 [s3-sync] pushChanged.delete 0ms deleted=0
10:03:05 [s3-sync] pushChanged.v2writeback 293ms
10:03:17 [s3-sync] pullChanged.fastpath 46ms hit
10:03:18 [s3-sync] pullChanged.fastpath 22ms hit
10:03:47 [s3-sync] pullChanged.fastpath 51ms miss
10:03:48 [s3-sync] pullChanged.pullIndex 1437ms
10:03:49 [s3-sync] pullChanged.remoteListing 78ms keys=291
10:03:49 [s3-sync] pullChanged.walk 187ms toPull=0
10:03:49 [s3-sync] pullChanged.download 0ms pulled=0
10:03:49 [s3-sync] pullChanged.fastpath 20ms miss
10:03:50 [s3-sync] pullChanged.pullIndex 1115ms
10:03:51 [s3-sync] pullChanged.remoteListing 1204ms keys=8454
10:03:52 [s3-sync] pullChanged.walk 813ms toPull=0
10:03:52 [s3-sync] pullChanged.download 0ms pulled=0
10:04:11 [s3-sync] pullChanged.fastpath 28ms miss
10:04:12 [s3-sync] pullChanged.pullIndex 1104ms
10:04:15 [s3-sync] pullChanged.remoteListing 2572ms keys=34969
10:04:19 [s3-sync] pullChanged.walk 4119ms toPull=2
10:04:19 [s3-sync] pullChanged.download 41ms pulled=2
10:04:19 [s3-sync] pushChanged.fastpath 0ms miss
10:04:19 [s3-sync] pushChanged.shardAssembly 289ms scoped=16
10:04:20 [s3-sync] pushChanged.walk 510ms toPush=27 toDelete=0
10:04:20 [s3-sync] pushChanged.upload 156ms pushed=27
10:04:20 [s3-sync] pushChanged.delete 0ms deleted=0
10:04:20 [s3-sync] pushChanged.v2writeback 517ms
10:04:21 [s3-sync] pullChanged.fastpath 25ms hit
10:04:22 [s3-sync] pullChanged.fastpath 30ms hit
10:04:47 [s3-sync] pullChanged.fastpath 52ms hit
10:04:48 [s3-sync] pullChanged.fastpath 26ms hit
10:05:11 [s3-sync] pullChanged.fastpath 25ms miss
10:05:12 [s3-sync] pullChanged.pullIndex 1178ms
10:05:14 [s3-sync] pullChanged.remoteListing 2573ms keys=35005
10:05:18 [s3-sync] pullChanged.walk 4009ms toPull=0
10:05:18 [s3-sync] pullChanged.download 0ms pulled=0
10:05:19 [s3-sync] pushChanged.fastpath 0ms miss
10:05:19 [s3-sync] pushChanged.shardAssembly 165ms scoped=5
10:05:19 [s3-sync] pushChanged.walk 6ms toPush=10 toDelete=0
10:05:19 [s3-sync] pushChanged.upload 84ms pushed=10
10:05:19 [s3-sync] pushChanged.delete 0ms deleted=0
10:05:19 [s3-sync] pushChanged.v2writeback 297ms
10:05:20 [s3-sync] pullChanged.fastpath 34ms hit
10:05:21 [s3-sync] pullChanged.fastpath 26ms hit
10:05:47 [s3-sync] pullChanged.fastpath 43ms hit
10:05:48 [s3-sync] pullChanged.fastpath 37ms hit
10:06:37 [s3-sync] pullChanged.fastpath 30ms hit
10:06:45 [s3-sync] pullChanged.fastpath 3924ms hit
10:06:45 [s3-sync] pushChanged.fastpath 0ms miss
10:06:45 [s3-sync] pushChanged.shardAssembly 281ms scoped=17
10:06:46 [s3-sync] pushChanged.walk 707ms toPush=7 toDelete=19
10:06:46 [s3-sync] pushChanged.upload 72ms pushed=7
10:06:46 [s3-sync] pushChanged.delete 78ms deleted=19
10:06:47 [s3-sync] pushChanged.v2writeback 479ms
10:06:47 [s3-sync] pullChanged.fastpath 31ms hit
10:06:47 [s3-sync] pushChanged.fastpath 0ms miss
10:06:47 [s3-sync] pushChanged.shardAssembly 153ms scoped=13
10:06:47 [s3-sync] pushChanged.walk 5ms toPush=0 toDelete=19
10:06:47 [s3-sync] pushChanged.upload 0ms pushed=0
10:06:47 [s3-sync] pushChanged.delete 67ms deleted=19
10:06:48 [s3-sync] pushChanged.v2writeback 321ms
10:06:48 [s3-sync] pullChanged.fastpath 25ms hit
10:06:48 [s3-sync] pushChanged.fastpath 0ms miss
10:06:48 [s3-sync] pushChanged.shardAssembly 142ms scoped=13
10:06:48 [s3-sync] pushChanged.walk 4ms toPush=0 toDelete=19
10:06:48 [s3-sync] pushChanged.upload 0ms pushed=0
10:06:48 [s3-sync] pushChanged.delete 46ms deleted=19
10:06:49 [s3-sync] pushChanged.v2writeback 357ms
10:07:08 [s3-sync] pullChanged.fastpath 58ms miss
10:07:09 [s3-sync] pullChanged.pullIndex 1256ms
10:07:10 [s3-sync] pullChanged.remoteListing 76ms keys=291
10:07:10 [s3-sync] pullChanged.walk 187ms toPull=0
10:07:10 [s3-sync] pullChanged.download 0ms pulled=0
10:07:10 [s3-sync] pullChanged.fastpath 27ms miss
10:07:11 [s3-sync] pullChanged.pullIndex 1126ms
10:07:13 [s3-sync] pullChanged.remoteListing 1333ms keys=8457
10:07:14 [s3-sync] pullChanged.walk 1145ms toPull=0
10:07:14 [s3-sync] pullChanged.download 0ms pulled=0
10:07:14 [s3-sync] pullChanged.fastpath 27ms miss
10:07:17 [s3-sync] pullChanged.pullIndex 3170ms
10:07:20 [s3-sync] pullChanged.remoteListing 2478ms keys=34973
10:07:24 [s3-sync] pullChanged.walk 4127ms toPull=0
10:07:24 [s3-sync] pullChanged.download 0ms pulled=0
10:07:24 [s3-sync] pushChanged.fastpath 0ms miss
10:07:24 [s3-sync] pushChanged.shardAssembly 127ms scoped=5
10:07:24 [s3-sync] pushChanged.walk 15ms toPush=10 toDelete=0
10:07:25 [s3-sync] pushChanged.upload 87ms pushed=10
10:07:25 [s3-sync] pushChanged.delete 0ms deleted=0
10:07:25 [s3-sync] pushChanged.v2writeback 281ms
10:07:38 [s3-sync] pullChanged.fastpath 48ms hit
10:07:39 [s3-sync] pullChanged.fastpath 20ms hit
10:08:08 [s3-sync] pullChanged.fastpath 29ms miss
10:08:09 [s3-sync] pullChanged.pullIndex 1348ms
10:08:10 [s3-sync] pullChanged.remoteListing 84ms keys=291
10:08:10 [s3-sync] pullChanged.walk 152ms toPull=0
10:08:10 [s3-sync] pullChanged.download 0ms pulled=0
10:08:10 [s3-sync] pullChanged.fastpath 30ms miss
10:08:11 [s3-sync] pullChanged.pullIndex 1186ms
10:08:13 [s3-sync] pullChanged.remoteListing 1303ms keys=8488
10:08:14 [s3-sync] pullChanged.walk 930ms toPull=2
10:08:14 [s3-sync] pullChanged.download 41ms pulled=2
10:08:17 [s3-sync] pullChanged.fastpath 26ms miss
10:08:18 [s3-sync] pullChanged.pullIndex 1134ms
10:08:23 [s3-sync] pullChanged.remoteListing 4858ms keys=35010
10:08:27 [s3-sync] pullChanged.walk 3934ms toPull=0
10:08:27 [s3-sync] pullChanged.download 0ms pulled=0
10:08:27 [s3-sync] pushChanged.fastpath 0ms miss
10:08:27 [s3-sync] pushChanged.shardAssembly 277ms scoped=16
10:08:28 [s3-sync] pushChanged.walk 483ms toPush=27 toDelete=0
10:08:28 [s3-sync] pushChanged.upload 150ms pushed=27
10:08:28 [s3-sync] pushChanged.delete 0ms deleted=0
10:08:28 [s3-sync] pushChanged.v2writeback 437ms
10:08:38 [s3-sync] pullChanged.fastpath 132ms hit
10:08:39 [s3-sync] pullChanged.fastpath 31ms hit
10:09:02 [s3-sync] pullChanged.fastpath 23ms miss
10:09:03 [s3-sync] pullChanged.pullIndex 60ms
10:09:06 [s3-sync] pullChanged.remoteListing 3054ms keys=35047
10:09:10 [s3-sync] pullChanged.walk 3981ms toPull=1
10:09:10 [s3-sync] pullChanged.download 40ms pulled=1
10:09:10 [s3-sync] pullChanged.fastpath 22ms miss
10:09:11 [s3-sync] pullChanged.pullIndex 1181ms
10:09:11 [s3-sync] pullChanged.remoteListing 58ms keys=291
10:09:11 [s3-sync] pullChanged.walk 144ms toPull=0
10:09:11 [s3-sync] pullChanged.download 0ms pulled=0
10:09:12 [s3-sync] pullChanged.fastpath 21ms miss
10:09:13 [s3-sync] pullChanged.pullIndex 1088ms
10:09:14 [s3-sync] pullChanged.remoteListing 1046ms keys=8520
10:09:15 [s3-sync] pullChanged.walk 1062ms toPull=3
10:09:15 [s3-sync] pullChanged.download 40ms pulled=3
10:09:15 [s3-sync] preparePush.fastpath 0ms miss
10:09:16 [s3-sync] preparePush.shardAssembly 226ms scoped=16
10:09:16 [s3-sync] preparePush.walk 831ms toPush=27 toDelete=0
10:09:16 [s3-sync] preparePush.upload 139ms pushed=21
10:09:16 [s3-sync] preparePush.delete 0ms deleted=0
10:09:17 [s3-sync] commitPush.shards 342ms dirty=10
10:09:17 [s3-sync] commitPush.localIndex 133ms
10:09:17 [s3-sync] commitPush 511ms shard-first changes=21
10:09:18 [s3-sync] pullChanged.fastpath 33ms miss
10:09:18 [s3-sync] pullChanged.pullIndex 59ms
10:09:20 [s3-sync] pullChanged.remoteListing 2732ms keys=35065
10:09:26 [s3-sync] pullChanged.walk 5990ms toPull=1
10:09:26 [s3-sync] pullChanged.download 34ms pulled=1
10:09:29 [s3-sync] preparePush.fastpath 0ms miss
10:09:29 [s3-sync] preparePush.shardAssembly 240ms scoped=17
10:09:29 [s3-sync] preparePush.walk 869ms toPush=34 toDelete=0
10:09:30 [s3-sync] preparePush.upload 148ms pushed=25
10:09:30 [s3-sync] preparePush.delete 0ms deleted=0
10:09:30 [s3-sync] commitPush.shards 324ms dirty=9
10:09:30 [s3-sync] commitPush.localIndex 136ms
10:09:30 [s3-sync] commitPush 492ms shard-first changes=25
10:09:31 [s3-sync] pullChanged.fastpath 34ms miss
10:09:31 [s3-sync] pullChanged.pullIndex 57ms
10:09:34 [s3-sync] pullChanged.remoteListing 3109ms keys=35086
10:09:38 [s3-sync] pullChanged.walk 3988ms toPull=1
10:09:38 [s3-sync] pullChanged.download 27ms pulled=1
10:09:39 [s3-sync] pullChanged.fastpath 17ms miss
10:09:40 [s3-sync] pullChanged.pullIndex 1185ms
10:09:40 [s3-sync] pullChanged.remoteListing 61ms keys=291
10:09:41 [s3-sync] pullChanged.walk 144ms toPull=0
10:09:41 [s3-sync] pullChanged.download 0ms pulled=0
10:09:41 [s3-sync] pullChanged.fastpath 30ms miss
10:09:42 [s3-sync] pullChanged.pullIndex 1330ms
10:09:43 [s3-sync] pullChanged.remoteListing 1015ms keys=8551
10:09:44 [s3-sync] pullChanged.walk 838ms toPull=1
10:09:44 [s3-sync] pullChanged.download 35ms pulled=1
10:09:46 [s3-sync] preparePush.fastpath 0ms miss
10:09:46 [s3-sync] preparePush.shardAssembly 208ms scoped=17
10:09:47 [s3-sync] preparePush.walk 843ms toPush=34 toDelete=0
10:09:47 [s3-sync] preparePush.upload 169ms pushed=25
10:09:47 [s3-sync] preparePush.delete 0ms deleted=0
10:09:48 [s3-sync] commitPush.shards 343ms dirty=9
10:09:48 [s3-sync] commitPush.localIndex 118ms
10:09:48 [s3-sync] commitPush 499ms shard-first changes=25
10:09:48 [s3-sync] pullChanged.fastpath 22ms miss
10:09:48 [s3-sync] pullChanged.pullIndex 74ms
10:09:51 [s3-sync] pullChanged.remoteListing 2997ms keys=35107
10:09:55 [s3-sync] pullChanged.walk 3967ms toPull=1
10:09:55 [s3-sync] pullChanged.download 29ms pulled=1
10:10:03 [s3-sync] preparePush.fastpath 0ms miss
10:10:03 [s3-sync] preparePush.shardAssembly 312ms scoped=17
10:10:04 [s3-sync] preparePush.walk 936ms toPush=34 toDelete=0
10:10:04 [s3-sync] preparePush.upload 160ms pushed=25
10:10:04 [s3-sync] preparePush.delete 0ms deleted=0
10:10:05 [s3-sync] commitPush.shards 330ms dirty=9
10:10:05 [s3-sync] commitPush.localIndex 136ms
10:10:05 [s3-sync] commitPush 492ms shard-first changes=25
10:10:05 [s3-sync] pullChanged.fastpath 21ms miss
10:10:06 [s3-sync] pullChanged.pullIndex 69ms
10:10:08 [s3-sync] pullChanged.remoteListing 2866ms keys=35128
10:10:13 [s3-sync] pullChanged.walk 4203ms toPull=1
10:10:13 [s3-sync] pullChanged.download 34ms pulled=1
10:10:13 [s3-sync] pullChanged.fastpath 26ms miss
10:10:14 [s3-sync] pullChanged.pullIndex 1123ms
10:10:14 [s3-sync] pullChanged.remoteListing 44ms keys=291
10:10:14 [s3-sync] pullChanged.walk 179ms toPull=0
10:10:14 [s3-sync] pullChanged.download 0ms pulled=0
10:10:15 [s3-sync] pullChanged.fastpath 33ms miss
10:10:16 [s3-sync] pullChanged.pullIndex 1040ms
10:10:17 [s3-sync] pullChanged.remoteListing 1092ms keys=8587
10:10:19 [s3-sync] pullChanged.walk 1744ms toPull=1
10:10:19 [s3-sync] pullChanged.download 34ms pulled=1
10:10:23 [s3-sync] preparePush.fastpath 0ms miss
10:10:23 [s3-sync] preparePush.shardAssembly 230ms scoped=17
10:10:24 [s3-sync] preparePush.walk 849ms toPush=34 toDelete=0
10:10:24 [s3-sync] preparePush.upload 164ms pushed=25
10:10:24 [s3-sync] preparePush.delete 0ms deleted=0
10:10:25 [s3-sync] commitPush.shards 313ms dirty=9
10:10:25 [s3-sync] commitPush.localIndex 128ms
10:10:25 [s3-sync] commitPush 474ms shard-first changes=25
10:10:25 [s3-sync] pushChanged.fastpath 0ms miss
10:10:25 [s3-sync] pushChanged.shardAssembly 135ms scoped=3
10:10:25 [s3-sync] pushChanged.walk 17ms toPush=7 toDelete=0
10:10:25 [s3-sync] pushChanged.upload 41ms pushed=7
10:10:25 [s3-sync] pushChanged.delete 0ms deleted=0
10:10:26 [s3-sync] pushChanged.v2writeback 272ms
10:10:32 [s3-sync] pullChanged.fastpath 24ms miss
10:10:32 [s3-sync] pullChanged.pullIndex 45ms
10:10:34 [s3-sync] pullChanged.remoteListing 2547ms keys=35153
10:10:38 [s3-sync] pullChanged.walk 4192ms toPull=1
10:10:39 [s3-sync] pullChanged.download 70ms pulled=1
10:10:39 [s3-sync] pullChanged.fastpath 19ms miss
10:10:40 [s3-sync] pullChanged.pullIndex 1179ms
10:10:40 [s3-sync] pullChanged.remoteListing 63ms keys=291
10:10:41 [s3-sync] pullChanged.walk 176ms toPull=0
10:10:41 [s3-sync] pullChanged.download 0ms pulled=0
10:10:41 [s3-sync] preparePush.fastpath 0ms miss
10:10:41 [s3-sync] preparePush.shardAssembly 244ms scoped=16
10:10:41 [s3-sync] preparePush.walk 802ms toPush=28 toDelete=0
10:10:42 [s3-sync] preparePush.upload 148ms pushed=22
10:10:42 [s3-sync] preparePush.delete 0ms deleted=0
10:10:42 [s3-sync] commitPush.shards 327ms dirty=11
10:10:42 [s3-sync] commitPush.localIndex 132ms
10:10:42 [s3-sync] commitPush 482ms shard-first changes=22
10:10:43 [s3-sync] pullChanged.fastpath 25ms miss
10:10:43 [s3-sync] pullChanged.pullIndex 53ms
10:10:46 [s3-sync] pullChanged.remoteListing 2657ms keys=35171
10:10:50 [s3-sync] pullChanged.walk 4058ms toPull=1
10:10:50 [s3-sync] pullChanged.download 40ms pulled=1
10:10:50 [s3-sync] pullChanged.fastpath 23ms miss
10:10:51 [s3-sync] pullChanged.pullIndex 1212ms
10:10:53 [s3-sync] pullChanged.remoteListing 1243ms keys=8622
10:10:54 [s3-sync] pullChanged.walk 1109ms toPull=3
10:10:54 [s3-sync] pullChanged.download 36ms pulled=3
10:10:54 [s3-sync] preparePush.fastpath 0ms miss
10:10:54 [s3-sync] preparePush.shardAssembly 235ms scoped=17
10:10:55 [s3-sync] preparePush.walk 937ms toPush=33 toDelete=0
10:10:55 [s3-sync] preparePush.upload 146ms pushed=24
10:10:55 [s3-sync] preparePush.delete 0ms deleted=0
10:10:56 [s3-sync] commitPush.shards 304ms dirty=8
10:10:56 [s3-sync] commitPush.localIndex 138ms
10:10:56 [s3-sync] commitPush 475ms shard-first changes=24
10:10:56 [s3-sync] pullChanged.fastpath 24ms miss
10:10:56 [s3-sync] pullChanged.pullIndex 64ms
10:10:59 [s3-sync] pullChanged.remoteListing 2533ms keys=35192
10:11:05 [s3-sync] pullChanged.walk 6327ms toPull=1
10:11:05 [s3-sync] pullChanged.download 40ms pulled=1
10:11:08 [s3-sync] pullChanged.fastpath 26ms miss
10:11:09 [s3-sync] pullChanged.pullIndex 1173ms
10:11:09 [s3-sync] pullChanged.remoteListing 62ms keys=291
10:11:09 [s3-sync] pullChanged.walk 144ms toPull=0
10:11:09 [s3-sync] pullChanged.download 0ms pulled=0
10:11:10 [s3-sync] pullChanged.fastpath 20ms miss
10:11:11 [s3-sync] pullChanged.pullIndex 1124ms
10:11:13 [s3-sync] pullChanged.remoteListing 1322ms keys=8640
10:11:14 [s3-sync] pullChanged.walk 1074ms toPull=1
10:11:14 [s3-sync] pullChanged.download 24ms pulled=1
10:11:21 [s3-sync] preparePush.fastpath 0ms miss
10:11:21 [s3-sync] preparePush.shardAssembly 300ms scoped=17
10:11:22 [s3-sync] preparePush.walk 995ms toPush=34 toDelete=0
10:11:22 [s3-sync] preparePush.upload 160ms pushed=25
10:11:22 [s3-sync] preparePush.delete 0ms deleted=0
10:11:22 [s3-sync] commitPush.shards 322ms dirty=9
10:11:23 [s3-sync] commitPush.localIndex 127ms
10:11:23 [s3-sync] commitPush 483ms shard-first changes=25
10:11:23 [s3-sync] pullChanged.fastpath 27ms miss
10:11:23 [s3-sync] pullChanged.pullIndex 62ms
10:11:26 [s3-sync] pullChanged.remoteListing 2947ms keys=35213
10:11:30 [s3-sync] pullChanged.walk 4276ms toPull=1
10:11:30 [s3-sync] pullChanged.download 32ms pulled=1
10:11:38 Handler slow: "capability.queryData" for worker "lab-worker-2" took 5523ms
10:11:38 [s3-sync] preparePush.fastpath 0ms miss
10:11:38 [s3-sync] preparePush.shardAssembly 273ms scoped=17
10:11:39 [s3-sync] preparePush.walk 888ms toPush=34 toDelete=0
10:11:39 [s3-sync] preparePush.upload 169ms pushed=25
10:11:39 [s3-sync] preparePush.delete 0ms deleted=0
10:11:40 [s3-sync] commitPush.shards 344ms dirty=9
10:11:40 [s3-sync] commitPush.localIndex 138ms
10:11:40 [s3-sync] commitPush 513ms shard-first changes=25
10:11:40 [s3-sync] pullChanged.fastpath 25ms miss
10:11:41 [s3-sync] pullChanged.pullIndex 1116ms
10:11:41 [s3-sync] pullChanged.remoteListing 45ms keys=291
10:11:41 [s3-sync] pullChanged.walk 140ms toPull=0
10:11:41 [s3-sync] pullChanged.download 0ms pulled=0
10:11:41 [s3-sync] pullChanged.fastpath 31ms miss
10:11:41 [s3-sync] pullChanged.pullIndex 52ms
10:11:44 [s3-sync] pullChanged.remoteListing 2667ms keys=35234
10:11:48 [s3-sync] pullChanged.walk 4241ms toPull=1
10:11:48 [s3-sync] pullChanged.download 29ms pulled=1
10:11:49 [s3-sync] pullChanged.fastpath 24ms miss
10:11:50 [s3-sync] pullChanged.pullIndex 1183ms
10:11:51 [s3-sync] pullChanged.remoteListing 989ms keys=8676
10:11:53 [s3-sync] pullChanged.walk 1486ms toPull=1
10:11:53 [s3-sync] pullChanged.download 31ms pulled=1
10:11:58 [s3-sync] preparePush.fastpath 0ms miss
10:11:58 [s3-sync] preparePush.shardAssembly 235ms scoped=17
10:11:59 [s3-sync] preparePush.walk 855ms toPush=34 toDelete=0
10:11:59 [s3-sync] preparePush.upload 152ms pushed=25
10:11:59 [s3-sync] preparePush.delete 0ms deleted=0
10:11:59 [s3-sync] commitPush.shards 304ms dirty=9
10:12:00 [s3-sync] commitPush.localIndex 126ms
10:12:00 [s3-sync] commitPush 459ms shard-first changes=25
10:12:00 [s3-sync] pushChanged.fastpath 0ms miss
10:12:00 [s3-sync] pushChanged.shardAssembly 126ms scoped=3
10:12:00 [s3-sync] pushChanged.walk 8ms toPush=7 toDelete=0
10:12:00 [s3-sync] pushChanged.upload 43ms pushed=7
10:12:00 [s3-sync] pushChanged.delete 0ms deleted=0
10:12:00 [s3-sync] pushChanged.v2writeback 281ms
10:12:08 [s3-sync] pullChanged.fastpath 77ms miss
10:12:09 [s3-sync] pullChanged.pullIndex 1167ms
10:12:10 [s3-sync] pullChanged.remoteListing 75ms keys=291
10:12:10 [s3-sync] pullChanged.walk 165ms toPull=0
10:12:10 [s3-sync] pullChanged.download 0ms pulled=0
10:12:10 [s3-sync] pullChanged.fastpath 31ms miss
10:12:11 [s3-sync] pullChanged.pullIndex 1095ms
10:12:14 [s3-sync] pullChanged.remoteListing 2901ms keys=35258
10:12:18 [s3-sync] pullChanged.walk 4114ms toPull=0
10:12:18 [s3-sync] pullChanged.download 0ms pulled=0
10:12:18 [s3-sync] pushChanged.fastpath 0ms miss
10:12:18 [s3-sync] pushChanged.shardAssembly 142ms scoped=5
10:12:18 [s3-sync] pushChanged.walk 7ms toPush=10 toDelete=0
10:12:18 [s3-sync] pushChanged.upload 101ms pushed=10
10:12:18 [s3-sync] pushChanged.delete 0ms deleted=0
10:12:19 [s3-sync] pushChanged.v2writeback 308ms
10:12:19 [s3-sync] pullChanged.fastpath 23ms hit
10:12:38 [s3-sync] pullChanged.fastpath 35ms hit
10:12:41 [s3-sync] pullChanged.fastpath 25ms hit
10:13:08 [s3-sync] pullChanged.fastpath 43ms hit
10:13:09 [s3-sync] pullChanged.fastpath 21ms hit
10:13:38 [s3-sync] pullChanged.fastpath 29ms hit
10:13:39 [s3-sync] pullChanged.fastpath 21ms hit
10:14:08 [s3-sync] pullChanged.fastpath 34ms hit
10:14:09 [s3-sync] pullChanged.fastpath 23ms hit
10:14:38 [s3-sync] pullChanged.fastpath 41ms hit
10:14:39 [s3-sync] pullChanged.fastpath 29ms hit
10:15:08 [s3-sync] pullChanged.fastpath 44ms hit
10:15:09 [s3-sync] pullChanged.fastpath 33ms hit
10:15:38 [s3-sync] pullChanged.fastpath 41ms hit
10:15:39 [s3-sync] pullChanged.fastpath 91ms hit
10:16:08 [s3-sync] pullChanged.fastpath 43ms hit
10:16:09 [s3-sync] pullChanged.fastpath 25ms hit
10:16:38 [s3-sync] pullChanged.fastpath 36ms hit
10:16:39 [s3-sync] pullChanged.fastpath 25ms hit
10:17:08 [s3-sync] pullChanged.fastpath 33ms hit
10:17:09 [s3-sync] pullChanged.fastpath 27ms hit
10:17:38 [s3-sync] pullChanged.fastpath 37ms hit
10:17:39 [s3-sync] pullChanged.fastpath 27ms hit
10🔞08 [s3-sync] pullChanged.fastpath 41ms hit
10🔞09 [s3-sync] pullChanged.fastpath 31ms hit
10🔞38 [s3-sync] pullChanged.fastpath 37ms hit
10🔞39 [s3-sync] pullChanged.fastpath 22ms hit
10:19:09 [s3-sync] pullChanged.fastpath 89ms hit
10:19:09 [s3-sync] pullChanged.fastpath 32ms hit
10:19:38 [s3-sync] pullChanged.fastpath 41ms hit
10:19:41 [s3-sync] pullChanged.fastpath 28ms hit
10:20:08 [s3-sync] pullChanged.fastpath 43ms hit
10:20:09 [s3-sync] pullChanged.fastpath 28ms hit
10:20:38 [s3-sync] pullChanged.fastpath 43ms hit
10:20:39 [s3-sync] pullChanged.fastpath 23ms hit
10:21:08 [s3-sync] pullChanged.fastpath 41ms hit
10:21:09 [s3-sync] pullChanged.fastpath 25ms hit
10:21:38 [s3-sync] pullChanged.fastpath 35ms hit
10:21:39 [s3-sync] pullChanged.fastpath 19ms hit
10:22:09 [s3-sync] pullChanged.fastpath 50ms hit
10:22:09 [s3-sync] pullChanged.fastpath 29ms hit
10:22:38 [s3-sync] pullChanged.fastpath 40ms hit
10:22:39 [s3-sync] pullChanged.fastpath 27ms hit
10:23:08 [s3-sync] pullChanged.fastpath 34ms hit
10:23:09 [s3-sync] pullChanged.fastpath 28ms hit
10:23:38 [s3-sync] pullChanged.fastpath 38ms hit
10:23:39 [s3-sync] pullChanged.fastpath 24ms hit
10:24:08 [s3-sync] pullChanged.fastpath 34ms hit
10:24:09 [s3-sync] pullChanged.fastpath 28ms hit
10:24:31 [s3-sync] pullChanged.fastpath 25ms miss
10:24:32 [s3-sync] pullChanged.pullIndex 1163ms
10:24:35 [s3-sync] pullChanged.remoteListing 2610ms keys=35277
10:24:39 [s3-sync] pullChanged.walk 4016ms toPull=0
10:24:39 [s3-sync] pullChanged.download 0ms pulled=0
10:24:39 [s3-sync] pushChanged.fastpath 0ms miss
10:24:40 [s3-sync] pushChanged.shardAssembly 194ms scoped=5
10:24:40 [s3-sync] pushChanged.walk 16ms toPush=10 toDelete=0
10:24:40 [s3-sync] pushChanged.upload 90ms pushed=10
10:24:40 [s3-sync] pushChanged.delete 0ms deleted=0
10:24:40 [s3-sync] pushChanged.v2writeback 351ms
10:24:40 [s3-sync] pullChanged.fastpath 33ms hit
10:24:41 [s3-sync] pullChanged.fastpath 22ms hit
10:25:09 [s3-sync] pullChanged.fastpath 34ms hit
10:25:18 [s3-sync] pullChanged.fastpath 4498ms hit
10:25:18 [s3-sync] pushChanged.fastpath 0ms miss
10:25:19 [s3-sync] pushChanged.shardAssembly 271ms scoped=17
10:25:19 [s3-sync] pushChanged.walk 720ms toPush=7 toDelete=19
10:25:20 [s3-sync] pushChanged.upload 72ms pushed=7
10:25:20 [s3-sync] pushChanged.delete 86ms deleted=19
10:25:20 [s3-sync] pushChanged.v2writeback 494ms
10:25:20 [s3-sync] pullChanged.fastpath 25ms hit
10:25:21 [s3-sync] pushChanged.fastpath 0ms miss
10:25:21 [s3-sync] pushChanged.shardAssembly 209ms scoped=13
10:25:21 [s3-sync] pushChanged.walk 6ms toPush=0 toDelete=19
10:25:21 [s3-sync] pushChanged.upload 0ms pushed=0
10:25:21 [s3-sync] pushChanged.delete 49ms deleted=19
10:25:21 [s3-sync] pushChanged.v2writeback 326ms
10:25:21 [s3-sync] pullChanged.fastpath 28ms hit
10:25:22 [s3-sync] pushChanged.fastpath 0ms miss
10:25:22 [s3-sync] pushChanged.shardAssembly 160ms scoped=13
10:25:22 [s3-sync] pushChanged.walk 4ms toPush=0 toDelete=19
10:25:22 [s3-sync] pushChanged.upload 0ms pushed=0
10:25:22 [s3-sync] pushChanged.delete 48ms deleted=19
10:25:22 [s3-sync] pushChanged.v2writeback 346ms
10:25:23 [s3-sync] pullChanged.fastpath 26ms hit
10:25:23 [s3-sync] pushChanged.fastpath 0ms miss
10:25:23 [s3-sync] pushChanged.shardAssembly 132ms scoped=13
10:25:23 [s3-sync] pushChanged.walk 4ms toPush=0 toDelete=19
10:25:23 [s3-sync] pushChanged.upload 0ms pushed=0
10:25:23 [s3-sync] pushChanged.delete 47ms deleted=19
10:25:23 [s3-sync] pushChanged.v2writeback 349ms
10:25:41 [s3-sync] pullChanged.fastpath 42ms miss
10:25:42 [s3-sync] pullChanged.pullIndex 1261ms
10:25:42 [s3-sync] pullChanged.remoteListing 70ms keys=291
10:25:42 [s3-sync] pullChanged.walk 154ms toPull=0
10:25:42 [s3-sync] pullChanged.download 0ms pulled=0
10:25:43 [s3-sync] pullChanged.fastpath 26ms miss
10:25:44 [s3-sync] pullChanged.pullIndex 1182ms
10:25:45 [s3-sync] pullChanged.remoteListing 1204ms keys=8669
10:25:46 [s3-sync] pullChanged.walk 931ms toPull=0
10:25:46 [s3-sync] pullChanged.download 0ms pulled=0
10:25:46 [s3-sync] pullChanged.fastpath 30ms miss
10:25:48 [s3-sync] pullChanged.pullIndex 1152ms
10:25:52 [s3-sync] pullChanged.remoteListing 4485ms keys=35226
10:25:56 [s3-sync] pullChanged.walk 4107ms toPull=0
10:25:56 [s3-sync] pullChanged.download 0ms pulled=0
10:25:56 [s3-sync] pushChanged.fastpath 0ms miss
10:25:57 [s3-sync] pushChanged.shardAssembly 136ms scoped=5
10:25:57 [s3-sync] pushChanged.walk 18ms toPush=10 toDelete=0
10:25:57 [s3-sync] pushChanged.upload 96ms pushed=10
10:25:57 [s3-sync] pushChanged.delete 0ms deleted=0
10:25:57 [s3-sync] pushChanged.v2writeback 304ms
10:26:11 [s3-sync] pullChanged.fastpath 357ms hit
10:26:12 [s3-sync] pullChanged.fastpath 25ms hit
10:26:40 [s3-sync] pullChanged.fastpath 29ms miss
10:26:41 [s3-sync] pullChanged.pullIndex 1338ms
10:26:44 [s3-sync] pullChanged.remoteListing 2736ms keys=35253
10:26:48 [s3-sync] pullChanged.walk 4190ms toPull=2
10:26:48 [s3-sync] pullChanged.download 36ms pulled=2
10:26:48 [s3-sync] pushChanged.fastpath 0ms miss
10:26:49 [s3-sync] pushChanged.shardAssembly 275ms scoped=16
10:26:49 [s3-sync] pushChanged.walk 499ms toPush=37 toDelete=0
10:26:49 [s3-sync] pushChanged.upload 155ms pushed=37
10:26:49 [s3-sync] pushChanged.delete 0ms deleted=0
10:26:50 [s3-sync] pushChanged.v2writeback 435ms
10:26:51 [s3-sync] pullChanged.fastpath 27ms hit
10:26:52 [s3-sync] pullChanged.fastpath 23ms hit
10:27:11 [s3-sync] pullChanged.fastpath 35ms miss
10:27:12 [s3-sync] pullChanged.pullIndex 1431ms
10:27:12 [s3-sync] pullChanged.remoteListing 78ms keys=291
10:27:12 [s3-sync] pullChanged.walk 175ms toPull=0
10:27:12 [s3-sync] pullChanged.download 0ms pulled=0
10:27:13 [s3-sync] pullChanged.fastpath 18ms miss
10:27:14 [s3-sync] pullChanged.pullIndex 1061ms
10:27:15 [s3-sync] pullChanged.remoteListing 1196ms keys=8731
10:27:16 [s3-sync] pullChanged.walk 1059ms toPull=0
10:27:16 [s3-sync] pullChanged.download 0ms pulled=0
10:27:16 [s3-sync] pushChanged.fastpath 0ms hit
10:27:16 [s3-sync] pullChanged.fastpath 31ms miss
10:27:17 [s3-sync] pullChanged.pullIndex 1166ms
10:27:18 [s3-sync] pullChanged.remoteListing 77ms keys=291
10:27:18 [s3-sync] pullChanged.walk 144ms toPull=0
10:27:18 [s3-sync] pullChanged.download 0ms pulled=0
10:27:41 [s3-sync] pullChanged.fastpath 28ms miss
10:27:42 [s3-sync] pullChanged.pullIndex 1227ms
10:27:42 [s3-sync] pullChanged.remoteListing 69ms keys=291
10:27:42 [s3-sync] pullChanged.walk 149ms toPull=0
10:27:42 [s3-sync] pullChanged.download 0ms pulled=0
10:27:43 [s3-sync] pullChanged.fastpath 23ms miss
10:27:44 [s3-sync] pullChanged.pullIndex 1211ms
10:27:45 [s3-sync] pullChanged.remoteListing 1208ms keys=8751
10:27:46 [s3-sync] pullChanged.walk 1134ms toPull=0
10:27:46 [s3-sync] pullChanged.download 0ms pulled=0
10:28:11 [s3-sync] pullChanged.fastpath 25ms miss
10:28:12 [s3-sync] pullChanged.pullIndex 1174ms
10:28:12 [s3-sync] pullChanged.remoteListing 77ms keys=291
10:28:12 [s3-sync] pullChanged.walk 180ms toPull=0
10:28:12 [s3-sync] pullChanged.download 0ms pulled=0
10:28:13 [s3-sync] pullChanged.fastpath 29ms miss
10:28:14 [s3-sync] pullChanged.pullIndex 1444ms
10:28:15 [s3-sync] pullChanged.remoteListing 1091ms keys=8761
10:28:16 [s3-sync] pullChanged.walk 961ms toPull=0
10:28:16 [s3-sync] pullChanged.download 0ms pulled=0
10:28:41 [s3-sync] pullChanged.fastpath 23ms miss
10:28:42 [s3-sync] pullChanged.pullIndex 1283ms
10:28:42 [s3-sync] pullChanged.remoteListing 86ms keys=291
10:28:42 [s3-sync] pullChanged.walk 143ms toPull=0
10:28:42 [s3-sync] pullChanged.download 0ms pulled=0
10:28:43 [s3-sync] pullChanged.fastpath 24ms miss
10:28:44 [s3-sync] pullChanged.pullIndex 1152ms
10:28:45 [s3-sync] pullChanged.remoteListing 1155ms keys=8761
10:28:46 [s3-sync] pullChanged.walk 794ms toPull=0
10:28:46 [s3-sync] pullChanged.download 0ms pulled=0
10:29:11 [s3-sync] pullChanged.fastpath 18ms miss
10:29:12 [s3-sync] pullChanged.pullIndex 1638ms
10:29:13 [s3-sync] pullChanged.remoteListing 87ms keys=291
10:29:13 [s3-sync] pullChanged.walk 263ms toPull=0
10:29:13 [s3-sync] pullChanged.download 0ms pulled=0
10:29:13 [s3-sync] pullChanged.fastpath 28ms miss
10:29:13 [s3-sync] pullChanged.pullIndex 91ms
10:29:16 [s3-sync] pullChanged.remoteListing 2738ms keys=35336
10:29:20 [s3-sync] pullChanged.walk 4042ms toPull=1
10:29:20 [s3-sync] pullChanged.download 70ms pulled=1
10:29:21 [s3-sync] pullChanged.fastpath 21ms miss
10:29:22 [s3-sync] pullChanged.pullIndex 1184ms
10:29:23 [s3-sync] pullChanged.remoteListing 1091ms keys=8762
10:29:24 [s3-sync] pullChanged.walk 1100ms toPull=3
10:29:24 [s3-sync] pullChanged.download 27ms pulled=3
10:29:24 [s3-sync] preparePush.fastpath 0ms miss
10:29:25 [s3-sync] preparePush.shardAssembly 219ms scoped=22
10:29:25 [s3-sync] preparePush.walk 939ms toPush=41 toDelete=0
10:29:25 [s3-sync] preparePush.upload 193ms pushed=29
10:29:25 [s3-sync] preparePush.delete 0ms deleted=0
10:29:26 [s3-sync] commitPush.shards 353ms dirty=10
10:29:26 [s3-sync] commitPush.localIndex 135ms
10:29:26 [s3-sync] commitPush 520ms shard-first changes=29
10:29:27 [s3-sync] pullChanged.fastpath 22ms miss
10:29:27 [s3-sync] pullChanged.pullIndex 59ms
10:29:30 [s3-sync] pullChanged.remoteListing 2528ms keys=35362
10:29:34 [s3-sync] pullChanged.walk 4081ms toPull=1
10:29:34 [s3-sync] pullChanged.download 30ms pulled=1
10:29:37 [s3-sync] preparePush.fastpath 0ms miss
10:29:38 [s3-sync] preparePush.shardAssembly 376ms scoped=14
10:29:38 [s3-sync] preparePush.walk 1107ms toPush=28 toDelete=0
10:29:39 [s3-sync] preparePush.upload 681ms pushed=22
10:29:39 [s3-sync] preparePush.delete 0ms deleted=0
10:29:40 [s3-sync] commitPush.shards 370ms dirty=9
10:29:40 [s3-sync] commitPush.localIndex 122ms
10:29:40 [s3-sync] commitPush 526ms shard-first changes=22
10:29:41 [s3-sync] pullChanged.fastpath 22ms miss
10:29:42 [s3-sync] pullChanged.pullIndex 1162ms
10:29:42 [s3-sync] pullChanged.remoteListing 67ms keys=291
10:29:42 [s3-sync] pullChanged.walk 149ms toPull=0
10:29:42 [s3-sync] pullChanged.download 0ms pulled=0
10:29:42 [s3-sync] pullChanged.fastpath 28ms miss
10:29:42 [s3-sync] pullChanged.pullIndex 60ms
10:29:45 [s3-sync] pullChanged.remoteListing 2981ms keys=35380
10:29:49 [s3-sync] pullChanged.walk 4124ms toPull=1
10:29:49 [s3-sync] pullChanged.download 30ms pulled=1
10:29:50 [s3-sync] pullChanged.fastpath 26ms miss
10:29:51 [s3-sync] pullChanged.pullIndex 1207ms
10:29:52 [s3-sync] pullChanged.remoteListing 1016ms keys=8798
10:29:53 [s3-sync] pullChanged.walk 1104ms toPull=3
10:29:53 [s3-sync] pullChanged.download 29ms pulled=3
10:29:53 [s3-sync] preparePush.fastpath 0ms miss
10:29:53 [s3-sync] preparePush.shardAssembly 216ms scoped=14
10:29:54 [s3-sync] preparePush.walk 825ms toPush=25 toDelete=0
10:29:54 [s3-sync] preparePush.upload 113ms pushed=19
10:29:54 [s3-sync] preparePush.delete 0ms deleted=0
10:29:55 [s3-sync] commitPush.shards 362ms dirty=8
10:29:55 [s3-sync] commitPush.localIndex 122ms
10:29:55 [s3-sync] commitPush 518ms shard-first changes=19
10:29:56 [s3-sync] pullChanged.fastpath 30ms miss
10:29:56 [s3-sync] pullChanged.pullIndex 50ms
10:29:59 [s3-sync] pullChanged.remoteListing 2912ms keys=35396
10:30:04 [s3-sync] pullChanged.walk 4778ms toPull=1
10:30:04 [s3-sync] pullChanged.download 37ms pulled=1
10:30:06 [s3-sync] preparePush.fastpath 0ms miss
10:30:06 [s3-sync] preparePush.shardAssembly 262ms scoped=20
10:30:07 [s3-sync] preparePush.walk 967ms toPush=40 toDelete=0
10:30:07 [s3-sync] preparePush.upload 169ms pushed=28
10:30:07 [s3-sync] preparePush.delete 0ms deleted=0
10:30:07 [s3-sync] commitPush.shards 335ms dirty=9
10:30:08 [s3-sync] commitPush.localIndex 136ms
10:30:08 [s3-sync] commitPush 506ms shard-first changes=28
10:30:11 [s3-sync] pullChanged.fastpath 31ms miss
10:30:11 [s3-sync] pullChanged.pullIndex 65ms
10:30:14 [s3-sync] pullChanged.remoteListing 2917ms keys=35420
10:30:18 [s3-sync] pullChanged.walk 4211ms toPull=1
10:30:18 [s3-sync] pullChanged.download 40ms pulled=1
10:30:19 [s3-sync] pullChanged.fastpath 20ms miss
10:30:20 [s3-sync] pullChanged.pullIndex 1201ms
10:30:20 [s3-sync] pullChanged.remoteListing 62ms keys=291
10:30:20 [s3-sync] pullChanged.walk 134ms toPull=0
10:30:20 [s3-sync] pullChanged.download 0ms pulled=0
10:30:21 [s3-sync] pullChanged.fastpath 20ms miss
10:30:22 [s3-sync] pullChanged.pullIndex 1097ms
10:30:23 [s3-sync] pullChanged.remoteListing 1068ms keys=8832
10:30:24 [s3-sync] pullChanged.walk 900ms toPull=1
10:30:24 [s3-sync] pullChanged.download 33ms pulled=1
10:30:27 [s3-sync] preparePush.fastpath 0ms miss
10:30:28 [s3-sync] preparePush.shardAssembly 234ms scoped=17
10:30:28 [s3-sync] preparePush.walk 900ms toPush=34 toDelete=0
10:30:28 [s3-sync] preparePush.upload 152ms pushed=25
10:30:28 [s3-sync] preparePush.delete 0ms deleted=0
10:30:29 [s3-sync] commitPush.shards 335ms dirty=9
10:30:29 [s3-sync] commitPush.localIndex 124ms
10:30:29 [s3-sync] commitPush 479ms shard-first changes=25
10:30:30 [s3-sync] pullChanged.fastpath 25ms miss
10:30:30 [s3-sync] pullChanged.pullIndex 55ms
10:30:33 [s3-sync] pullChanged.remoteListing 2915ms keys=35441
10:30:37 [s3-sync] pullChanged.walk 4197ms toPull=1
10:30:37 [s3-sync] pullChanged.download 37ms pulled=1
10:30:42 [s3-sync] pullChanged.fastpath 76ms miss
10:30:44 [s3-sync] pullChanged.pullIndex 1148ms
10:30:44 [s3-sync] pullChanged.remoteListing 51ms keys=291
10:30:44 [s3-sync] pullChanged.walk 152ms toPull=0
10:30:44 [s3-sync] pullChanged.download 0ms pulled=0
10:30:44 [s3-sync] pullChanged.fastpath 23ms miss
10:30:45 [s3-sync] pullChanged.pullIndex 1138ms
10:30:47 [s3-sync] pullChanged.remoteListing 1251ms keys=8850
10:30:48 Handler slow: "capability.queryData" for worker "lab-worker-2" took 7946ms
10:30:49 [s3-sync] pullChanged.walk 1871ms toPull=1
10:30:49 [s3-sync] pullChanged.download 28ms pulled=1
10:30:49 [s3-sync] preparePush.fastpath 0ms miss
10:30:49 [s3-sync] preparePush.shardAssembly 211ms scoped=23
10:30:50 [s3-sync] preparePush.walk 950ms toPush=45 toDelete=0
10:30:50 [s3-sync] preparePush.upload 168ms pushed=30
10:30:50 [s3-sync] preparePush.delete 0ms deleted=0
10:30:51 [s3-sync] commitPush.shards 326ms dirty=8
10:30:51 [s3-sync] commitPush.localIndex 143ms
10:30:51 [s3-sync] commitPush 487ms shard-first changes=30
10:30:52 [s3-sync] pullChanged.fastpath 31ms miss
10:30:52 [s3-sync] pullChanged.pullIndex 60ms
10:30:55 [s3-sync] pullChanged.remoteListing 2922ms keys=35468
10:30:59 [s3-sync] pullChanged.walk 4104ms toPull=1
10:30:59 [s3-sync] pullChanged.download 40ms pulled=1
10:31:04 [s3-sync] preparePush.fastpath 0ms miss
10:31:04 [s3-sync] preparePush.shardAssembly 258ms scoped=14
10:31:05 [s3-sync] preparePush.walk 860ms toPush=28 toDelete=0
10:31:06 [s3-sync] preparePush.upload 705ms pushed=22
10:31:06 [s3-sync] preparePush.delete 0ms deleted=0
10:31:06 [s3-sync] commitPush.shards 352ms dirty=9
10:31:06 [s3-sync] commitPush.localIndex 131ms
10:31:06 [s3-sync] commitPush 515ms shard-first changes=22
10:31:07 [s3-sync] pullChanged.fastpath 33ms miss
10:31:07 [s3-sync] pullChanged.pullIndex 61ms
10:31:10 [s3-sync] pullChanged.remoteListing 2606ms keys=35486
10:31:16 [s3-sync] pullChanged.walk 6341ms toPull=1
10:31:16 [s3-sync] pullChanged.download 31ms pulled=1
10:31:17 [s3-sync] pullChanged.fastpath 20ms miss
10:31:18 [s3-sync] pullChanged.pullIndex 1150ms
10:31:18 [s3-sync] pullChanged.remoteListing 62ms keys=291
10:31:18 [s3-sync] pullChanged.walk 151ms toPull=0
10:31:18 [s3-sync] pullChanged.download 0ms pulled=0
10:31:18 [s3-sync] preparePush.fastpath 0ms miss
10:31:19 [s3-sync] preparePush.shardAssembly 215ms scoped=14
10:31:19 [s3-sync] preparePush.walk 818ms toPush=28 toDelete=0
10:31:19 [s3-sync] preparePush.upload 145ms pushed=22
10:31:19 [s3-sync] preparePush.delete 0ms deleted=0
10:31:20 [s3-sync] commitPush.shards 323ms dirty=9
10:31:20 [s3-sync] commitPush.localIndex 138ms
10:31:20 [s3-sync] commitPush 488ms shard-first changes=22
10:31:21 [s3-sync] pullChanged.fastpath 51ms miss
10:31:22 [s3-sync] pullChanged.pullIndex 1096ms
10:31:23 [s3-sync] pullChanged.remoteListing 1043ms keys=8904
10:31:24 [s3-sync] pullChanged.walk 1205ms toPull=0
10:31:24 [s3-sync] pullChanged.download 0ms pulled=0
10:31:24 [s3-sync] pullChanged.fastpath 33ms miss
10:31:24 [s3-sync] pullChanged.pullIndex 58ms
10:31:27 [s3-sync] pullChanged.remoteListing 2825ms keys=35504
10:31:31 [s3-sync] pullChanged.walk 4329ms toPull=1
10:31:31 [s3-sync] pullChanged.download 38ms pulled=1
10:31:34 [s3-sync] preparePush.fastpath 1ms miss
10:31:34 [s3-sync] preparePush.shardAssembly 248ms scoped=14
10:31:34 [s3-sync] preparePush.walk 867ms toPush=26 toDelete=0
10:31:35 [s3-sync] preparePush.upload 146ms pushed=20
10:31:35 [s3-sync] preparePush.delete 0ms deleted=0
10:31:35 [s3-sync] commitPush.shards 350ms dirty=9
10:31:35 [s3-sync] commitPush.localIndex 128ms
10:31:35 [s3-sync] commitPush 503ms shard-first changes=20
10:31:36 [s3-sync] pullChanged.fastpath 28ms miss
10:31:36 [s3-sync] pullChanged.pullIndex 55ms
10:31:39 [s3-sync] pullChanged.remoteListing 2500ms keys=35520
10:31:43 [s3-sync] pullChanged.walk 4156ms toPull=1
10:31:43 [s3-sync] pullChanged.download 36ms pulled=1
10:31:43 [s3-sync] pullChanged.fastpath 25ms miss
10:31:44 [s3-sync] pullChanged.pullIndex 1126ms
10:31:45 [s3-sync] pullChanged.remoteListing 72ms keys=291
10:31:45 [s3-sync] pullChanged.walk 261ms toPull=0
10:31:45 [s3-sync] pullChanged.download 0ms pulled=0
10:31:47 [s3-sync] pullChanged.fastpath 28ms miss
10:31:48 [s3-sync] pullChanged.pullIndex 1119ms
10:31:50 [s3-sync] pullChanged.remoteListing 1243ms keys=8917vcjdeboer commented 10/31/2026, 12:40:42 PM
Full [s3-sync] / Handler slow / Handler error / heap-limit log of the traced sitting, part 2 of 4 (10:31:51-10:53:10 UTC):
10:31:51 [s3-sync] pullChanged.walk 1166ms toPull=3
10:31:51 [s3-sync] pullChanged.download 29ms pulled=3
10:31:51 [s3-sync] preparePush.fastpath 0ms miss
10:31:51 [s3-sync] preparePush.shardAssembly 214ms scoped=20
10:31:52 [s3-sync] preparePush.walk 915ms toPush=39 toDelete=0
10:31:52 [s3-sync] preparePush.upload 169ms pushed=27
10:31:52 [s3-sync] preparePush.delete 0ms deleted=0
10:31:53 [s3-sync] commitPush.shards 410ms dirty=8
10:31:53 [s3-sync] commitPush.localIndex 119ms
10:31:53 [s3-sync] commitPush 560ms shard-first changes=27
10:31:54 [s3-sync] pullChanged.fastpath 24ms miss
10:31:54 [s3-sync] pullChanged.pullIndex 56ms
10:31:57 [s3-sync] pullChanged.remoteListing 2992ms keys=35544
10:32:01 [s3-sync] pullChanged.walk 4371ms toPull=1
10:32:01 [s3-sync] pullChanged.download 47ms pulled=1
10:32:12 [s3-sync] pullChanged.fastpath 41ms miss
10:32:13 [s3-sync] pullChanged.pullIndex 1158ms
10:32:14 [s3-sync] pullChanged.remoteListing 52ms keys=291
10:32:14 [s3-sync] pullChanged.walk 149ms toPull=0
10:32:14 [s3-sync] pullChanged.download 0ms pulled=0
10:32:14 [s3-sync] pullChanged.fastpath 23ms miss
10:32:15 [s3-sync] pullChanged.pullIndex 1109ms
10:32:17 [s3-sync] pullChanged.remoteListing 1002ms keys=8938
10:32:20 [s3-sync] pullChanged.walk 3298ms toPull=1
10:32:20 [s3-sync] pullChanged.download 40ms pulled=1
10:32:20 [s3-sync] preparePush.fastpath 0ms miss
10:32:20 [s3-sync] preparePush.shardAssembly 226ms scoped=17
10:32:21 [s3-sync] preparePush.walk 880ms toPush=33 toDelete=0
10:32:22 [s3-sync] preparePush.upload 646ms pushed=24
10:32:22 [s3-sync] preparePush.delete 0ms deleted=0
10:32:22 [s3-sync] commitPush.shards 317ms dirty=8
10:32:22 [s3-sync] commitPush.localIndex 126ms
10:32:22 [s3-sync] commitPush 466ms shard-first changes=24
10:32:23 [s3-sync] pullChanged.fastpath 27ms miss
10:32:23 [s3-sync] pullChanged.pullIndex 71ms
10:32:26 [s3-sync] pullChanged.remoteListing 2925ms keys=35565
10:32:30 [s3-sync] pullChanged.walk 4233ms toPull=1
10:32:30 [s3-sync] pullChanged.download 33ms pulled=1
10:32:36 [s3-sync] preparePush.fastpath 0ms miss
10:32:37 [s3-sync] preparePush.shardAssembly 256ms scoped=35
10:32:38 [s3-sync] preparePush.walk 1153ms toPush=70 toDelete=0
10:32:38 [s3-sync] preparePush.upload 235ms pushed=43
10:32:38 [s3-sync] preparePush.delete 0ms deleted=0
10:32:38 [s3-sync] commitPush.shards 335ms dirty=9
10:32:38 [s3-sync] commitPush.localIndex 136ms
10:32:38 [s3-sync] commitPush 501ms shard-first changes=43
10:32:39 [s3-sync] pullChanged.fastpath 30ms miss
10:32:39 [s3-sync] pullChanged.pullIndex 60ms
10:32:42 [s3-sync] pullChanged.remoteListing 2470ms keys=35604
10:32:46 [s3-sync] pullChanged.walk 4483ms toPull=1
10:32:46 [s3-sync] pullChanged.download 50ms pulled=1
10:32:47 [s3-sync] pullChanged.fastpath 28ms miss
10:32:48 [s3-sync] pullChanged.pullIndex 1175ms
10:32:49 [s3-sync] pullChanged.remoteListing 57ms keys=291
10:32:49 [s3-sync] pullChanged.walk 165ms toPull=0
10:32:49 [s3-sync] pullChanged.download 0ms pulled=0
10:32:49 [s3-sync] pullChanged.fastpath 23ms miss
10:32:53 [s3-sync] pullChanged.pullIndex 3227ms
10:32:54 [s3-sync] pullChanged.remoteListing 1149ms keys=8992
10:32:55 Handler slow: "capability.queryData" for worker "lab-worker-2" took 7149ms
10:32:56 [s3-sync] pullChanged.walk 1991ms toPull=1
10:32:56 [s3-sync] pullChanged.download 30ms pulled=1
10:32:56 [s3-sync] preparePush.fastpath 0ms miss
10:32:56 [s3-sync] preparePush.shardAssembly 229ms scoped=14
10:32:57 [s3-sync] preparePush.walk 827ms toPush=27 toDelete=0
10:32:57 [s3-sync] preparePush.upload 660ms pushed=21
10:32:57 [s3-sync] preparePush.delete 0ms deleted=0
10:32:58 [s3-sync] commitPush.shards 289ms dirty=8
10:32:58 [s3-sync] commitPush.localIndex 119ms
10:32:58 [s3-sync] commitPush 435ms shard-first changes=21
10:32:59 [s3-sync] pullChanged.fastpath 27ms miss
10:32:59 [s3-sync] pullChanged.pullIndex 49ms
10:33:02 [s3-sync] pullChanged.remoteListing 2511ms keys=35622
10:33:06 [s3-sync] pullChanged.walk 4216ms toPull=1
10:33:06 [s3-sync] pullChanged.download 31ms pulled=1
10:33:12 [s3-sync] pullChanged.fastpath 34ms miss
10:33:13 [s3-sync] pullChanged.pullIndex 1101ms
10:33:14 [s3-sync] pullChanged.remoteListing 40ms keys=291
10:33:14 [s3-sync] pullChanged.walk 128ms toPull=0
10:33:14 [s3-sync] pullChanged.download 0ms pulled=0
10:33:14 [s3-sync] pullChanged.fastpath 29ms miss
10:33:15 [s3-sync] pullChanged.pullIndex 1107ms
10:33:17 [s3-sync] pullChanged.remoteListing 1115ms keys=9007
10:33:17 [s3-sync] pullChanged.walk 774ms toPull=1
10:33:18 [s3-sync] pullChanged.download 26ms pulled=1
10:33:20 [s3-sync] preparePush.fastpath 0ms miss
10:33:21 [s3-sync] preparePush.shardAssembly 216ms scoped=17
10:33:21 [s3-sync] preparePush.walk 843ms toPush=34 toDelete=0
10:33:21 [s3-sync] preparePush.upload 159ms pushed=25
10:33:21 [s3-sync] preparePush.delete 0ms deleted=0
10:33:22 [s3-sync] commitPush.shards 351ms dirty=9
10:33:24 [s3-sync] commitPush.localIndex 2237ms
10:33:24 [s3-sync] commitPush 2619ms shard-first changes=25
10:33:25 [s3-sync] pullChanged.fastpath 22ms miss
10:33:25 [s3-sync] pullChanged.pullIndex 60ms
10:33:28 [s3-sync] pullChanged.remoteListing 2789ms keys=35643
10:33:32 [s3-sync] pullChanged.walk 4269ms toPull=1
10:33:32 [s3-sync] pullChanged.download 40ms pulled=1
10:33:38 [s3-sync] preparePush.fastpath 0ms miss
10:33:38 [s3-sync] preparePush.shardAssembly 234ms scoped=17
10:33:39 [s3-sync] preparePush.walk 853ms toPush=34 toDelete=0
10:33:39 [s3-sync] preparePush.upload 196ms pushed=25
10:33:39 [s3-sync] preparePush.delete 0ms deleted=0
10:33:39 [s3-sync] commitPush.shards 345ms dirty=9
10:33:40 [s3-sync] commitPush.localIndex 126ms
10:33:40 [s3-sync] commitPush 503ms shard-first changes=25
10:33:40 [s3-sync] pushChanged.fastpath 0ms miss
10:33:40 [s3-sync] pushChanged.shardAssembly 127ms scoped=3
10:33:41 [s3-sync] pushChanged.walk 10ms toPush=7 toDelete=0
10:33:41 [s3-sync] pushChanged.upload 61ms pushed=7
10:33:41 [s3-sync] pushChanged.delete 0ms deleted=0
10:33:41 [s3-sync] pushChanged.v2writeback 302ms
10:33:42 [s3-sync] pullChanged.fastpath 28ms miss
10:33:43 [s3-sync] pullChanged.pullIndex 1147ms
10:33:44 [s3-sync] pullChanged.remoteListing 42ms keys=291
10:33:44 [s3-sync] pullChanged.walk 152ms toPull=0
10:33:44 [s3-sync] pullChanged.download 0ms pulled=0
10:33:44 [s3-sync] pullChanged.fastpath 30ms miss
10:33:45 [s3-sync] pullChanged.pullIndex 1103ms
10:33:47 [s3-sync] pullChanged.remoteListing 1078ms keys=9046
10:33:48 [s3-sync] pullChanged.walk 888ms toPull=0
10:33:48 [s3-sync] pullChanged.download 0ms pulled=0
10:33:49 [s3-sync] pullChanged.fastpath 16ms miss
10:33:49 [s3-sync] pullChanged.pullIndex 53ms
10:33:52 [s3-sync] pullChanged.remoteListing 2468ms keys=35668
10:33:58 [s3-sync] pullChanged.walk 6209ms toPull=1
10:33:58 [s3-sync] pullChanged.download 63ms pulled=1
10:34:00 [s3-sync] preparePush.fastpath 0ms miss
10:34:00 [s3-sync] preparePush.shardAssembly 228ms scoped=19
10:34:01 [s3-sync] preparePush.walk 890ms toPush=36 toDelete=0
10:34:01 [s3-sync] preparePush.upload 145ms pushed=27
10:34:01 [s3-sync] preparePush.delete 0ms deleted=0
10:34:02 [s3-sync] commitPush.shards 296ms dirty=11
10:34:02 [s3-sync] commitPush.localIndex 126ms
10:34:02 [s3-sync] commitPush 452ms shard-first changes=27
10:34:02 [s3-sync] pullChanged.fastpath 82ms miss
10:34:03 [s3-sync] pullChanged.pullIndex 63ms
10:34:05 [s3-sync] pullChanged.remoteListing 2861ms keys=35693
10:34:10 [s3-sync] pullChanged.walk 4147ms toPull=1
10:34:10 [s3-sync] pullChanged.download 24ms pulled=1
10:34:11 [s3-sync] preparePush.fastpath 0ms miss
10:34:12 [s3-sync] preparePush.shardAssembly 236ms scoped=17
10:34:12 [s3-sync] preparePush.walk 846ms toPush=34 toDelete=0
10:34:12 [s3-sync] preparePush.upload 151ms pushed=25
10:34:12 [s3-sync] preparePush.delete 0ms deleted=0
10:34:13 [s3-sync] commitPush.shards 309ms dirty=9
10:34:13 [s3-sync] commitPush.localIndex 121ms
10:34:13 [s3-sync] commitPush 464ms shard-first changes=25
10:34:13 [s3-sync] pullChanged.fastpath 19ms miss
10:34:14 [s3-sync] pullChanged.pullIndex 1133ms
10:34:15 [s3-sync] pullChanged.remoteListing 52ms keys=291
10:34:15 [s3-sync] pullChanged.walk 143ms toPull=0
10:34:15 [s3-sync] pullChanged.download 0ms pulled=0
10:34:15 [s3-sync] pullChanged.fastpath 27ms miss
10:34:15 [s3-sync] pullChanged.pullIndex 91ms
10:34:17 [s3-sync] pullChanged.remoteListing 2531ms keys=35716
10:34:22 [s3-sync] pullChanged.walk 4150ms toPull=1
10:34:22 [s3-sync] pullChanged.download 25ms pulled=1
10:34:22 [s3-sync] pullChanged.fastpath 25ms miss
10:34:24 [s3-sync] pullChanged.pullIndex 1184ms
10:34:25 [s3-sync] pullChanged.remoteListing 1099ms keys=9087
10:34:26 [s3-sync] pullChanged.walk 1143ms toPull=1
10:34:26 [s3-sync] pullChanged.download 28ms pulled=1
10:34:26 [s3-sync] preparePush.fastpath 0ms miss
10:34:26 [s3-sync] preparePush.shardAssembly 208ms scoped=17
10:34:27 [s3-sync] preparePush.walk 857ms toPush=33 toDelete=0
10:34:29 [s3-sync] preparePush.upload 2405ms pushed=24
10:34:29 [s3-sync] preparePush.delete 0ms deleted=0
10:34:30 [s3-sync] commitPush.shards 348ms dirty=8
10:34:30 [s3-sync] commitPush.localIndex 127ms
10:34:30 [s3-sync] commitPush 495ms shard-first changes=24
10:34:31 [s3-sync] pullChanged.fastpath 26ms miss
10:34:31 [s3-sync] pullChanged.pullIndex 64ms
10:34:34 [s3-sync] pullChanged.remoteListing 2961ms keys=35739
10:34:38 [s3-sync] pullChanged.walk 4287ms toPull=1
10:34:38 [s3-sync] pullChanged.download 30ms pulled=1
10:34:40 [s3-sync] preparePush.fastpath 0ms miss
10:34:40 [s3-sync] preparePush.shardAssembly 262ms scoped=17
10:34:41 [s3-sync] preparePush.walk 900ms toPush=34 toDelete=0
10:34:41 [s3-sync] preparePush.upload 139ms pushed=25
10:34:41 [s3-sync] preparePush.delete 0ms deleted=0
10:34:41 [s3-sync] commitPush.shards 364ms dirty=9
10:34:42 [s3-sync] commitPush.localIndex 128ms
10:34:42 [s3-sync] commitPush 525ms shard-first changes=25
10:34:42 [s3-sync] pullChanged.fastpath 27ms miss
10:34:42 [s3-sync] pullChanged.pullIndex 55ms
10:34:45 [s3-sync] pullChanged.remoteListing 2483ms keys=35762
10:34:49 [s3-sync] pullChanged.walk 4370ms toPull=1
10:34:49 [s3-sync] pullChanged.download 28ms pulled=1
10:34:49 [s3-sync] pullChanged.fastpath 30ms miss
10:34:51 [s3-sync] pullChanged.pullIndex 1149ms
10:34:51 [s3-sync] pullChanged.remoteListing 67ms keys=291
10:34:51 [s3-sync] pullChanged.walk 187ms toPull=0
10:34:51 [s3-sync] pullChanged.download 0ms pulled=0
10:34:51 [s3-sync] preparePush.fastpath 0ms miss
10:34:51 [s3-sync] preparePush.shardAssembly 193ms scoped=17
10:34:52 [s3-sync] preparePush.walk 824ms toPush=34 toDelete=0
10:34:52 [s3-sync] preparePush.upload 182ms pushed=25
10:34:52 [s3-sync] preparePush.delete 0ms deleted=0
10:34:53 [s3-sync] commitPush.shards 349ms dirty=9
10:34:53 [s3-sync] commitPush.localIndex 129ms
10:34:53 [s3-sync] commitPush 502ms shard-first changes=25
10:34:53 [s3-sync] pushChanged.fastpath 0ms miss
10:34:53 [s3-sync] pushChanged.shardAssembly 129ms scoped=3
10:34:53 [s3-sync] pushChanged.walk 11ms toPush=7 toDelete=0
10:34:53 [s3-sync] pushChanged.upload 45ms pushed=7
10:34:53 [s3-sync] pushChanged.delete 0ms deleted=0
10:34:54 [s3-sync] pushChanged.v2writeback 290ms
10:34:54 [s3-sync] pullChanged.fastpath 27ms miss
10:34:56 [s3-sync] pullChanged.pullIndex 1113ms
10:34:57 [s3-sync] pullChanged.remoteListing 1188ms keys=9150
10:34:58 [s3-sync] pullChanged.walk 1591ms toPull=0
10:34:58 [s3-sync] pullChanged.download 0ms pulled=0
10:35:13 [s3-sync] pullChanged.fastpath 34ms miss
10:35:14 [s3-sync] pullChanged.pullIndex 1247ms
10:35:15 [s3-sync] pullChanged.remoteListing 60ms keys=291
10:35:15 [s3-sync] pullChanged.walk 176ms toPull=0
10:35:15 [s3-sync] pullChanged.download 0ms pulled=0
10:35:15 [s3-sync] pullChanged.fastpath 26ms miss
10:35:16 [s3-sync] pullChanged.pullIndex 1236ms
10:35:18 [s3-sync] pullChanged.remoteListing 1330ms keys=9150
10:35:19 [s3-sync] pullChanged.walk 929ms toPull=0
10:35:19 [s3-sync] pullChanged.download 0ms pulled=0
10:35:19 [s3-sync] pullChanged.fastpath 30ms miss
10:35:19 [s3-sync] pullChanged.pullIndex 63ms
10:35:22 [s3-sync] pullChanged.remoteListing 2966ms keys=35789
10:35:27 [s3-sync] pullChanged.walk 4234ms toPull=1
10:35:27 [s3-sync] pullChanged.download 31ms pulled=1
10:35:32 [s3-sync] preparePush.fastpath 0ms miss
10:35:32 [s3-sync] preparePush.shardAssembly 278ms scoped=19
10:35:35 [s3-sync] preparePush.walk 3037ms toPush=36 toDelete=0
10:35:35 [s3-sync] preparePush.upload 163ms pushed=27
10:35:35 [s3-sync] preparePush.delete 0ms deleted=0
10:35:36 [s3-sync] commitPush.shards 337ms dirty=11
10:35:36 [s3-sync] commitPush.localIndex 138ms
10:35:36 [s3-sync] commitPush 502ms shard-first changes=27
10:35:36 [s3-sync] pushChanged.fastpath 0ms miss
10:35:36 [s3-sync] pushChanged.shardAssembly 138ms scoped=3
10:35:36 [s3-sync] pushChanged.walk 9ms toPush=7 toDelete=0
10:35:36 [s3-sync] pushChanged.upload 40ms pushed=7
10:35:36 [s3-sync] pushChanged.delete 0ms deleted=0
10:35:36 [s3-sync] pushChanged.v2writeback 280ms
10:35:43 [s3-sync] pullChanged.fastpath 24ms miss
10:36:00 [s3-sync] pullChanged.pullIndex 16456ms
10:36:00 [s3-sync] pullChanged.remoteListing 67ms keys=291
10:36:05 [s3-sync] pullChanged.walk 5263ms toPull=0
10:36:05 [s3-sync] pullChanged.download 0ms pulled=0
10:36:08 [s3-sync] pullChanged.fastpath 1924ms miss
10:36:10 [s3-sync] pullChanged.pullIndex 1574ms
10:36:11 [s3-sync] pullChanged.remoteListing 1312ms keys=9175
10:36:13 [s3-sync] pullChanged.walk 1887ms toPull=0
10:36:13 [s3-sync] pullChanged.download 0ms pulled=0
10:36:13 [s3-sync] pullChanged.fastpath 25ms miss
10:36:13 [s3-sync] pullChanged.pullIndex 65ms
10:36:16 [s3-sync] pullChanged.remoteListing 2856ms keys=35818
10:36:22 [s3-sync] pullChanged.walk 6472ms toPull=1
10:36:23 [s3-sync] pullChanged.download 103ms pulled=1
10:36:23 [s3-sync] pullChanged.fastpath 20ms miss
10:36:25 [s3-sync] pullChanged.pullIndex 1185ms
10:36:25 [s3-sync] pullChanged.remoteListing 51ms keys=291
10:36:25 [s3-sync] pullChanged.walk 152ms toPull=0
10:36:25 [s3-sync] pullChanged.download 0ms pulled=0
10:36:25 [s3-sync] preparePush.fastpath 0ms miss
10:36:25 [s3-sync] preparePush.shardAssembly 245ms scoped=22
10:36:26 [s3-sync] preparePush.walk 972ms toPush=42 toDelete=0
10:36:26 [s3-sync] preparePush.upload 184ms pushed=30
10:36:26 [s3-sync] preparePush.delete 0ms deleted=0
10:36:27 [s3-sync] commitPush.shards 318ms dirty=11
10:36:27 [s3-sync] commitPush.localIndex 129ms
10:36:27 [s3-sync] commitPush 477ms shard-first changes=30
10:36:27 [s3-sync] pullChanged.fastpath 19ms miss
10:36:34 [s3-sync] pullChanged.pullIndex 6288ms
10:36:42 [s3-sync] pullChanged.remoteListing 3280ms keys=9195
10:36:43 [s3-sync] pullChanged.walk 1144ms toPull=0
10:36:43 [s3-sync] pullChanged.download 0ms pulled=0
10:36:43 [s3-sync] pullChanged.fastpath 28ms miss
10:36:43 [s3-sync] pullChanged.pullIndex 58ms
10:36:46 [s3-sync] pullChanged.remoteListing 2578ms keys=35844
10:36:50 [s3-sync] pullChanged.walk 4087ms toPull=1
10:36:50 [s3-sync] pullChanged.download 39ms pulled=1
10:36:50 [s3-sync] pullChanged.fastpath 27ms miss
10:36:52 [s3-sync] pullChanged.pullIndex 1182ms
10:36:52 [s3-sync] pullChanged.remoteListing 47ms keys=291
10:36:52 [s3-sync] pullChanged.walk 156ms toPull=0
10:36:52 [s3-sync] pullChanged.download 0ms pulled=0
10:36:52 [s3-sync] pullChanged.fastpath 26ms miss
10:36:54 [s3-sync] pullChanged.pullIndex 1119ms
10:36:55 [s3-sync] pullChanged.remoteListing 1252ms keys=9196
10:36:56 [s3-sync] pullChanged.walk 1165ms toPull=3
10:36:56 [s3-sync] pullChanged.download 30ms pulled=3
10:36:56 [s3-sync] preparePush.fastpath 0ms miss
10:36:56 [s3-sync] preparePush.shardAssembly 213ms scoped=32
10:36:57 [s3-sync] preparePush.walk 1085ms toPush=63 toDelete=0
10:36:58 [s3-sync] preparePush.upload 229ms pushed=39
10:36:58 [s3-sync] preparePush.delete 0ms deleted=0
10:36:58 [s3-sync] commitPush.shards 328ms dirty=8
10:36:58 [s3-sync] commitPush.localIndex 140ms
10:36:58 [s3-sync] commitPush 500ms shard-first changes=39
10:37:11 [s3-sync] pullChanged.fastpath 39ms miss
10:37:11 [s3-sync] pullChanged.pullIndex 52ms
10:37:16 [s3-sync] pullChanged.remoteListing 5162ms keys=35880
10:37:20 [s3-sync] pullChanged.walk 4208ms toPull=1
10:37:20 [s3-sync] pullChanged.download 38ms pulled=1
10:37:21 [s3-sync] pullChanged.fastpath 23ms miss
10:37:22 [s3-sync] pullChanged.pullIndex 1241ms
10:37:23 [s3-sync] pullChanged.remoteListing 65ms keys=291
10:37:23 [s3-sync] pullChanged.walk 149ms toPull=0
10:37:23 [s3-sync] pullChanged.download 0ms pulled=0
10:37:23 [s3-sync] pullChanged.fastpath 34ms miss
10:37:24 [s3-sync] pullChanged.pullIndex 1156ms
10:37:26 [s3-sync] pullChanged.remoteListing 1149ms keys=9229
10:37:27 [s3-sync] pullChanged.walk 1090ms toPull=1
10:37:27 [s3-sync] pullChanged.download 26ms pulled=1
10:37:32 [s3-sync] preparePush.fastpath 0ms miss
10:37:32 [s3-sync] preparePush.shardAssembly 224ms scoped=17
10:37:33 [s3-sync] preparePush.walk 881ms toPush=34 toDelete=0
10:37:33 [s3-sync] preparePush.upload 150ms pushed=25
10:37:33 [s3-sync] preparePush.delete 0ms deleted=0
10:37:33 [s3-sync] commitPush.shards 328ms dirty=9
10:37:33 [s3-sync] commitPush.localIndex 142ms
10:37:33 [s3-sync] commitPush 526ms shard-first changes=25
10:37:44 [s3-sync] pullChanged.fastpath 53ms miss
10:37:48 [s3-sync] pullChanged.pullIndex 3464ms
10:37:48 [s3-sync] pullChanged.remoteListing 68ms keys=291
10:37:48 [s3-sync] pullChanged.walk 162ms toPull=0
10:37:48 [s3-sync] pullChanged.download 0ms pulled=0
10:37:48 [s3-sync] pullChanged.fastpath 22ms miss
10:37:48 [s3-sync] pullChanged.pullIndex 63ms
10:37:51 [s3-sync] pullChanged.remoteListing 2621ms keys=35901
10:37:55 [s3-sync] pullChanged.walk 4211ms toPull=1
10:37:55 [s3-sync] pullChanged.download 41ms pulled=1
10:37:56 [s3-sync] pullChanged.fastpath 25ms miss
10:37:57 [s3-sync] pullChanged.pullIndex 1191ms
10:37:58 [s3-sync] pullChanged.remoteListing 1273ms keys=9247
10:38:00 [s3-sync] pullChanged.walk 1632ms toPull=1
10:38:00 [s3-sync] pullChanged.download 34ms pulled=1
10:38:03 Handler slow: "capability.queryData" for worker "lab-worker-2" took 6018ms
10:38:07 [s3-sync] preparePush.fastpath 0ms miss
10:38:07 [s3-sync] preparePush.shardAssembly 270ms scoped=23
10:38:08 [s3-sync] preparePush.walk 1032ms toPush=46 toDelete=0
10:38:08 [s3-sync] preparePush.upload 176ms pushed=31
10:38:08 [s3-sync] preparePush.delete 0ms deleted=0
10:38:09 [s3-sync] commitPush.shards 347ms dirty=9
10:38:09 [s3-sync] commitPush.localIndex 141ms
10:38:09 [s3-sync] commitPush 519ms shard-first changes=31
10:38:14 [s3-sync] pullChanged.fastpath 32ms miss
10:38:21 [s3-sync] pullChanged.pullIndex 6337ms
10:38:21 [s3-sync] pullChanged.remoteListing 64ms keys=291
10:38:21 [s3-sync] pullChanged.walk 144ms toPull=0
10:38:21 [s3-sync] pullChanged.download 0ms pulled=0
10:38:21 [s3-sync] pullChanged.fastpath 28ms miss
10:38:25 [s3-sync] pullChanged.pullIndex 3316ms
10:38:26 [s3-sync] pullChanged.remoteListing 1350ms keys=9271
10:38:27 [s3-sync] pullChanged.walk 1143ms toPull=0
10:38:27 [s3-sync] pullChanged.download 0ms pulled=0
10:38:27 [s3-sync] pullChanged.fastpath 29ms miss
10:38:27 [s3-sync] pullChanged.pullIndex 66ms
10:38:30 [s3-sync] pullChanged.remoteListing 3007ms keys=35928
10:38:34 [s3-sync] pullChanged.walk 4201ms toPull=1
10:38:34 [s3-sync] pullChanged.download 29ms pulled=1
10:38:37 [s3-sync] preparePush.fastpath 0ms miss
10:38:37 [s3-sync] preparePush.shardAssembly 232ms scoped=20
10:38:38 [s3-sync] preparePush.walk 914ms toPush=40 toDelete=0
10:38:38 [s3-sync] preparePush.upload 606ms pushed=28
10:38:38 [s3-sync] preparePush.delete 0ms deleted=0
10:38:39 [s3-sync] commitPush.shards 300ms dirty=9
10:38:39 [s3-sync] commitPush.localIndex 137ms
10:38:39 [s3-sync] commitPush 465ms shard-first changes=28
10:38:45 [s3-sync] pullChanged.fastpath 25ms miss
10:38:51 [s3-sync] pullChanged.pullIndex 6326ms
10:38:51 [s3-sync] pullChanged.remoteListing 55ms keys=291
10:38:51 [s3-sync] pullChanged.walk 160ms toPull=0
10:38:51 [s3-sync] pullChanged.download 0ms pulled=0
10:38:51 [s3-sync] pullChanged.fastpath 25ms miss
10:38:51 [s3-sync] pullChanged.pullIndex 55ms
10:38:54 [s3-sync] pullChanged.remoteListing 2494ms keys=35952
10:39:00 [s3-sync] pullChanged.walk 6443ms toPull=1
10:39:00 [s3-sync] pullChanged.download 55ms pulled=1
10:39:01 [s3-sync] pullChanged.fastpath 27ms miss
10:39:02 [s3-sync] pullChanged.pullIndex 1193ms
10:39:03 [s3-sync] pullChanged.remoteListing 1339ms keys=9292
10:39:05 [s3-sync] pullChanged.walk 1604ms toPull=1
10:39:05 [s3-sync] pullChanged.download 25ms pulled=1
10:39:15 [s3-sync] pullChanged.fastpath 63ms miss
10:39:16 [s3-sync] pullChanged.pullIndex 1202ms
10:39:16 [s3-sync] pullChanged.remoteListing 62ms keys=291
10:39:16 [s3-sync] pullChanged.walk 140ms toPull=0
10:39:16 [s3-sync] pullChanged.download 0ms pulled=0
10:39:17 [s3-sync] pullChanged.fastpath 31ms miss
10:39:18 [s3-sync] pullChanged.pullIndex 1174ms
10:39:19 [s3-sync] pullChanged.remoteListing 1100ms keys=9292
10:39:20 [s3-sync] pullChanged.walk 978ms toPull=0
10:39:20 [s3-sync] pullChanged.download 0ms pulled=0
10:39:26 [s3-sync] preparePush.fastpath 0ms miss
10:39:26 [s3-sync] preparePush.shardAssembly 333ms scoped=17
10:39:27 [s3-sync] preparePush.walk 1035ms toPush=34 toDelete=0
10:39:27 [s3-sync] preparePush.upload 167ms pushed=25
10:39:27 [s3-sync] preparePush.delete 0ms deleted=0
10:39:29 [s3-sync] commitPush.shards 2344ms dirty=9
10:39:29 [s3-sync] commitPush.localIndex 140ms
10:39:29 [s3-sync] commitPush 2515ms shard-first changes=25
10:39:41 [s3-sync] pullChanged.fastpath 40ms miss
10:39:41 [s3-sync] pullChanged.pullIndex 61ms
10:39:44 [s3-sync] pullChanged.remoteListing 2528ms keys=35973
10:39:48 [s3-sync] pullChanged.walk 4327ms toPull=1
10:39:48 [s3-sync] pullChanged.download 50ms pulled=1
10:39:49 [s3-sync] pullChanged.fastpath 27ms miss
10:39:50 [s3-sync] pullChanged.pullIndex 1259ms
10:39:50 [s3-sync] pullChanged.remoteListing 47ms keys=291
10:39:50 [s3-sync] pullChanged.walk 177ms toPull=0
10:39:50 [s3-sync] pullChanged.download 0ms pulled=0
10:39:51 [s3-sync] pullChanged.fastpath 35ms miss
10:39:52 [s3-sync] pullChanged.pullIndex 1145ms
10:39:53 [s3-sync] pullChanged.remoteListing 1106ms keys=9310
10:39:55 Handler slow: "capability.queryData" for worker "lab-worker-2" took 5076ms
10:39:55 [s3-sync] pullChanged.walk 2226ms toPull=1
10:39:55 [s3-sync] pullChanged.download 24ms pulled=1
10:39:55 Handler error: "capability.getData" for worker "lab-worker-2" after 2ms : "getData: behavior type '@ours/session-write' is outside the active dispatch scope"
10:39:55 Handler error: "capability.getData" for worker "lab-worker-2" after 0ms : "getData: behavior type '@ours/session-write' is outside the active dispatch scope"
10:39:55 Handler error: "capability.getData" for worker "lab-worker-2" after 0ms : "getData: behavior type '@ours/session-write' is outside the active dispatch scope"
10:39:55 Handler error: "capability.getData" for worker "lab-worker-2" after 0ms : "getData: behavior type '@ours/session-write' is outside the active dispatch scope"
10:39:59 Handler error: "capability.getData" for worker "lab-worker-2" after 0ms : "getData: behavior type '@ours/session-write' is outside the active dispatch scope"
10:39:59 Handler error: "capability.getData" for worker "lab-worker-2" after 0ms : "getData: behavior type '@ours/session-write' is outside the active dispatch scope"
10:39:59 Handler error: "capability.getData" for worker "lab-worker-2" after 0ms : "getData: behavior type '@ours/session-write' is outside the active dispatch scope"
10:39:59 Handler error: "capability.getData" for worker "lab-worker-2" after 0ms : "getData: behavior type '@ours/session-write' is outside the active dispatch scope"
10:39:59 [s3-sync] preparePush.fastpath 0ms miss
10:40:02 [s3-sync] preparePush.shardAssembly 2613ms scoped=35
10:40:03 [s3-sync] preparePush.walk 3579ms toPush=70 toDelete=0
10:40:04 [s3-sync] preparePush.upload 865ms pushed=43
10:40:04 [s3-sync] preparePush.delete 0ms deleted=0
10:40:04 [s3-sync] commitPush.shards 345ms dirty=9
10:40:04 [s3-sync] commitPush.localIndex 131ms
10:40:04 [s3-sync] commitPush 502ms shard-first changes=43
10:40:15 [s3-sync] pullChanged.fastpath 47ms miss
10:40:17 [s3-sync] pullChanged.pullIndex 1282ms
10:40:17 [s3-sync] pullChanged.remoteListing 43ms keys=291
10:40:17 [s3-sync] pullChanged.walk 150ms toPull=0
10:40:17 [s3-sync] pullChanged.download 0ms pulled=0
10:40:17 [s3-sync] pullChanged.fastpath 27ms miss
10:40:17 [s3-sync] pullChanged.pullIndex 59ms
10:40:19 [s3-sync] pullChanged.remoteListing 2289ms keys=36012
10:40:24 [s3-sync] pullChanged.walk 4190ms toPull=1
10:40:24 [s3-sync] pullChanged.download 33ms pulled=1
10:40:24 [s3-sync] pullChanged.fastpath 33ms miss
10:40:26 [s3-sync] pullChanged.pullIndex 1179ms
10:40:27 [s3-sync] pullChanged.remoteListing 1372ms keys=9346
10:40:28 [s3-sync] pullChanged.walk 1170ms toPull=3
10:40:28 [s3-sync] pullChanged.download 31ms pulled=3
10:40:28 [s3-sync] preparePush.fastpath 0ms miss
10:40:29 [s3-sync] preparePush.shardAssembly 231ms scoped=20
10:40:29 [s3-sync] preparePush.walk 945ms toPush=39 toDelete=0
10:40:30 [s3-sync] preparePush.upload 179ms pushed=27
10:40:30 [s3-sync] preparePush.delete 0ms deleted=0
10:40:30 [s3-sync] commitPush.shards 339ms dirty=8
10:40:30 [s3-sync] commitPush.localIndex 164ms
10:40:30 [s3-sync] commitPush 536ms shard-first changes=27
10:40:45 [s3-sync] pullChanged.fastpath 103ms miss
10:40:47 [s3-sync] pullChanged.pullIndex 1221ms
10:40:47 [s3-sync] pullChanged.remoteListing 70ms keys=291
10:40:47 [s3-sync] pullChanged.walk 151ms toPull=0
10:40:47 [s3-sync] pullChanged.download 0ms pulled=0
10:40:47 [s3-sync] pullChanged.fastpath 30ms miss
10:40:47 [s3-sync] pullChanged.pullIndex 62ms
10:40:50 [s3-sync] pullChanged.remoteListing 2474ms keys=36036
10:40:56 [s3-sync] pullChanged.walk 6120ms toPull=1
10:40:56 [s3-sync] pullChanged.download 42ms pulled=1
10:40:56 [s3-sync] pullChanged.fastpath 23ms miss
10:40:58 [s3-sync] pullChanged.pullIndex 1225ms
10:40:59 [s3-sync] pullChanged.remoteListing 1109ms keys=9367
10:41:01 [s3-sync] pullChanged.walk 1668ms toPull=1
10:41:01 [s3-sync] pullChanged.download 28ms pulled=1
10:41:04 Handler slow: "capability.queryData" for worker "lab-worker-2" took 5542ms
10:41:15 [s3-sync] pullChanged.fastpath 36ms miss
10:41:17 [s3-sync] pullChanged.pullIndex 1241ms
10:41:17 [s3-sync] pullChanged.remoteListing 67ms keys=291
10:41:17 [s3-sync] pullChanged.walk 192ms toPull=0
10:41:17 [s3-sync] pullChanged.download 0ms pulled=0
10:41:17 [s3-sync] pullChanged.fastpath 25ms miss
10:41:19 [s3-sync] pullChanged.pullIndex 1177ms
10:41:20 [s3-sync] pullChanged.remoteListing 1118ms keys=9367
10:41:21 [s3-sync] pullChanged.walk 873ms toPull=0
10:41:21 [s3-sync] pullChanged.download 0ms pulled=0
10:41:45 [s3-sync] preparePush.fastpath 0ms miss
10:41:45 [s3-sync] preparePush.shardAssembly 292ms scoped=17
10:41:46 [s3-sync] preparePush.walk 990ms toPush=34 toDelete=0
10:41:46 [s3-sync] preparePush.upload 154ms pushed=25
10:41:46 [s3-sync] preparePush.delete 0ms deleted=0
10:41:47 [s3-sync] commitPush.shards 329ms dirty=9
10:41:47 [s3-sync] commitPush.localIndex 134ms
10:41:47 [s3-sync] commitPush 495ms shard-first changes=25
10:41:49 [s3-sync] pullChanged.fastpath 1750ms miss
10:41:59 [s3-sync] pullChanged.pullIndex 9783ms
10:41:59 [s3-sync] pullChanged.remoteListing 80ms keys=291
10:42:00 [s3-sync] pullChanged.walk 260ms toPull=0
10:42:00 [s3-sync] pullChanged.download 0ms pulled=0
10:42:00 [s3-sync] pullChanged.fastpath 25ms miss
10:42:00 [s3-sync] pullChanged.pullIndex 66ms
10:42:03 [s3-sync] pullChanged.remoteListing 2533ms keys=36057
10:42:07 [s3-sync] pullChanged.walk 4288ms toPull=1
10:42:07 [s3-sync] pullChanged.download 33ms pulled=1
10:42:07 [s3-sync] pullChanged.fastpath 19ms miss
10:42:09 [s3-sync] pullChanged.pullIndex 1234ms
10:42:10 [s3-sync] pullChanged.remoteListing 1358ms keys=9385
10:42:12 [s3-sync] pullChanged.walk 1722ms toPull=1
10:42:12 [s3-sync] pullChanged.download 31ms pulled=1
10:42:14 Handler error: "capability.getData" for worker "lab-worker-2" after 0ms : "getData: behavior type '@ours/session-write' is outside the active dispatch scope"
10:42:16 [s3-sync] pullChanged.fastpath 78ms miss
10:42:17 [s3-sync] pullChanged.pullIndex 1138ms
10:42:18 [s3-sync] pullChanged.remoteListing 56ms keys=291
10:42:18 [s3-sync] pullChanged.walk 175ms toPull=0
10:42:18 [s3-sync] pullChanged.download 0ms pulled=0
10:42:18 [s3-sync] pullChanged.fastpath 29ms miss
10:42:19 [s3-sync] pullChanged.pullIndex 1124ms
10:42:21 [s3-sync] pullChanged.remoteListing 1202ms keys=9385
10:42:22 Handler slow: "capability.queryData" for worker "lab-worker-2" took 7859ms
10:42:22 Handler error: "capability.getData" for worker "lab-worker-2" after 0ms : "getData: behavior type '@ours/session-write' is outside the active dispatch scope"
10:42:23 [s3-sync] pullChanged.walk 1878ms toPull=0
10:42:23 [s3-sync] pullChanged.download 0ms pulled=0
10:42:23 [s3-sync] preparePush.fastpath 0ms miss
10:42:23 [s3-sync] preparePush.shardAssembly 221ms scoped=26
10:42:24 [s3-sync] preparePush.walk 1055ms toPush=52 toDelete=0
10:42:24 [s3-sync] preparePush.upload 249ms pushed=34
10:42:24 [s3-sync] preparePush.delete 0ms deleted=0
10:42:24 [s3-sync] commitPush.shards 305ms dirty=9
10:42:25 [s3-sync] commitPush.localIndex 140ms
10:42:25 [s3-sync] commitPush 470ms shard-first changes=34
10:42:42 [s3-sync] pushChanged.fastpath 0ms miss
10:42:42 [s3-sync] pushChanged.shardAssembly 162ms scoped=3
10:42:42 [s3-sync] pushChanged.walk 52ms toPush=7 toDelete=0
10:42:43 [s3-sync] pushChanged.upload 79ms pushed=7
10:42:43 [s3-sync] pushChanged.delete 1ms deleted=0
10:42:43 [s3-sync] pushChanged.v2writeback 285ms
10:42:48 [s3-sync] pullChanged.fastpath 33ms miss
10:42:52 [s3-sync] pullChanged.pullIndex 3692ms
10:42:52 [s3-sync] pullChanged.remoteListing 42ms keys=291
10:42:52 [s3-sync] pullChanged.walk 159ms toPull=0
10:42:52 [s3-sync] pullChanged.download 0ms pulled=0
10:42:52 [s3-sync] pullChanged.fastpath 20ms miss
10:42:53 [s3-sync] pullChanged.pullIndex 1150ms
10:42:55 [s3-sync] pullChanged.remoteListing 1167ms keys=9415
10:42:56 [s3-sync] pullChanged.walk 1270ms toPull=0
10:42:56 [s3-sync] pullChanged.download 0ms pulled=0
10:43:18 [s3-sync] pullChanged.fastpath 51ms miss
10:43:19 [s3-sync] pullChanged.pullIndex 1428ms
10:43:20 [s3-sync] pullChanged.remoteListing 72ms keys=291
10:43:20 [s3-sync] pullChanged.walk 201ms toPull=0
10:43:20 [s3-sync] pullChanged.download 0ms pulled=0
10:43:20 [s3-sync] pullChanged.fastpath 26ms miss
10:43:23 [s3-sync] pullChanged.pullIndex 3377ms
10:43:25 [s3-sync] pullChanged.remoteListing 1233ms keys=9425
10:43:26 [s3-sync] pullChanged.walk 873ms toPull=0
10:43:26 [s3-sync] pullChanged.download 0ms pulled=0
10:43:52 [s3-sync] pullChanged.fastpath 3594ms miss
10:43:53 [s3-sync] pullChanged.pullIndex 1305ms
10:43:53 [s3-sync] pullChanged.remoteListing 93ms keys=291
10:43:53 [s3-sync] pullChanged.walk 318ms toPull=0
10:43:53 [s3-sync] pullChanged.download 0ms pulled=0
10:43:56 [s3-sync] pullChanged.fastpath 1915ms miss
10:43:57 [s3-sync] pullChanged.pullIndex 1233ms
10:43:58 [s3-sync] pullChanged.remoteListing 1277ms keys=9425
10:43:59 [s3-sync] pullChanged.walk 956ms toPull=0
10:43:59 [s3-sync] pullChanged.download 0ms pulled=0
10:44:18 [s3-sync] pullChanged.fastpath 40ms miss
10:44:19 [s3-sync] pullChanged.pullIndex 1292ms
10:44:19 [s3-sync] pullChanged.remoteListing 73ms keys=291
10:44:20 [s3-sync] pullChanged.walk 169ms toPull=0
10:44:20 [s3-sync] pullChanged.download 0ms pulled=0
10:44:20 [s3-sync] pullChanged.fastpath 28ms miss
10:44:21 [s3-sync] pullChanged.pullIndex 1180ms
10:44:23 [s3-sync] pullChanged.remoteListing 1353ms keys=9425
10:44:24 [s3-sync] pullChanged.walk 1070ms toPull=0
10:44:24 [s3-sync] pullChanged.download 0ms pulled=0
10:44:50 [s3-sync] pullChanged.fastpath 39ms miss
10:45:01 [s3-sync] pullChanged.pullIndex 11226ms
10:45:03 [s3-sync] pullChanged.remoteListing 100ms keys=291
10:45:09 [s3-sync] pullChanged.walk 5256ms toPull=0
10:45:09 [s3-sync] pullChanged.download 0ms pulled=0
10:45:14 [s3-sync] pullChanged.fastpath 3311ms miss
10:45:26 [s3-sync] pullChanged.pullIndex 11973ms
10:45:39 [s3-sync] pullChanged.remoteListing 13150ms keys=9425
10:45:51 [s3-sync] pullChanged.walk 11059ms toPull=0
10:45:51 [s3-sync] pullChanged.download 0ms pulled=0
10:45:51 [s3-sync] pullChanged.fastpath 46ms miss
10:46:07 [s3-sync] pullChanged.pullIndex 16297ms
10:46:08 [s3-sync] pullChanged.remoteListing 88ms keys=291
10:46:08 [s3-sync] pullChanged.walk 142ms toPull=0
10:46:08 [s3-sync] pullChanged.download 0ms pulled=0
10:46:21 [s3-sync] pullChanged.fastpath 38ms miss
10:46:22 [s3-sync] pullChanged.pullIndex 1158ms
10:46:23 [s3-sync] pullChanged.remoteListing 73ms keys=291
10:46:23 [s3-sync] pullChanged.walk 152ms toPull=0
10:46:23 [s3-sync] pullChanged.download 0ms pulled=0
10:46:23 [s3-sync] pushChanged.fastpath 0ms miss
10:46:23 [s3-sync] pushChanged.shardAssembly 130ms scoped=5
10:46:23 [s3-sync] pushChanged.walk 76ms toPush=9 toDelete=0
10:46:23 [s3-sync] pushChanged.upload 70ms pushed=9
10:46:23 [s3-sync] pushChanged.delete 0ms deleted=0
10:46:23 [s3-sync] pushChanged.v2writeback 399ms
10:46:24 [s3-sync] pullChanged.fastpath 24ms miss
10:46:25 [s3-sync] pullChanged.pullIndex 1150ms
10:46:36 [s3-sync] pullChanged.remoteListing 10604ms keys=9429
10:46:45 [s3-sync] pullChanged.walk 9105ms toPull=0
10:46:45 [s3-sync] pullChanged.download 0ms pulled=0
10:46:52 [s3-sync] pullChanged.fastpath 46ms miss
10:46:59 [s3-sync] pullChanged.pullIndex 6207ms
10:46:59 [s3-sync] pullChanged.remoteListing 73ms keys=291
10:46:59 [s3-sync] pullChanged.walk 155ms toPull=0
10:46:59 [s3-sync] pullChanged.download 0ms pulled=0
10:47:04 [s3-sync] pullChanged.fastpath 2863ms miss
10:47:10 [s3-sync] pullChanged.pullIndex 6165ms
10:47:22 [s3-sync] pullChanged.remoteListing 11201ms keys=9429
10:47:35 [s3-sync] pullChanged.walk 13322ms toPull=0
10:47:35 [s3-sync] pullChanged.download 0ms pulled=0
10:47:41 [s3-sync] pullChanged.fastpath 4740ms miss
10:47:47 [s3-sync] pullChanged.pullIndex 6330ms
10:47:47 [s3-sync] pullChanged.remoteListing 72ms keys=291
10:47:48 [s3-sync] pullChanged.walk 158ms toPull=0
10:47:48 [s3-sync] pullChanged.download 0ms pulled=0
10:47:54 [s3-sync] pullChanged.fastpath 42ms miss
10:48:02 [s3-sync] pullChanged.pullIndex 8677ms
10:48:03 [s3-sync] pullChanged.remoteListing 83ms keys=291
10:48:03 [s3-sync] pullChanged.walk 147ms toPull=0
10:48:03 [s3-sync] pullChanged.download 0ms pulled=0
10:48:03 [s3-sync] pushChanged.fastpath 0ms miss
10:48:03 [s3-sync] pushChanged.shardAssembly 133ms scoped=5
10:48:03 [s3-sync] pushChanged.walk 68ms toPush=9 toDelete=0
10:48:03 [s3-sync] pushChanged.upload 66ms pushed=9
10:48:03 [s3-sync] pushChanged.delete 0ms deleted=0
10:48:03 [s3-sync] pushChanged.v2writeback 362ms
10:48:04 [s3-sync] pullChanged.fastpath 27ms miss
10:48:05 [s3-sync] pullChanged.pullIndex 1154ms
10:48:13 [s3-sync] pullChanged.remoteListing 8374ms keys=9433
10:48:24 [s3-sync] pullChanged.walk 11078ms toPull=0
10:48:24 [s3-sync] pullChanged.download 0ms pulled=0
10:48:30 [s3-sync] pullChanged.fastpath 3070ms miss
10:48:31 [s3-sync] pullChanged.pullIndex 1185ms
10:48:31 [s3-sync] pullChanged.remoteListing 86ms keys=291
10:48:31 [s3-sync] pullChanged.walk 144ms toPull=0
10:48:31 [s3-sync] pullChanged.download 0ms pulled=0
10:48:55 [s3-sync] pullChanged.fastpath 41ms miss
10:49:01 [s3-sync] pullChanged.pullIndex 6173ms
10:49:06 [s3-sync] pullChanged.remoteListing 79ms keys=291
10:49:06 [s3-sync] pullChanged.walk 211ms toPull=0
10:49:06 [s3-sync] pullChanged.download 0ms pulled=0
10:49:11 [s3-sync] pullChanged.fastpath 3071ms miss
10:49:25 [s3-sync] pullChanged.pullIndex 13276ms
10:49:31 [s3-sync] pullChanged.remoteListing 6346ms keys=9433
10:49:38 [s3-sync] pullChanged.walk 6542ms toPull=2
10:49:38 [s3-sync] pullChanged.download 65ms pulled=2
10:49:38 [s3-sync] pushChanged.fastpath 0ms miss
10:49:38 [s3-sync] pushChanged.shardAssembly 150ms scoped=5
10:49:38 [s3-sync] pushChanged.walk 56ms toPush=7 toDelete=0
10:49:38 [s3-sync] pushChanged.upload 78ms pushed=7
10:49:38 [s3-sync] pushChanged.delete 0ms deleted=0
10:49:39 [s3-sync] pushChanged.v2writeback 330ms
10:49:39 [s3-sync] pullChanged.fastpath 22ms miss
10:49:40 [s3-sync] pullChanged.pullIndex 1534ms
10:49:40 [s3-sync] pullChanged.remoteListing 62ms keys=291
10:49:41 [s3-sync] pullChanged.walk 197ms toPull=0
10:49:41 [s3-sync] pullChanged.download 0ms pulled=0
10:49:57 [s3-sync] pullChanged.fastpath 109ms miss
10:50:08 [s3-sync] pullChanged.pullIndex 11403ms
10:50:09 [s3-sync] pullChanged.remoteListing 62ms keys=291
10:50:09 [s3-sync] pullChanged.walk 152ms toPull=0
10:50:09 [s3-sync] pullChanged.download 0ms pulled=0
10:50:14 [s3-sync] pullChanged.fastpath 3101ms miss
10:50:20 [s3-sync] pullChanged.pullIndex 6175ms
10:50:31 [s3-sync] pullChanged.remoteListing 11186ms keys=9437
10:50:39 [s3-sync] pullChanged.walk 8151ms toPull=0
10:50:39 [s3-sync] pullChanged.download 0ms pulled=0
10:50:45 [s3-sync] pullChanged.fastpath 3226ms miss
10:50:57 [s3-sync] pullChanged.pullIndex 11457ms
10:50:57 [s3-sync] pullChanged.remoteListing 77ms keys=291
10:50:57 [s3-sync] pullChanged.walk 149ms toPull=0
10:50:57 [s3-sync] pullChanged.download 0ms pulled=0
10:51:00 [s3-sync] pullChanged.fastpath 1593ms miss
10:51:09 [s3-sync] pullChanged.pullIndex 9216ms
10:51:09 [s3-sync] pullChanged.remoteListing 50ms keys=291
10:51:09 [s3-sync] pullChanged.walk 144ms toPull=0
10:51:09 [s3-sync] pullChanged.download 0ms pulled=0
10:51:09 [s3-sync] pullChanged.fastpath 26ms miss
10:51:24 [s3-sync] pullChanged.pullIndex 14425ms
10:51:25 [s3-sync] pullChanged.remoteListing 1133ms keys=9437
10:51:26 [s3-sync] pullChanged.walk 1252ms toPull=2
10:51:26 [s3-sync] pullChanged.download 28ms pulled=2
10:51:27 [s3-sync] pushChanged.fastpath 0ms miss
10:51:27 [s3-sync] pushChanged.shardAssembly 139ms scoped=5
10:51:27 [s3-sync] pushChanged.walk 68ms toPush=7 toDelete=0
10:51:27 [s3-sync] pushChanged.upload 62ms pushed=7
10:51:27 [s3-sync] pushChanged.delete 0ms deleted=0
10:51:27 [s3-sync] pushChanged.v2writeback 317ms
10:51:28 [s3-sync] pullChanged.fastpath 25ms miss
10:51:29 [s3-sync] pullChanged.pullIndex 1167ms
10:51:29 [s3-sync] pullChanged.remoteListing 72ms keys=291
10:51:30 [s3-sync] pullChanged.walk 168ms toPull=0
10:51:30 [s3-sync] pullChanged.download 0ms pulled=0
10:51:32 [s3-sync] pullChanged.fastpath 25ms miss
10:51:43 [s3-sync] pullChanged.pullIndex 11261ms
10:51:57 [s3-sync] pullChanged.remoteListing 13552ms keys=9441
10:52:08 [s3-sync] pullChanged.walk 11217ms toPull=0
10:52:08 [s3-sync] pullChanged.download 0ms pulled=0
10:52:11 [s3-sync] pullChanged.fastpath 2231ms miss
10:52:26 [s3-sync] pullChanged.pullIndex 15158ms
10:52:26 [s3-sync] pullChanged.remoteListing 79ms keys=291
10:52:26 [s3-sync] pullChanged.walk 133ms toPull=0
10:52:26 [s3-sync] pullChanged.download 0ms pulled=0
10:52:31 [s3-sync] pullChanged.fastpath 25ms miss
10:52:45 [s3-sync] pullChanged.pullIndex 13503ms
10:52:45 [s3-sync] pullChanged.remoteListing 57ms keys=291
10:52:45 [s3-sync] pullChanged.walk 145ms toPull=0
10:52:45 [s3-sync] pullChanged.download 0ms pulled=0
10:52:48 [s3-sync] pullChanged.fastpath 1868ms miss
10:52:57 [s3-sync] pullChanged.pullIndex 9144ms
10:53:03 [s3-sync] pullChanged.remoteListing 6292ms keys=9441
10:53:07 [s3-sync] pullChanged.walk 3394ms toPull=2
10:53:07 [s3-sync] pullChanged.download 37ms pulled=2
10:53:07 [s3-sync] pushChanged.fastpath 0ms miss
10:53:07 [s3-sync] pushChanged.shardAssembly 158ms scoped=5
10:53:07 [s3-sync] pushChanged.walk 56ms toPush=7 toDelete=0
10:53:07 [s3-sync] pushChanged.upload 79ms pushed=7
10:53:07 [s3-sync] pushChanged.delete 0ms deleted=0
10:53:08 [s3-sync] pushChanged.v2writeback 336ms
10:53:08 [s3-sync] pullChanged.fastpath 24ms miss
10:53:09 [s3-sync] pullChanged.pullIndex 1167ms
10:53:10 [s3-sync] pullChanged.remoteListing 57ms keys=291
10:53:10 [s3-sync] pullChanged.walk 161ms toPull=0vcjdeboer commented 10/31/2026, 12:40:54 PM
Full [s3-sync] / Handler slow / Handler error / heap-limit log of the traced sitting, part 3 of 4 (10:53:10-11🔞45 UTC):
10:53:10 [s3-sync] pullChanged.download 0ms pulled=0
10:53:35 [s3-sync] pullChanged.fastpath 37ms miss
10:53:38 [s3-sync] pullChanged.pullIndex 3132ms
10:53:39 [s3-sync] pullChanged.remoteListing 420ms keys=291
10:53:39 [s3-sync] pullChanged.walk 157ms toPull=0
10:53:39 [s3-sync] pullChanged.download 0ms pulled=0
10:53:39 [s3-sync] pullChanged.fastpath 32ms miss
10:53:39 [s3-sync] pullChanged.pullIndex 68ms
10:53:42 [s3-sync] pullChanged.remoteListing 2892ms keys=36133
10:53:47 [s3-sync] pullChanged.walk 5223ms toPull=1
10:53:47 [s3-sync] pullChanged.download 26ms pulled=1
10:53:48 [s3-sync] pullChanged.fastpath 27ms miss
10:53:50 [s3-sync] pullChanged.pullIndex 1185ms
10:53:51 [s3-sync] pullChanged.remoteListing 1118ms keys=9446
10:53:52 [s3-sync] pullChanged.walk 1211ms toPull=3
10:53:52 [s3-sync] pullChanged.download 31ms pulled=3
10:53:52 [s3-sync] preparePush.fastpath 0ms miss
10:53:52 [s3-sync] preparePush.shardAssembly 234ms scoped=22
10:53:53 [s3-sync] preparePush.walk 952ms toPush=41 toDelete=0
10:53:53 [s3-sync] preparePush.upload 185ms pushed=29
10:53:53 [s3-sync] preparePush.delete 0ms deleted=0
10:53:54 [s3-sync] commitPush.shards 346ms dirty=10
10:53:54 [s3-sync] commitPush.localIndex 124ms
10:53:54 [s3-sync] commitPush 502ms shard-first changes=29
10:54:05 [s3-sync] pullChanged.fastpath 44ms miss
10:54:06 [s3-sync] pullChanged.pullIndex 1165ms
10:54:07 [s3-sync] pullChanged.remoteListing 72ms keys=291
10:54:07 [s3-sync] pullChanged.walk 153ms toPull=0
10:54:07 [s3-sync] pullChanged.download 0ms pulled=0
10:54:07 [s3-sync] pullChanged.fastpath 23ms miss
10:54:07 [s3-sync] pullChanged.pullIndex 63ms
10:54:12 [s3-sync] pullChanged.remoteListing 4902ms keys=36159
10:54:16 [s3-sync] pullChanged.walk 4257ms toPull=1
10:54:16 [s3-sync] pullChanged.download 39ms pulled=1
10:54:16 [s3-sync] pullChanged.fastpath 22ms miss
10:54:18 [s3-sync] pullChanged.pullIndex 1233ms
10:54:19 [s3-sync] pullChanged.remoteListing 1130ms keys=9467
10:54:20 [s3-sync] pullChanged.walk 1229ms toPull=3
10:54:20 [s3-sync] pullChanged.download 34ms pulled=3
10:54:20 [s3-sync] preparePush.fastpath 0ms miss
10:54:20 [s3-sync] preparePush.shardAssembly 260ms scoped=14
10:54:21 [s3-sync] preparePush.walk 923ms toPush=27 toDelete=0
10:54:22 [s3-sync] preparePush.upload 678ms pushed=21
10:54:22 [s3-sync] preparePush.delete 0ms deleted=0
10:54:22 [s3-sync] commitPush.shards 343ms dirty=8
10:54:23 [s3-sync] commitPush.localIndex 127ms
10:54:23 [s3-sync] commitPush 497ms shard-first changes=21
10:54:35 [s3-sync] pullChanged.fastpath 33ms miss
10:54:41 [s3-sync] pullChanged.pullIndex 6182ms
10:54:47 [s3-sync] pullChanged.remoteListing 3137ms keys=291
10:54:47 [s3-sync] pullChanged.walk 207ms toPull=0
10:54:47 [s3-sync] pullChanged.download 0ms pulled=0
10:54:49 [s3-sync] pullChanged.fastpath 49ms miss
10:55:00 [s3-sync] pullChanged.pullIndex 11572ms
10:55:11 [s3-sync] pullChanged.remoteListing 10880ms keys=9481
10:55:18 [s3-sync] pullChanged.walk 6170ms toPull=0
10:55:18 [s3-sync] pullChanged.download 0ms pulled=0
10:55:23 [s3-sync] pullChanged.fastpath 3049ms miss
10:55:31 [s3-sync] pullChanged.pullIndex 8518ms
10:55:32 [s3-sync] pullChanged.remoteListing 64ms keys=291
10:55:32 [s3-sync] pullChanged.walk 145ms toPull=0
10:55:32 [s3-sync] pullChanged.download 0ms pulled=0
10:55:32 [s3-sync] pushChanged.fastpath 0ms miss
10:55:32 [s3-sync] pushChanged.shardAssembly 122ms scoped=3
10:55:32 [s3-sync] pushChanged.walk 71ms toPush=7 toDelete=0
10:55:32 [s3-sync] pushChanged.upload 74ms pushed=7
10:55:32 [s3-sync] pushChanged.delete 0ms deleted=0
10:55:32 [s3-sync] pushChanged.v2writeback 282ms
10:55:36 [s3-sync] pullChanged.fastpath 22ms miss
10:55:48 [s3-sync] pullChanged.pullIndex 11380ms
10:55:48 [s3-sync] pullChanged.remoteListing 45ms keys=291
10:55:48 [s3-sync] pullChanged.walk 145ms toPull=0
10:55:48 [s3-sync] pullChanged.download 0ms pulled=0
10:55:49 [s3-sync] pullChanged.fastpath 25ms miss
10:56:02 [s3-sync] pullChanged.pullIndex 13081ms
10:56:03 [s3-sync] pullChanged.remoteListing 1326ms keys=9485
10:56:05 [s3-sync] pullChanged.walk 1197ms toPull=0
10:56:05 [s3-sync] pullChanged.download 0ms pulled=0
10:56:05 [s3-sync] pullChanged.fastpath 19ms miss
10:56:05 [s3-sync] pullChanged.pullIndex 65ms
10:56:07 [s3-sync] pullChanged.remoteListing 2517ms keys=36181
10:56:11 [s3-sync] pullChanged.walk 4156ms toPull=1
10:56:11 [s3-sync] pullChanged.download 42ms pulled=1
10:56:12 [s3-sync] pullChanged.fastpath 24ms miss
10:56:14 [s3-sync] pullChanged.pullIndex 1190ms
10:56:14 [s3-sync] pullChanged.remoteListing 64ms keys=291
10:56:14 [s3-sync] pullChanged.walk 190ms toPull=0
10:56:14 [s3-sync] pullChanged.download 0ms pulled=0
10:56:14 [s3-sync] preparePush.fastpath 0ms miss
10:56:14 [s3-sync] preparePush.shardAssembly 221ms scoped=22
10:56:15 [s3-sync] preparePush.walk 988ms toPush=42 toDelete=0
10:56:15 [s3-sync] preparePush.upload 181ms pushed=30
10:56:15 [s3-sync] preparePush.delete 0ms deleted=0
10:56:16 [s3-sync] commitPush.shards 334ms dirty=11
10:56:16 [s3-sync] commitPush.localIndex 123ms
10:56:16 [s3-sync] commitPush 478ms shard-first changes=30
10:56:21 [s3-sync] pullChanged.fastpath 2892ms miss
10:56:28 [s3-sync] pullChanged.pullIndex 6214ms
10:56:29 [s3-sync] pullChanged.remoteListing 1137ms keys=9507
10:56:30 [s3-sync] pullChanged.walk 1254ms toPull=0
10:56:30 [s3-sync] pullChanged.download 0ms pulled=0
10:56:30 [s3-sync] pullChanged.fastpath 27ms miss
10:56:30 [s3-sync] pullChanged.pullIndex 60ms
10:56:35 [s3-sync] pullChanged.remoteListing 4687ms keys=36207
10:56:39 [s3-sync] pullChanged.walk 4255ms toPull=1
10:56:39 [s3-sync] pullChanged.download 46ms pulled=1
10:56:39 [s3-sync] pullChanged.fastpath 19ms miss
10:56:41 [s3-sync] pullChanged.pullIndex 1161ms
10:56:41 [s3-sync] pullChanged.remoteListing 43ms keys=291
10:56:41 [s3-sync] pullChanged.walk 139ms toPull=0
10:56:41 [s3-sync] pullChanged.download 0ms pulled=0
10:56:41 [s3-sync] preparePush.fastpath 0ms miss
10:56:42 [s3-sync] preparePush.shardAssembly 212ms scoped=14
10:56:42 [s3-sync] preparePush.walk 879ms toPush=28 toDelete=0
10:56:43 [s3-sync] preparePush.upload 644ms pushed=22
10:56:43 [s3-sync] preparePush.delete 0ms deleted=0
10:56:43 [s3-sync] commitPush.shards 338ms dirty=9
10:56:44 [s3-sync] commitPush.localIndex 134ms
10:56:44 [s3-sync] commitPush 503ms shard-first changes=22
10:56:49 [s3-sync] pullChanged.fastpath 2871ms miss
10:56:55 [s3-sync] pullChanged.pullIndex 6340ms
10:57:02 [s3-sync] pullChanged.remoteListing 6190ms keys=9521
10:57:16 [s3-sync] pullChanged.walk 13924ms toPull=0
10:57:16 [s3-sync] pullChanged.download 0ms pulled=0
10:57:21 [s3-sync] pullChanged.fastpath 3066ms miss
10:57:27 [s3-sync] pullChanged.pullIndex 6172ms
10:57:27 [s3-sync] pullChanged.remoteListing 61ms keys=291
10:57:27 [s3-sync] pullChanged.walk 153ms toPull=0
10:57:27 [s3-sync] pullChanged.download 0ms pulled=0
10:57:39 [s3-sync] pullChanged.fastpath 43ms miss
10:57:42 [s3-sync] pullChanged.pullIndex 3094ms
10:57:47 [s3-sync] pullChanged.remoteListing 3007ms keys=291
10:57:48 [s3-sync] pullChanged.walk 324ms toPull=0
10:57:48 [s3-sync] pullChanged.download 0ms pulled=0
10:57:53 [s3-sync] pullChanged.fastpath 3071ms miss
10:57:54 [s3-sync] pullChanged.pullIndex 1165ms
10:57:55 [s3-sync] pullChanged.remoteListing 1105ms keys=9521
10:57:57 [s3-sync] pullChanged.walk 1207ms toPull=2
10:57:57 [s3-sync] pullChanged.download 31ms pulled=2
10:57:57 [s3-sync] pushChanged.fastpath 0ms miss
10:57:57 [s3-sync] pushChanged.shardAssembly 124ms scoped=3
10:57:57 [s3-sync] pushChanged.walk 63ms toPush=5 toDelete=0
10:57:57 [s3-sync] pushChanged.upload 80ms pushed=5
10:57:57 [s3-sync] pushChanged.delete 0ms deleted=0
10:57:57 [s3-sync] pushChanged.v2writeback 298ms
10:58:12 [s3-sync] pullChanged.fastpath 3044ms miss
10:58:13 [s3-sync] pullChanged.pullIndex 1079ms
10:58:14 [s3-sync] pullChanged.remoteListing 61ms keys=291
10:58:14 [s3-sync] pullChanged.walk 161ms toPull=0
10:58:14 [s3-sync] pullChanged.download 0ms pulled=0
10:58:19 [s3-sync] pullChanged.fastpath 3075ms miss
10:58:27 [s3-sync] pullChanged.pullIndex 8381ms
10:58:29 [s3-sync] pullChanged.remoteListing 1237ms keys=9525
10:58:30 [s3-sync] pullChanged.walk 1268ms toPull=0
10:58:30 [s3-sync] pullChanged.download 0ms pulled=0
10:58:30 [s3-sync] pullChanged.fastpath 26ms miss
10:58:30 [s3-sync] pullChanged.pullIndex 54ms
10:58:33 [s3-sync] pullChanged.remoteListing 3048ms keys=36229
10:58:37 [s3-sync] pullChanged.walk 4130ms toPull=1
10:58:37 [s3-sync] pullChanged.download 38ms pulled=1
10:58:39 [s3-sync] preparePush.fastpath 0ms miss
10:58:40 [s3-sync] preparePush.shardAssembly 259ms scoped=22
10:58:40 [s3-sync] preparePush.walk 972ms toPush=42 toDelete=0
10:58:41 [s3-sync] preparePush.upload 784ms pushed=30
10:58:41 [s3-sync] preparePush.delete 0ms deleted=0
10:58:42 [s3-sync] commitPush.shards 319ms dirty=11
10:58:42 [s3-sync] commitPush.localIndex 157ms
10:58:42 [s3-sync] commitPush 503ms shard-first changes=30
10:58:47 [s3-sync] pullChanged.fastpath 4762ms miss
10:58:48 [s3-sync] pullChanged.pullIndex 1126ms
10:58:49 [s3-sync] pullChanged.remoteListing 42ms keys=291
10:58:49 [s3-sync] pullChanged.walk 174ms toPull=0
10:58:49 [s3-sync] pullChanged.download 0ms pulled=0
10:58:54 [s3-sync] pullChanged.fastpath 2963ms miss
10:58:57 [s3-sync] pullChanged.pullIndex 3571ms
10:58:59 [s3-sync] pullChanged.remoteListing 1211ms keys=9547
10:59:00 [s3-sync] pullChanged.walk 1305ms toPull=0
10:59:00 [s3-sync] pullChanged.download 0ms pulled=0
10:59:00 [s3-sync] pullChanged.fastpath 18ms miss
10:59:00 [s3-sync] pullChanged.pullIndex 46ms
10:59:03 [s3-sync] pullChanged.remoteListing 2847ms keys=36255
10:59:07 [s3-sync] pullChanged.walk 4131ms toPull=1
10:59:07 [s3-sync] pullChanged.download 31ms pulled=1
10:59:09 [s3-sync] pullChanged.fastpath 16ms miss
10:59:10 [s3-sync] pullChanged.pullIndex 1107ms
10:59:11 [s3-sync] pullChanged.remoteListing 67ms keys=291
10:59:11 [s3-sync] pullChanged.walk 152ms toPull=0
10:59:11 [s3-sync] pullChanged.download 0ms pulled=0
10:59:11 [s3-sync] preparePush.fastpath 0ms miss
10:59:11 [s3-sync] preparePush.shardAssembly 219ms scoped=14
10:59:12 [s3-sync] preparePush.walk 849ms toPush=28 toDelete=0
10:59:12 [s3-sync] preparePush.upload 554ms pushed=22
10:59:12 [s3-sync] preparePush.delete 0ms deleted=0
10:59:13 [s3-sync] commitPush.shards 317ms dirty=9
10:59:13 [s3-sync] commitPush.localIndex 121ms
10:59:13 [s3-sync] commitPush 471ms shard-first changes=22
10:59:13 [s3-sync] pullChanged.fastpath 31ms miss
10:59:30 [s3-sync] pullChanged.pullIndex 16911ms
10:59:44 [s3-sync] pullChanged.remoteListing 13299ms keys=9561
10:59:50 [s3-sync] pullChanged.walk 6090ms toPull=0
10:59:50 [s3-sync] pullChanged.download 0ms pulled=0
10:59:50 [s3-sync] pullChanged.fastpath 71ms miss
11:00:02 [s3-sync] pullChanged.pullIndex 11603ms
11:00:02 [s3-sync] pullChanged.remoteListing 68ms keys=291
11:00:02 [s3-sync] pullChanged.walk 147ms toPull=0
11:00:02 [s3-sync] pullChanged.download 0ms pulled=0
11:00:15 [s3-sync] pullChanged.fastpath 2207ms miss
11:00:22 [s3-sync] pullChanged.pullIndex 6445ms
11:00:22 [s3-sync] pullChanged.remoteListing 78ms keys=291
11:00:22 [s3-sync] pullChanged.walk 168ms toPull=0
11:00:22 [s3-sync] pullChanged.download 0ms pulled=0
11:00:22 [s3-sync] pushChanged.fastpath 0ms miss
11:00:22 [s3-sync] pushChanged.shardAssembly 135ms scoped=3
11:00:22 [s3-sync] pushChanged.walk 69ms toPush=7 toDelete=0
11:00:23 [s3-sync] pushChanged.upload 64ms pushed=7
11:00:23 [s3-sync] pushChanged.delete 0ms deleted=0
11:00:23 [s3-sync] pushChanged.v2writeback 313ms
11:00:23 [s3-sync] pullChanged.fastpath 23ms miss
11:00:24 [s3-sync] pullChanged.pullIndex 1119ms
11:00:33 [s3-sync] pullChanged.remoteListing 8193ms keys=9565
11:00:39 [s3-sync] pullChanged.walk 6478ms toPull=0
11:00:39 [s3-sync] pullChanged.download 0ms pulled=0
11:00:44 [s3-sync] pullChanged.fastpath 36ms miss
11:00:53 [s3-sync] pullChanged.pullIndex 8829ms
11:00:54 [s3-sync] pullChanged.remoteListing 60ms keys=291
11:00:54 [s3-sync] pullChanged.walk 155ms toPull=0
11:00:54 [s3-sync] pullChanged.download 0ms pulled=0
11:00:54 [s3-sync] pullChanged.fastpath 31ms miss
11:00:54 [s3-sync] pullChanged.pullIndex 57ms
11:00:57 [s3-sync] pullChanged.remoteListing 2571ms keys=36277
11:01:01 [s3-sync] pullChanged.walk 4593ms toPull=1
11:01:01 [s3-sync] pullChanged.download 28ms pulled=1
11:01:02 [s3-sync] pullChanged.fastpath 23ms miss
11:01:04 [s3-sync] pullChanged.pullIndex 1633ms
11:01:05 [s3-sync] pullChanged.remoteListing 1100ms keys=9566
11:01:06 [s3-sync] pullChanged.walk 1215ms toPull=3
11:01:06 [s3-sync] pullChanged.download 31ms pulled=3
11:01:06 [s3-sync] preparePush.fastpath 0ms miss
11:01:07 [s3-sync] preparePush.shardAssembly 240ms scoped=22
11:01:07 [s3-sync] preparePush.walk 976ms toPush=41 toDelete=0
11:01:08 [s3-sync] preparePush.upload 266ms pushed=29
11:01:08 [s3-sync] preparePush.delete 0ms deleted=0
11:01:08 [s3-sync] commitPush.shards 355ms dirty=10
11:01:08 [s3-sync] commitPush.localIndex 130ms
11:01:08 [s3-sync] commitPush 518ms shard-first changes=29
11:01:14 [s3-sync] pullChanged.fastpath 22ms miss
11:01:23 [s3-sync] pullChanged.pullIndex 8479ms
11:01:23 [s3-sync] pullChanged.remoteListing 56ms keys=291
11:01:23 [s3-sync] pullChanged.walk 149ms toPull=0
11:01:23 [s3-sync] pullChanged.download 0ms pulled=0
11:01:23 [s3-sync] pullChanged.fastpath 18ms miss
11:01:23 [s3-sync] pullChanged.pullIndex 52ms
11:01:26 [s3-sync] pullChanged.remoteListing 2900ms keys=36303
11:01:31 [s3-sync] pullChanged.walk 4550ms toPull=1
11:01:31 [s3-sync] pullChanged.download 42ms pulled=1
11:01:31 [s3-sync] pullChanged.fastpath 23ms miss
11:01:33 [s3-sync] pullChanged.pullIndex 1168ms
11:01:34 [s3-sync] pullChanged.remoteListing 1377ms keys=9587
11:01:35 [s3-sync] pullChanged.walk 1265ms toPull=3
11:01:35 [s3-sync] pullChanged.download 34ms pulled=3
11:01:36 [s3-sync] preparePush.fastpath 0ms miss
11:01:36 [s3-sync] preparePush.shardAssembly 227ms scoped=14
11:01:36 [s3-sync] preparePush.walk 879ms toPush=27 toDelete=0
11:01:37 [s3-sync] preparePush.upload 149ms pushed=21
11:01:37 [s3-sync] preparePush.delete 0ms deleted=0
11:01:37 [s3-sync] commitPush.shards 311ms dirty=8
11:01:37 [s3-sync] commitPush.localIndex 124ms
11:01:37 [s3-sync] commitPush 460ms shard-first changes=21
11:01:49 [s3-sync] pullChanged.fastpath 3087ms miss
11:01:55 [s3-sync] pullChanged.pullIndex 6447ms
11:01:57 [s3-sync] pullChanged.remoteListing 130ms keys=291
11:02:03 [s3-sync] pullChanged.walk 5295ms toPull=0
11:02:03 [s3-sync] pullChanged.download 0ms pulled=0
11:02:03 [s3-sync] pullChanged.fastpath 30ms miss
11:02:15 [s3-sync] pullChanged.pullIndex 11537ms
11:02:22 [s3-sync] pullChanged.remoteListing 6628ms keys=9601
11:02:41 [s3-sync] pullChanged.walk 19009ms toPull=2
11:02:46 [s3-sync] pullChanged.download 5123ms pulled=2
11:02:46 [s3-sync] pushChanged.fastpath 0ms miss
11:02:47 [s3-sync] pushChanged.shardAssembly 377ms scoped=3
11:02:47 [s3-sync] pushChanged.walk 69ms toPush=5 toDelete=0
11:02:47 [s3-sync] pushChanged.upload 77ms pushed=5
11:02:47 [s3-sync] pushChanged.delete 0ms deleted=0
11:02:47 [s3-sync] pushChanged.v2writeback 291ms
11:02:48 [s3-sync] pullChanged.fastpath 22ms miss
11:02:49 [s3-sync] pullChanged.pullIndex 1238ms
11:02:49 [s3-sync] pullChanged.remoteListing 71ms keys=291
11:02:50 [s3-sync] pullChanged.walk 154ms toPull=0
11:02:50 [s3-sync] pullChanged.download 0ms pulled=0
11:03:17 [s3-sync] pullChanged.fastpath 36ms miss
11:03:18 [s3-sync] pullChanged.pullIndex 1184ms
11:03:18 [s3-sync] pullChanged.remoteListing 77ms keys=291
11:03:19 [s3-sync] pullChanged.walk 165ms toPull=0
11:03:19 [s3-sync] pullChanged.download 0ms pulled=0
11:03:19 [s3-sync] pullChanged.fastpath 19ms miss
11:03:19 [s3-sync] pullChanged.pullIndex 48ms
11:03:21 [s3-sync] pullChanged.remoteListing 2673ms keys=36325
11:03:26 [s3-sync] pullChanged.walk 4299ms toPull=1
11:03:26 [s3-sync] pullChanged.download 47ms pulled=1
11:03:26 [s3-sync] pullChanged.fastpath 32ms miss
11:03:27 [s3-sync] pullChanged.pullIndex 1185ms
11:03:29 [s3-sync] pullChanged.remoteListing 1351ms keys=9606
11:03:30 [s3-sync] pullChanged.walk 1302ms toPull=3
11:03:30 [s3-sync] pullChanged.download 47ms pulled=3
11:03:30 [s3-sync] preparePush.fastpath 0ms miss
11:03:31 [s3-sync] preparePush.shardAssembly 244ms scoped=22
11:03:31 [s3-sync] preparePush.walk 1016ms toPush=41 toDelete=0
11:03:32 [s3-sync] preparePush.upload 183ms pushed=29
11:03:32 [s3-sync] preparePush.delete 0ms deleted=0
11:03:32 [s3-sync] commitPush.shards 411ms dirty=10
11:03:32 [s3-sync] commitPush.localIndex 137ms
11:03:32 [s3-sync] commitPush 581ms shard-first changes=29
11:03:45 [s3-sync] pullChanged.fastpath 35ms miss
11:03:45 [s3-sync] pullChanged.pullIndex 51ms
11:03:50 [s3-sync] pullChanged.remoteListing 4544ms keys=36351
11:03:54 [s3-sync] pullChanged.walk 4278ms toPull=1
11:03:54 [s3-sync] pullChanged.download 38ms pulled=1
11:03:55 [s3-sync] pullChanged.fastpath 20ms miss
11:03:56 [s3-sync] pullChanged.pullIndex 1206ms
11:03:56 [s3-sync] pullChanged.remoteListing 69ms keys=291
11:03:57 [s3-sync] pullChanged.walk 147ms toPull=0
11:03:57 [s3-sync] pullChanged.download 0ms pulled=0
11:03:57 [s3-sync] preparePush.fastpath 0ms miss
11:03:57 [s3-sync] preparePush.shardAssembly 214ms scoped=14
11:03:58 [s3-sync] preparePush.walk 868ms toPush=28 toDelete=0
11:03:58 [s3-sync] preparePush.upload 148ms pushed=22
11:03:58 [s3-sync] preparePush.delete 0ms deleted=0
11:03:58 [s3-sync] commitPush.shards 324ms dirty=9
11:03:58 [s3-sync] commitPush.localIndex 124ms
11:03:58 [s3-sync] commitPush 477ms shard-first changes=22
11:04:04 [s3-sync] pullChanged.fastpath 2979ms miss
11:04:10 [s3-sync] pullChanged.pullIndex 6287ms
11:04:17 [s3-sync] pullChanged.remoteListing 6387ms keys=9641
11:04:31 [s3-sync] pullChanged.walk 13931ms toPull=0
11:04:31 [s3-sync] pullChanged.download 0ms pulled=0
11:04:31 [s3-sync] pullChanged.fastpath 33ms miss
11:04:45 [s3-sync] pullChanged.pullIndex 13936ms
11:04:48 [s3-sync] pullChanged.remoteListing 76ms keys=291
11:04:48 [s3-sync] pullChanged.walk 146ms toPull=0
11:04:48 [s3-sync] pullChanged.download 0ms pulled=0
11:04:49 [s3-sync] pullChanged.fastpath 27ms miss
11:05:03 [s3-sync] pullChanged.pullIndex 14164ms
11:05:10 [s3-sync] pullChanged.remoteListing 6261ms keys=9641
11:05:11 [s3-sync] pullChanged.walk 1231ms toPull=2
11:05:11 [s3-sync] pullChanged.download 40ms pulled=2
11:05:11 [s3-sync] pushChanged.fastpath 0ms miss
11:05:11 [s3-sync] pushChanged.shardAssembly 141ms scoped=3
11:05:12 [s3-sync] pushChanged.walk 58ms toPush=5 toDelete=0
11:05:12 [s3-sync] pushChanged.upload 78ms pushed=5
11:05:12 [s3-sync] pushChanged.delete 0ms deleted=0
11:05:12 [s3-sync] pushChanged.v2writeback 285ms
11:05:22 [s3-sync] pullChanged.fastpath 3079ms miss
11:05:30 [s3-sync] pullChanged.pullIndex 8271ms
11:05:36 [s3-sync] pullChanged.remoteListing 82ms keys=291
11:05:36 [s3-sync] pullChanged.walk 269ms toPull=0
11:05:36 [s3-sync] pullChanged.download 0ms pulled=0
11:05:41 [s3-sync] pullChanged.fastpath 3140ms miss
11:05:42 [s3-sync] pullChanged.pullIndex 1233ms
11:05:44 [s3-sync] pullChanged.remoteListing 1247ms keys=9645
11:05:45 [s3-sync] pullChanged.walk 1451ms toPull=0
11:05:45 [s3-sync] pullChanged.download 0ms pulled=0
11:05:45 [s3-sync] pullChanged.fastpath 29ms miss
11:05:45 [s3-sync] pullChanged.pullIndex 50ms
11:05:48 [s3-sync] pullChanged.remoteListing 2481ms keys=36373
11:05:52 [s3-sync] pullChanged.walk 4301ms toPull=1
11:05:52 [s3-sync] pullChanged.download 40ms pulled=1
11:05:53 [s3-sync] pullChanged.fastpath 23ms miss
11:05:54 [s3-sync] pullChanged.pullIndex 1227ms
11:05:54 [s3-sync] pullChanged.remoteListing 49ms keys=291
11:05:54 [s3-sync] pullChanged.walk 160ms toPull=0
11:05:54 [s3-sync] pullChanged.download 0ms pulled=0
11:05:54 [s3-sync] preparePush.fastpath 0ms miss
11:05:55 [s3-sync] preparePush.shardAssembly 239ms scoped=22
11:05:55 [s3-sync] preparePush.walk 979ms toPush=42 toDelete=0
11:05:57 [s3-sync] preparePush.upload 1246ms pushed=30
11:05:57 [s3-sync] preparePush.delete 0ms deleted=0
11:05:57 [s3-sync] commitPush.shards 328ms dirty=11
11:05:57 [s3-sync] commitPush.localIndex 125ms
11:05:57 [s3-sync] commitPush 489ms shard-first changes=30
11:06:03 [s3-sync] pullChanged.fastpath 5175ms miss
11:06:12 [s3-sync] pullChanged.pullIndex 8650ms
11:06:13 [s3-sync] pullChanged.remoteListing 1324ms keys=9666
11:06:14 [s3-sync] pullChanged.walk 1222ms toPull=0
11:06:14 [s3-sync] pullChanged.download 0ms pulled=0
11:06:14 [s3-sync] pullChanged.fastpath 26ms miss
11:06:14 [s3-sync] pullChanged.pullIndex 49ms
11:06:17 [s3-sync] pullChanged.remoteListing 2871ms keys=36399
11:06:21 [s3-sync] pullChanged.walk 4249ms toPull=1
11:06:22 [s3-sync] pullChanged.download 41ms pulled=1
11:06:22 [s3-sync] pullChanged.fastpath 22ms miss
11:06:23 [s3-sync] pullChanged.pullIndex 1094ms
11:06:23 [s3-sync] pullChanged.remoteListing 72ms keys=291
11:06:23 [s3-sync] pullChanged.walk 139ms toPull=0
11:06:23 [s3-sync] pullChanged.download 0ms pulled=0
11:06:24 [s3-sync] pullChanged.fastpath 25ms miss
11:06:25 [s3-sync] pullChanged.pullIndex 1052ms
11:06:26 [s3-sync] pullChanged.remoteListing 1245ms keys=9667
11:06:27 [s3-sync] pullChanged.walk 1235ms toPull=3
11:06:27 [s3-sync] pullChanged.download 30ms pulled=3
11:06:28 [s3-sync] preparePush.fastpath 0ms miss
11:06:28 [s3-sync] preparePush.shardAssembly 232ms scoped=14
11:06:28 [s3-sync] preparePush.walk 890ms toPush=27 toDelete=0
11:06:29 [s3-sync] preparePush.upload 744ms pushed=21
11:06:29 [s3-sync] preparePush.delete 0ms deleted=0
11:06:30 [s3-sync] commitPush.shards 328ms dirty=8
11:06:30 [s3-sync] commitPush.localIndex 126ms
11:06:30 [s3-sync] commitPush 483ms shard-first changes=21
11:06:50 [s3-sync] pullChanged.fastpath 38ms miss
11:06:51 [s3-sync] pullChanged.pullIndex 1250ms
11:06:56 [s3-sync] pullChanged.remoteListing 82ms keys=291
11:06:57 [s3-sync] pullChanged.walk 153ms toPull=0
11:06:57 [s3-sync] pullChanged.download 0ms pulled=0
11:07:02 [s3-sync] pullChanged.fastpath 3027ms miss
11:07:09 [s3-sync] pullChanged.pullIndex 6530ms
11:07:20 [s3-sync] pullChanged.remoteListing 8621ms keys=9681
11:07:34 [s3-sync] pullChanged.walk 14252ms toPull=0
11:07:34 [s3-sync] pullChanged.download 0ms pulled=0
11:07:35 [s3-sync] pullChanged.fastpath 68ms miss
11:07:41 [s3-sync] pullChanged.pullIndex 6149ms
11:07:41 [s3-sync] pullChanged.remoteListing 72ms keys=291
11:07:41 [s3-sync] pullChanged.walk 205ms toPull=0
11:07:41 [s3-sync] pullChanged.download 0ms pulled=0
11:07:41 [s3-sync] pushChanged.fastpath 0ms miss
11:07:44 [s3-sync] pushChanged.shardAssembly 2179ms scoped=3
11:07:44 [s3-sync] pushChanged.walk 69ms toPush=7 toDelete=0
11:07:44 [s3-sync] pushChanged.upload 64ms pushed=7
11:07:44 [s3-sync] pushChanged.delete 0ms deleted=0
11:07:44 [s3-sync] pushChanged.v2writeback 299ms
11:07:54 [s3-sync] pullChanged.fastpath 43ms miss
11:07:55 [s3-sync] pullChanged.pullIndex 1124ms
11:07:55 [s3-sync] pullChanged.remoteListing 65ms keys=291
11:07:56 [s3-sync] pullChanged.walk 158ms toPull=0
11:07:56 [s3-sync] pullChanged.download 0ms pulled=0
11:08:01 [s3-sync] pullChanged.fastpath 2914ms miss
11:08:07 [s3-sync] pullChanged.pullIndex 6419ms
11:08:16 [s3-sync] pullChanged.remoteListing 8491ms keys=9685
11:08:17 [s3-sync] pullChanged.walk 1334ms toPull=0
11:08:17 [s3-sync] pullChanged.download 0ms pulled=0
11:08:17 [s3-sync] pullChanged.fastpath 36ms miss
11:08:17 [s3-sync] pullChanged.pullIndex 48ms
11:08:20 [s3-sync] pullChanged.remoteListing 3032ms keys=36421
11:08:25 [s3-sync] pullChanged.walk 4315ms toPull=1
11:08:25 [s3-sync] pullChanged.download 40ms pulled=1
11:08:25 [s3-sync] pullChanged.fastpath 18ms miss
11:08:26 [s3-sync] pullChanged.pullIndex 1279ms
11:08:27 [s3-sync] pullChanged.remoteListing 63ms keys=291
11:08:27 [s3-sync] pullChanged.walk 176ms toPull=0
11:08:27 [s3-sync] pullChanged.download 0ms pulled=0
11:08:27 [s3-sync] pullChanged.fastpath 29ms miss
11:08:28 [s3-sync] pullChanged.pullIndex 1122ms
11:08:30 [s3-sync] pullChanged.remoteListing 1400ms keys=9686
11:08:31 [s3-sync] pullChanged.walk 1264ms toPull=3
11:08:31 [s3-sync] pullChanged.download 33ms pulled=3
11:08:31 [s3-sync] preparePush.fastpath 0ms miss
11:08:31 [s3-sync] preparePush.shardAssembly 222ms scoped=22
11:08:32 [s3-sync] preparePush.walk 976ms toPush=41 toDelete=0
11:08:32 [s3-sync] preparePush.upload 173ms pushed=29
11:08:32 [s3-sync] preparePush.delete 0ms deleted=0
11:08:33 [s3-sync] commitPush.shards 373ms dirty=10
11:08:33 [s3-sync] commitPush.localIndex 140ms
11:08:33 [s3-sync] commitPush 538ms shard-first changes=29
11:08:45 [s3-sync] pullChanged.fastpath 83ms miss
11:08:45 [s3-sync] pullChanged.pullIndex 56ms
11:08:51 [s3-sync] pullChanged.remoteListing 5532ms keys=36447
11:08:55 [s3-sync] pullChanged.walk 4418ms toPull=1
11:08:55 [s3-sync] pullChanged.download 32ms pulled=1
11:08:56 [s3-sync] pullChanged.fastpath 19ms miss
11:08:57 [s3-sync] pullChanged.pullIndex 1137ms
11:08:57 [s3-sync] pullChanged.remoteListing 41ms keys=291
11:08:58 [s3-sync] pullChanged.walk 162ms toPull=0
11:08:58 [s3-sync] pullChanged.download 0ms pulled=0
11:08:58 [s3-sync] preparePush.fastpath 0ms miss
11:08:58 [s3-sync] preparePush.shardAssembly 227ms scoped=14
11:08:59 [s3-sync] preparePush.walk 863ms toPush=28 toDelete=0
11:08:59 [s3-sync] preparePush.upload 147ms pushed=22
11:08:59 [s3-sync] preparePush.delete 0ms deleted=0
11:08:59 [s3-sync] commitPush.shards 329ms dirty=9
11:08:59 [s3-sync] commitPush.localIndex 134ms
11:08:59 [s3-sync] commitPush 500ms shard-first changes=22
11:09:05 [s3-sync] pullChanged.fastpath 3009ms miss
11:09:06 [s3-sync] pullChanged.pullIndex 1155ms
11:09:18 [s3-sync] pullChanged.remoteListing 11581ms keys=9721
11:09:27 [s3-sync] pullChanged.walk 8804ms toPull=0
11:09:27 [s3-sync] pullChanged.download 0ms pulled=0
11:09:27 [s3-sync] pullChanged.fastpath 35ms miss
11:09:39 [s3-sync] pullChanged.pullIndex 11944ms
11:09:40 [s3-sync] pullChanged.remoteListing 80ms keys=291
11:09:40 [s3-sync] pullChanged.walk 150ms toPull=0
11:09:40 [s3-sync] pullChanged.download 0ms pulled=0
11:09:58 [s3-sync] pullChanged.fastpath 2984ms miss
11:10:12 [s3-sync] pullChanged.pullIndex 14506ms
11:10:12 [s3-sync] pullChanged.remoteListing 55ms keys=291
11:10:13 [s3-sync] pullChanged.walk 154ms toPull=0
11:10:13 [s3-sync] pullChanged.download 0ms pulled=0
11:10:13 [s3-sync] pushChanged.fastpath 0ms miss
11:10:13 [s3-sync] pushChanged.shardAssembly 138ms scoped=3
11:10:13 [s3-sync] pushChanged.walk 344ms toPush=7 toDelete=0
11:10:13 [s3-sync] pushChanged.upload 61ms pushed=7
11:10:13 [s3-sync] pushChanged.delete 0ms deleted=0
11:10:14 [s3-sync] pushChanged.v2writeback 295ms
11:10:14 [s3-sync] pullChanged.fastpath 22ms miss
11:10:15 [s3-sync] pullChanged.pullIndex 1220ms
11:10:24 [s3-sync] pullChanged.remoteListing 8392ms keys=9725
11:10:35 [s3-sync] pullChanged.walk 11767ms toPull=0
11:10:35 [s3-sync] pullChanged.download 0ms pulled=0
11:10:38 [s3-sync] pullChanged.fastpath 169ms miss
11:10:44 [s3-sync] pullChanged.pullIndex 6483ms
11:10:44 [s3-sync] pullChanged.remoteListing 64ms keys=291
11:10:45 [s3-sync] pullChanged.walk 236ms toPull=0
11:10:45 [s3-sync] pullChanged.download 0ms pulled=0
11:10:45 [s3-sync] pullChanged.fastpath 21ms miss
11:10:45 [s3-sync] pullChanged.pullIndex 53ms
11:10:48 [s3-sync] pullChanged.remoteListing 2863ms keys=36469
11:10:52 [s3-sync] pullChanged.walk 4324ms toPull=1
11:10:52 [s3-sync] pullChanged.download 47ms pulled=1
11:10:54 [s3-sync] preparePush.fastpath 0ms miss
11:10:55 [s3-sync] preparePush.shardAssembly 273ms scoped=22
11:10:55 [s3-sync] preparePush.walk 1050ms toPush=42 toDelete=0
11:10:55 [s3-sync] preparePush.upload 173ms pushed=30
11:10:55 [s3-sync] preparePush.delete 0ms deleted=0
11:10:56 [s3-sync] commitPush.shards 327ms dirty=11
11:10:56 [s3-sync] commitPush.localIndex 140ms
11:10:56 [s3-sync] commitPush 500ms shard-first changes=30
11:10:57 [s3-sync] pullChanged.fastpath 21ms miss
11:11:03 [s3-sync] pullChanged.pullIndex 6303ms
11:11:03 [s3-sync] pullChanged.remoteListing 56ms keys=291
11:11:03 [s3-sync] pullChanged.walk 139ms toPull=0
11:11:03 [s3-sync] pullChanged.download 0ms pulled=0
11:11:08 [s3-sync] pullChanged.fastpath 2902ms miss
11:11:12 [s3-sync] pullChanged.pullIndex 3416ms
11:11:13 [s3-sync] pullChanged.remoteListing 1296ms keys=9747
11:11:15 [s3-sync] pullChanged.walk 1298ms toPull=0
11:11:15 [s3-sync] pullChanged.download 0ms pulled=0
11:11:15 [s3-sync] pullChanged.fastpath 27ms miss
11:11:15 [s3-sync] pullChanged.pullIndex 47ms
11:11:17 [s3-sync] pullChanged.remoteListing 2613ms keys=36495
11:11:22 [s3-sync] pullChanged.walk 4450ms toPull=1
11:11:22 [s3-sync] pullChanged.download 36ms pulled=1
11:11:24 [s3-sync] preparePush.fastpath 0ms miss
11:11:24 [s3-sync] preparePush.shardAssembly 256ms scoped=14
11:11:25 [s3-sync] preparePush.walk 912ms toPush=28 toDelete=0
11:11:25 [s3-sync] preparePush.upload 143ms pushed=22
11:11:25 [s3-sync] preparePush.delete 0ms deleted=0
11:11:25 [s3-sync] commitPush.shards 341ms dirty=9
11:11:25 [s3-sync] commitPush.localIndex 126ms
11:11:25 [s3-sync] commitPush 501ms shard-first changes=22
11:11:31 [s3-sync] pullChanged.fastpath 3026ms miss
11:11:37 [s3-sync] pullChanged.pullIndex 6289ms
11:11:38 [s3-sync] pullChanged.remoteListing 72ms keys=291
11:11:38 [s3-sync] pullChanged.walk 156ms toPull=0
11:11:38 [s3-sync] pullChanged.download 0ms pulled=0
11:11:43 [s3-sync] pullChanged.fastpath 3207ms miss
11:11:57 [s3-sync] pullChanged.pullIndex 13591ms
11:12:03 [s3-sync] pullChanged.remoteListing 6406ms keys=9761
11:12:10 [s3-sync] pullChanged.walk 6473ms toPull=0
11:12:10 [s3-sync] pullChanged.download 0ms pulled=0
11:12:10 [s3-sync] pullChanged.fastpath 35ms miss
11:12:19 [s3-sync] pullChanged.pullIndex 8518ms
11:12:19 [s3-sync] pullChanged.remoteListing 71ms keys=291
11:12:19 [s3-sync] pullChanged.walk 160ms toPull=0
11:12:19 [s3-sync] pullChanged.download 0ms pulled=0
11:12:30 [s3-sync] pullChanged.fastpath 34ms miss
11:12:37 [s3-sync] pullChanged.pullIndex 6552ms
11:12:37 [s3-sync] pullChanged.remoteListing 76ms keys=291
11:12:37 [s3-sync] pullChanged.walk 157ms toPull=0
11:12:37 [s3-sync] pullChanged.download 0ms pulled=0
11:12:37 [s3-sync] pushChanged.fastpath 0ms miss
11:12:37 [s3-sync] pushChanged.shardAssembly 133ms scoped=3
11:12:38 [s3-sync] pushChanged.walk 63ms toPush=7 toDelete=0
11:12:38 [s3-sync] pushChanged.upload 71ms pushed=7
11:12:38 [s3-sync] pushChanged.delete 0ms deleted=0
11:12:38 [s3-sync] pushChanged.v2writeback 557ms
11:12:39 [s3-sync] pullChanged.fastpath 24ms miss
11:12:40 [s3-sync] pullChanged.pullIndex 1144ms
11:12:50 [s3-sync] pullChanged.remoteListing 10432ms keys=9765
11:13:02 [s3-sync] pullChanged.walk 11996ms toPull=0
11:13:02 [s3-sync] pullChanged.download 0ms pulled=0
11:13:08 [s3-sync] pullChanged.fastpath 4978ms miss
11:13:10 [s3-sync] pullChanged.pullIndex 1174ms
11:13:15 [s3-sync] pullChanged.remoteListing 79ms keys=291
11:13:15 [s3-sync] pullChanged.walk 152ms toPull=0
11:13:15 [s3-sync] pullChanged.download 0ms pulled=0
11:13:16 [s3-sync] pullChanged.fastpath 33ms miss
11:13:16 [s3-sync] pullChanged.pullIndex 55ms
11:13:19 [s3-sync] pullChanged.remoteListing 2897ms keys=36517
11:13:23 [s3-sync] pullChanged.walk 4316ms toPull=1
11:13:23 [s3-sync] pullChanged.download 32ms pulled=1
11:13:25 [s3-sync] preparePush.fastpath 0ms miss
11:13:25 [s3-sync] preparePush.shardAssembly 255ms scoped=22
11:13:26 [s3-sync] preparePush.walk 1014ms toPush=42 toDelete=0
11:13:26 [s3-sync] preparePush.upload 181ms pushed=30
11:13:26 [s3-sync] preparePush.delete 0ms deleted=0
11:13:27 [s3-sync] commitPush.shards 347ms dirty=11
11:13:27 [s3-sync] commitPush.localIndex 139ms
11:13:27 [s3-sync] commitPush 513ms shard-first changes=30
11:13:40 [s3-sync] pullChanged.fastpath 5551ms miss
11:13:41 [s3-sync] pullChanged.pullIndex 1166ms
11:13:42 [s3-sync] pullChanged.remoteListing 69ms keys=291
11:13:42 [s3-sync] pullChanged.walk 157ms toPull=0
11:13:42 [s3-sync] pullChanged.download 0ms pulled=0
11:13:42 [s3-sync] pullChanged.fastpath 25ms miss
11:13:42 [s3-sync] pullChanged.pullIndex 55ms
11:13:45 [s3-sync] pullChanged.remoteListing 3069ms keys=36543
11:13:49 [s3-sync] pullChanged.walk 4271ms toPull=1
11:13:49 [s3-sync] pullChanged.download 46ms pulled=1
11:13:50 [s3-sync] pullChanged.fastpath 28ms miss
11:13:52 [s3-sync] pullChanged.pullIndex 1169ms
11:13:53 [s3-sync] pullChanged.remoteListing 1190ms keys=9787
11:13:54 [s3-sync] pullChanged.walk 1249ms toPull=3
11:13:54 [s3-sync] pullChanged.download 31ms pulled=3
11:13:54 [s3-sync] preparePush.fastpath 0ms miss
11:13:54 [s3-sync] preparePush.shardAssembly 243ms scoped=14
11:13:55 [s3-sync] preparePush.walk 879ms toPush=27 toDelete=0
11:13:55 [s3-sync] preparePush.upload 141ms pushed=21
11:13:55 [s3-sync] preparePush.delete 0ms deleted=0
11:13:56 [s3-sync] commitPush.shards 377ms dirty=8
11:13:56 [s3-sync] commitPush.localIndex 123ms
11:13:56 [s3-sync] commitPush 529ms shard-first changes=21
11:14:10 [s3-sync] pullChanged.fastpath 4988ms miss
11:14:16 [s3-sync] pullChanged.pullIndex 6482ms
11:14:17 [s3-sync] pullChanged.remoteListing 64ms keys=291
11:14:17 [s3-sync] pullChanged.walk 242ms toPull=0
11:14:17 [s3-sync] pullChanged.download 0ms pulled=0
11:14:22 [s3-sync] pullChanged.fastpath 2970ms miss
11:14:29 [s3-sync] pullChanged.pullIndex 6460ms
11:14:37 [s3-sync] pullChanged.remoteListing 8539ms keys=9801
11:14:49 [s3-sync] pullChanged.walk 11422ms toPull=0
11:14:49 [s3-sync] pullChanged.download 0ms pulled=0
11:14:49 [s3-sync] pullChanged.fastpath 39ms miss
11:14:56 [s3-sync] pullChanged.pullIndex 6251ms
11:14:56 [s3-sync] pullChanged.remoteListing 64ms keys=291
11:14:56 [s3-sync] pullChanged.walk 142ms toPull=0
11:14:56 [s3-sync] pullChanged.download 0ms pulled=0
11:15:07 [s3-sync] pullChanged.fastpath 55ms miss
11:15:11 [s3-sync] pullChanged.pullIndex 3520ms
11:15:11 [s3-sync] pullChanged.remoteListing 74ms keys=291
11:15:11 [s3-sync] pullChanged.walk 151ms toPull=0
11:15:11 [s3-sync] pullChanged.download 0ms pulled=0
11:15:11 [s3-sync] pushChanged.fastpath 0ms miss
11:15:11 [s3-sync] pushChanged.shardAssembly 132ms scoped=3
11:15:11 [s3-sync] pushChanged.walk 73ms toPush=7 toDelete=0
11:15:11 [s3-sync] pushChanged.upload 81ms pushed=7
11:15:11 [s3-sync] pushChanged.delete 0ms deleted=0
11:15:12 [s3-sync] pushChanged.v2writeback 298ms
11:15:12 [s3-sync] pullChanged.fastpath 24ms miss
11:15:13 [s3-sync] pullChanged.pullIndex 1158ms
11:15:27 [s3-sync] pullChanged.remoteListing 13325ms keys=9805
11:15:33 [s3-sync] pullChanged.walk 6299ms toPull=0
11:15:33 [s3-sync] pullChanged.download 0ms pulled=0
11:15:39 [s3-sync] pullChanged.fastpath 41ms miss
11:15:43 [s3-sync] pullChanged.pullIndex 3798ms
11:15:43 [s3-sync] pullChanged.remoteListing 56ms keys=291
11:15:43 [s3-sync] pullChanged.walk 149ms toPull=0
11:15:43 [s3-sync] pullChanged.download 0ms pulled=0
11:15:43 [s3-sync] pullChanged.fastpath 18ms miss
11:15:43 [s3-sync] pullChanged.pullIndex 55ms
11:15:46 [s3-sync] pullChanged.remoteListing 2486ms keys=36565
11:15:50 [s3-sync] pullChanged.walk 4283ms toPull=1
11:15:50 [s3-sync] pullChanged.download 38ms pulled=1
11:15:51 [s3-sync] pullChanged.fastpath 19ms miss
11:15:52 [s3-sync] pullChanged.pullIndex 1166ms
11:15:54 [s3-sync] pullChanged.remoteListing 1359ms keys=9806
11:15:55 [s3-sync] pullChanged.walk 1249ms toPull=3
11:15:55 [s3-sync] pullChanged.download 33ms pulled=3
11:15:55 [s3-sync] preparePush.fastpath 0ms miss
11:15:55 [s3-sync] preparePush.shardAssembly 226ms scoped=22
11:15:56 [s3-sync] preparePush.walk 996ms toPush=41 toDelete=0
11:15:56 [s3-sync] preparePush.upload 180ms pushed=29
11:15:56 [s3-sync] preparePush.delete 0ms deleted=0
11:15:57 [s3-sync] commitPush.shards 330ms dirty=10
11:15:57 [s3-sync] commitPush.localIndex 126ms
11:15:57 [s3-sync] commitPush 480ms shard-first changes=29
11:16:09 [s3-sync] pullChanged.fastpath 49ms miss
11:16:10 [s3-sync] pullChanged.pullIndex 1117ms
11:16:10 [s3-sync] pullChanged.remoteListing 64ms keys=291
11:16:11 [s3-sync] pullChanged.walk 154ms toPull=0
11:16:11 [s3-sync] pullChanged.download 0ms pulled=0
11:16:11 [s3-sync] pullChanged.fastpath 22ms miss
11:16:11 [s3-sync] pullChanged.pullIndex 55ms
11:16:16 [s3-sync] pullChanged.remoteListing 5040ms keys=36591
11:16:20 [s3-sync] pullChanged.walk 4306ms toPull=1
11:16:20 [s3-sync] pullChanged.download 42ms pulled=1
11:16:21 [s3-sync] pullChanged.fastpath 26ms miss
11:16:22 [s3-sync] pullChanged.pullIndex 1168ms
11:16:24 [s3-sync] pullChanged.remoteListing 1381ms keys=9827
11:16:25 [s3-sync] pullChanged.walk 1311ms toPull=3
11:16:25 [s3-sync] pullChanged.download 32ms pulled=3
11:16:25 [s3-sync] preparePush.fastpath 0ms miss
11:16:25 [s3-sync] preparePush.shardAssembly 212ms scoped=14
11:16:26 [s3-sync] preparePush.walk 879ms toPush=27 toDelete=0
11:16:26 [s3-sync] preparePush.upload 140ms pushed=21
11:16:26 [s3-sync] preparePush.delete 0ms deleted=0
11:16:27 [s3-sync] commitPush.shards 326ms dirty=8
11:16:27 [s3-sync] commitPush.localIndex 126ms
11:16:27 [s3-sync] commitPush 482ms shard-first changes=21
11:16:39 [s3-sync] pullChanged.fastpath 37ms miss
11:16:48 [s3-sync] pullChanged.pullIndex 8735ms
11:16:53 [s3-sync] pullChanged.remoteListing 81ms keys=291
11:16:53 [s3-sync] pullChanged.walk 152ms toPull=0
11:16:53 [s3-sync] pullChanged.download 0ms pulled=0
11:16:54 [s3-sync] pullChanged.fastpath 23ms miss
11:17:01 [s3-sync] pullChanged.pullIndex 6650ms
11:17:13 [s3-sync] pullChanged.remoteListing 6177ms keys=9841
11:17:26 [s3-sync] pullChanged.walk 13517ms toPull=0
11:17:26 [s3-sync] pullChanged.download 0ms pulled=0
11:17:32 [s3-sync] pullChanged.fastpath 2931ms miss
11:17:38 [s3-sync] pullChanged.pullIndex 6711ms
11:17:39 [s3-sync] pullChanged.remoteListing 49ms keys=291
11:17:39 [s3-sync] pullChanged.walk 155ms toPull=0
11:17:39 [s3-sync] pullChanged.download 0ms pulled=0
11:17:39 [s3-sync] pushChanged.fastpath 0ms miss
11:17:39 [s3-sync] pushChanged.shardAssembly 128ms scoped=3
11:17:39 [s3-sync] pushChanged.walk 64ms toPush=7 toDelete=0
11:17:39 [s3-sync] pushChanged.upload 68ms pushed=7
11:17:39 [s3-sync] pushChanged.delete 0ms deleted=0
11:17:39 [s3-sync] pushChanged.v2writeback 295ms
11:17:40 [s3-sync] pullChanged.fastpath 29ms miss
11:17:41 [s3-sync] pullChanged.pullIndex 1132ms
11:17:41 [s3-sync] pullChanged.remoteListing 57ms keys=291
11:17:42 [s3-sync] pullChanged.walk 235ms toPull=0
11:17:42 [s3-sync] pullChanged.download 0ms pulled=0
11:17:44 [s3-sync] pullChanged.fastpath 18ms miss
11:17:55 [s3-sync] pullChanged.pullIndex 11557ms
11🔞09 [s3-sync] pullChanged.remoteListing 11475ms keys=9845
11🔞10 [s3-sync] pullChanged.walk 1178ms toPull=0
11🔞10 [s3-sync] pullChanged.download 0ms pulled=0
11🔞10 [s3-sync] pullChanged.fastpath 32ms miss
11🔞10 [s3-sync] pullChanged.pullIndex 51ms
11🔞13 [s3-sync] pullChanged.remoteListing 2877ms keys=36613
11🔞17 [s3-sync] pullChanged.walk 4273ms toPull=1
11🔞17 [s3-sync] pullChanged.download 57ms pulled=1
11🔞18 [s3-sync] pullChanged.fastpath 27ms miss
11🔞19 [s3-sync] pullChanged.pullIndex 1199ms
11🔞20 [s3-sync] pullChanged.remoteListing 54ms keys=291
11🔞20 [s3-sync] pullChanged.walk 228ms toPull=0
11🔞20 [s3-sync] pullChanged.download 0ms pulled=0
11🔞20 [s3-sync] preparePush.fastpath 0ms miss
11🔞20 [s3-sync] preparePush.shardAssembly 248ms scoped=22
11🔞21 [s3-sync] preparePush.walk 1032ms toPush=42 toDelete=0
11🔞21 [s3-sync] preparePush.upload 180ms pushed=30
11🔞21 [s3-sync] preparePush.delete 0ms deleted=0
11🔞22 [s3-sync] commitPush.shards 368ms dirty=11
11🔞22 [s3-sync] commitPush.localIndex 128ms
11🔞22 [s3-sync] commitPush 520ms shard-first changes=30
11🔞37 [s3-sync] pullChanged.fastpath 31ms miss
11🔞37 [s3-sync] pullChanged.pullIndex 50ms
11🔞40 [s3-sync] pullChanged.remoteListing 2631ms keys=36639
11🔞44 [s3-sync] pullChanged.walk 4550ms toPull=1
11🔞44 [s3-sync] pullChanged.download 32ms pulled=1
11🔞45 [s3-sync] pullChanged.fastpath 24ms missvcjdeboer commented 10/31/2026, 12:41:06 PM
Full [s3-sync] / Handler slow / Handler error / heap-limit log of the traced sitting, part 4 of 4 (11🔞46-11:52:32 UTC):
11🔞46 [s3-sync] pullChanged.pullIndex 1248ms
11🔞47 [s3-sync] pullChanged.remoteListing 60ms keys=291
11🔞47 [s3-sync] pullChanged.walk 202ms toPull=0
11🔞47 [s3-sync] pullChanged.download 0ms pulled=0
11🔞47 [s3-sync] preparePush.fastpath 0ms miss
11🔞47 [s3-sync] preparePush.shardAssembly 205ms scoped=14
11🔞48 [s3-sync] preparePush.walk 876ms toPush=28 toDelete=0
11🔞48 [s3-sync] preparePush.upload 145ms pushed=22
11🔞48 [s3-sync] preparePush.delete 0ms deleted=0
11🔞48 [s3-sync] commitPush.shards 363ms dirty=9
11🔞48 [s3-sync] commitPush.localIndex 123ms
11🔞48 [s3-sync] commitPush 515ms shard-first changes=22
11🔞54 [s3-sync] pullChanged.fastpath 2980ms miss
11:19:03 [s3-sync] pullChanged.pullIndex 8719ms
11:19:09 [s3-sync] pullChanged.remoteListing 6379ms keys=9881
11:19:16 [s3-sync] pullChanged.walk 6392ms toPull=0
11:19:16 [s3-sync] pullChanged.download 0ms pulled=0
11:19:21 [s3-sync] pullChanged.fastpath 3010ms miss
11:19:28 [s3-sync] pullChanged.pullIndex 6413ms
11:19:28 [s3-sync] pullChanged.remoteListing 72ms keys=291
11:19:33 [s3-sync] pullChanged.walk 5474ms toPull=0
11:19:33 [s3-sync] pullChanged.download 0ms pulled=0
11:19:41 [s3-sync] pullChanged.fastpath 38ms miss
11:19:47 [s3-sync] pullChanged.pullIndex 6491ms
11:19:48 [s3-sync] pullChanged.remoteListing 61ms keys=291
11:19:54 [s3-sync] pullChanged.walk 6529ms toPull=0
11:19:54 [s3-sync] pullChanged.download 0ms pulled=0
11:19:55 [s3-sync] pullChanged.fastpath 300ms miss
11:20:02 [s3-sync] pullChanged.pullIndex 6837ms
11:20:04 [s3-sync] pullChanged.remoteListing 1174ms keys=9881
11:20:05 [s3-sync] pullChanged.walk 1311ms toPull=2
11:20:05 [s3-sync] pullChanged.download 32ms pulled=2
11:20:05 [s3-sync] pushChanged.fastpath 0ms miss
11:20:05 [s3-sync] pushChanged.shardAssembly 140ms scoped=3
11:20:05 [s3-sync] pushChanged.walk 76ms toPush=5 toDelete=0
11:20:05 [s3-sync] pushChanged.upload 60ms pushed=5
11:20:05 [s3-sync] pushChanged.delete 0ms deleted=0
11:20:06 [s3-sync] pushChanged.v2writeback 309ms
11:20:11 [s3-sync] pullChanged.fastpath 31ms miss
11:20:12 [s3-sync] pullChanged.pullIndex 1222ms
11:20:12 [s3-sync] pullChanged.remoteListing 69ms keys=291
11:20:13 [s3-sync] pullChanged.walk 216ms toPull=0
11:20:13 [s3-sync] pullChanged.download 0ms pulled=0
11:20:13 [s3-sync] pullChanged.fastpath 23ms miss
11:20:14 [s3-sync] pullChanged.pullIndex 1145ms
11:20:15 [s3-sync] pullChanged.remoteListing 1128ms keys=9885
11:20:18 [s3-sync] pullChanged.walk 2526ms toPull=0
11:20:18 [s3-sync] pullChanged.download 0ms pulled=0
11:20:43 [s3-sync] pullChanged.fastpath 47ms miss
11:20:52 [s3-sync] pullChanged.pullIndex 8785ms
11:20:52 [s3-sync] pullChanged.remoteListing 72ms keys=291
11:20:53 [s3-sync] pullChanged.walk 178ms toPull=0
11:20:53 [s3-sync] pullChanged.download 0ms pulled=0
11:20:53 [s3-sync] pullChanged.fastpath 25ms miss
11:20:53 [s3-sync] pullChanged.pullIndex 56ms
11:20:55 [s3-sync] pullChanged.remoteListing 2617ms keys=36661
11:20:59 [s3-sync] pullChanged.walk 3700ms toPull=1
11:20:59 [s3-sync] pullChanged.download 28ms pulled=1
11:20:59 [s3-sync] pullChanged.fastpath 20ms miss
11:21:01 [s3-sync] pullChanged.pullIndex 1227ms
11:21:02 [s3-sync] pullChanged.remoteListing 1304ms keys=9886
11:21:04 [s3-sync] pullChanged.walk 1776ms toPull=3
11:21:04 [s3-sync] pullChanged.download 31ms pulled=3
11:21:04 [s3-sync] preparePush.fastpath 0ms miss
11:21:04 [s3-sync] preparePush.shardAssembly 214ms scoped=22
11:21:05 [s3-sync] preparePush.walk 1169ms toPush=41 toDelete=0
11:21:06 [s3-sync] preparePush.upload 1231ms pushed=29
11:21:06 [s3-sync] preparePush.delete 0ms deleted=0
11:21:07 [s3-sync] commitPush.shards 373ms dirty=10
11:21:07 [s3-sync] commitPush.localIndex 143ms
11:21:07 [s3-sync] commitPush 553ms shard-first changes=29
11:21:13 [s3-sync] pullChanged.fastpath 28ms miss
11:21:22 [s3-sync] pullChanged.pullIndex 8683ms
11:21:22 [s3-sync] pullChanged.remoteListing 78ms keys=291
11:21:22 [s3-sync] pullChanged.walk 161ms toPull=0
11:21:22 [s3-sync] pullChanged.download 0ms pulled=0
11:21:22 [s3-sync] pullChanged.fastpath 33ms miss
11:21:24 [s3-sync] pullChanged.pullIndex 1091ms
11:21:26 [s3-sync] pullChanged.remoteListing 2669ms keys=36687
11:21:31 [s3-sync] pullChanged.walk 4641ms toPull=1
11:21:31 [s3-sync] pullChanged.download 50ms pulled=1
11:21:31 [s3-sync] pushChanged.fastpath 0ms miss
11:21:31 [s3-sync] pushChanged.shardAssembly 152ms scoped=6
11:21:32 [s3-sync] pushChanged.walk 13ms toPush=10 toDelete=0
11:21:32 [s3-sync] pushChanged.upload 84ms pushed=10
11:21:32 [s3-sync] pushChanged.delete 0ms deleted=0
11:21:32 [s3-sync] pushChanged.v2writeback 316ms
11:21:32 [s3-sync] pullChanged.fastpath 28ms hit
11:21:33 [s3-sync] pullChanged.fastpath 25ms hit
11:21:35 [s3-sync] preparePush.fastpath 0ms miss
11:21:35 [s3-sync] preparePush.shardAssembly 208ms scoped=31
11:21:39 [s3-sync] preparePush.walk 4277ms toPush=64 toDelete=0
11:21:39 [s3-sync] preparePush.upload 250ms pushed=40
11:21:39 [s3-sync] preparePush.delete 0ms deleted=0
11:21:40 [s3-sync] commitPush.shards 327ms dirty=9
11:21:40 [s3-sync] commitPush.localIndex 148ms
11:21:40 [s3-sync] commitPush 495ms shard-first changes=40
11:21:46 [s3-sync] pullChanged.fastpath 49ms hit
11:21:51 [s3-sync] pullChanged.fastpath 3133ms hit
11:21:56 [s3-sync] pullChanged.fastpath 23ms hit
11:21:56 [s3-sync] preparePush.fastpath 0ms miss
11:21:57 [s3-sync] preparePush.shardAssembly 156ms scoped=9
11:21:57 [s3-sync] preparePush.walk 351ms toPush=15 toDelete=0
11:21:57 [s3-sync] preparePush.upload 759ms pushed=9
11:21:57 [s3-sync] preparePush.delete 0ms deleted=0
11:21:58 [s3-sync] commitPush.shards 186ms dirty=5
11:21:58 [s3-sync] commitPush.localIndex 163ms
11:21:58 [s3-sync] commitPush 377ms shard-first changes=9
11:22:21 [s3-sync] pullChanged.fastpath 3145ms hit
11:22:23 [s3-sync] pullChanged.fastpath 102ms hit
11:22:34 # Fatal JavaScript out of memory: Ineffective mark-compacts near heap limit
11:22:48 [s3-sync] pullChanged.fastpath 46ms hit
11:23:06 [s3-sync] pullChanged.fastpath 25ms hit
11:23:09 [s3-sync] pushChanged.fastpath 1ms miss
11:23:09 [s3-sync] pushChanged.shardAssembly 371ms scoped=22
11:23:10 [s3-sync] pushChanged.walk 928ms toPush=23 toDelete=19
11:23:10 [s3-sync] pushChanged.upload 125ms pushed=23
11:23:10 [s3-sync] pushChanged.delete 44ms deleted=19
11:23:11 [s3-sync] pushChanged.v2writeback 543ms
11:23:11 [s3-sync] pullChanged.fastpath 22ms hit
11:23:11 [s3-sync] pushChanged.fastpath 0ms miss
11:23:11 [s3-sync] pushChanged.shardAssembly 148ms scoped=13
11:23:11 [s3-sync] pushChanged.walk 5ms toPush=0 toDelete=19
11:23:11 [s3-sync] pushChanged.upload 0ms pushed=0
11:23:11 [s3-sync] pushChanged.delete 40ms deleted=19
11:23:12 [s3-sync] pushChanged.v2writeback 354ms
11:23:12 [s3-sync] pullChanged.fastpath 24ms hit
11:23:12 [s3-sync] pushChanged.fastpath 0ms miss
11:23:12 [s3-sync] pushChanged.shardAssembly 208ms scoped=13
11:23:12 [s3-sync] pushChanged.walk 5ms toPush=0 toDelete=19
11:23:12 [s3-sync] pushChanged.upload 0ms pushed=0
11:23:12 [s3-sync] pushChanged.delete 40ms deleted=19
11:23:13 [s3-sync] pushChanged.v2writeback 386ms
11:23:20 [s3-sync] pullChanged.fastpath 48ms hit
11:23:23 [s3-sync] pullChanged.fastpath 51ms hit
11:23:52 [s3-sync] pullChanged.fastpath 1762ms hit
11:23:53 [s3-sync] pullChanged.fastpath 33ms hit
11:24:04 [s3-sync] pushChanged.fastpath 0ms miss
11:24:09 [s3-sync] pushChanged.shardAssembly 4565ms scoped=1
11:24:09 [s3-sync] pushChanged.walk 14ms toPush=1 toDelete=0
11:24:09 [s3-sync] pushChanged.upload 81ms pushed=1
11:24:09 [s3-sync] pushChanged.delete 0ms deleted=0
11:24:09 [s3-sync] pushChanged.v2writeback 317ms
11:24:20 [s3-sync] pullChanged.fastpath 39ms hit
11:24:21 [s3-sync] pullChanged.fastpath 33ms hit
11:24:50 [s3-sync] pullChanged.fastpath 30ms hit
11:24:51 [s3-sync] pullChanged.fastpath 32ms hit
11:25:21 [s3-sync] pullChanged.fastpath 38ms hit
11:25:27 [s3-sync] pullChanged.fastpath 3137ms hit
11:25:39 [s3-sync] pullChanged.fastpath 35ms hit
11:25:42 [s3-sync] preparePush.fastpath 0ms miss
11:25:42 [s3-sync] preparePush.shardAssembly 378ms scoped=22
11:25:43 [s3-sync] preparePush.walk 1145ms toPush=43 toDelete=0
11:25:43 [s3-sync] preparePush.upload 182ms pushed=31
11:25:43 [s3-sync] preparePush.delete 0ms deleted=0
11:25:44 [s3-sync] commitPush.shards 359ms dirty=11
11:25:44 [s3-sync] commitPush.localIndex 132ms
11:25:44 [s3-sync] commitPush 521ms shard-first changes=31
11:25:56 [s3-sync] pullChanged.fastpath 4719ms hit
11:25:58 [s3-sync] pullChanged.fastpath 110ms hit
11:25:59 [s3-sync] pullChanged.fastpath 22ms hit
11:26:01 [s3-sync] preparePush.fastpath 0ms miss
11:26:02 [s3-sync] preparePush.shardAssembly 267ms scoped=32
11:26:03 [s3-sync] preparePush.walk 1198ms toPush=65 toDelete=0
11:26:03 [s3-sync] preparePush.upload 220ms pushed=41
11:26:03 [s3-sync] preparePush.delete 0ms deleted=0
11:26:03 [s3-sync] commitPush.shards 363ms dirty=10
11:26:03 [s3-sync] commitPush.localIndex 124ms
11:26:03 [s3-sync] commitPush 516ms shard-first changes=41
11:26:16 [s3-sync] pullChanged.fastpath 28ms hit
11:26:21 [s3-sync] pullChanged.fastpath 33ms hit
11:26:22 [s3-sync] pullChanged.fastpath 21ms hit
11:26:27 [s3-sync] preparePush.fastpath 0ms miss
11:26:27 [s3-sync] preparePush.shardAssembly 319ms scoped=17
11:26:28 [s3-sync] preparePush.walk 1063ms toPush=35 toDelete=0
11:26:28 [s3-sync] preparePush.upload 175ms pushed=26
11:26:28 [s3-sync] preparePush.delete 0ms deleted=0
11:26:31 [s3-sync] commitPush.shards 2426ms dirty=10
11:26:31 [s3-sync] commitPush.localIndex 137ms
11:26:31 [s3-sync] commitPush 2599ms shard-first changes=26
11:26:43 [s3-sync] pullChanged.fastpath 27ms hit
11:26:46 [s3-sync] preparePush.fastpath 0ms miss
11:26:46 [s3-sync] preparePush.shardAssembly 289ms scoped=23
11:26:47 [s3-sync] preparePush.walk 1113ms toPush=47 toDelete=0
11:26:47 [s3-sync] preparePush.upload 189ms pushed=32
11:26:47 [s3-sync] preparePush.delete 0ms deleted=0
11:26:47 [s3-sync] commitPush.shards 393ms dirty=10
11:26:48 [s3-sync] commitPush.localIndex 143ms
11:26:48 [s3-sync] commitPush 565ms shard-first changes=32
11:26:53 [s3-sync] pullChanged.fastpath 36ms hit
11:26:54 [s3-sync] pullChanged.fastpath 29ms hit
11:27:00 [s3-sync] pullChanged.fastpath 39ms hit
11:27:04 [s3-sync] preparePush.fastpath 0ms miss
11:27:04 [s3-sync] preparePush.shardAssembly 270ms scoped=20
11:27:05 [s3-sync] preparePush.walk 1042ms toPush=41 toDelete=0
11:27:05 [s3-sync] preparePush.upload 169ms pushed=29
11:27:05 [s3-sync] preparePush.delete 0ms deleted=0
11:27:05 [s3-sync] commitPush.shards 324ms dirty=10
11:27:06 [s3-sync] commitPush.localIndex 142ms
11:27:06 [s3-sync] commitPush 493ms shard-first changes=29
11:27:19 [s3-sync] pullChanged.fastpath 27ms hit
11:27:23 [s3-sync] pullChanged.fastpath 28ms hit
11:27:24 [s3-sync] pullChanged.fastpath 22ms hit
11:27:36 [s3-sync] preparePush.fastpath 0ms miss
11:27:36 [s3-sync] preparePush.shardAssembly 354ms scoped=17
11:27:37 [s3-sync] preparePush.walk 1066ms toPush=35 toDelete=0
11:27:37 [s3-sync] preparePush.upload 169ms pushed=26
11:27:37 [s3-sync] preparePush.delete 0ms deleted=0
11:27:38 [s3-sync] commitPush.shards 327ms dirty=10
11:27:38 [s3-sync] commitPush.localIndex 133ms
11:27:38 [s3-sync] commitPush 494ms shard-first changes=26
11:27:50 [s3-sync] pullChanged.fastpath 44ms hit
11:27:52 Handler error: "capability.getData" for worker "lab-worker-2" after 1ms : "getData: behavior type '@ours/session-write' is outside the active dispatch scope"
11:27:52 Handler error: "capability.getData" for worker "lab-worker-2" after 0ms : "getData: behavior type '@ours/session-write' is outside the active dispatch scope"
11:27:52 Handler error: "capability.getData" for worker "lab-worker-2" after 0ms : "getData: behavior type '@ours/session-write' is outside the active dispatch scope"
11:27:52 Handler error: "capability.getData" for worker "lab-worker-2" after 0ms : "getData: behavior type '@ours/session-write' is outside the active dispatch scope"
11:27:53 Handler error: "capability.getData" for worker "lab-worker-2" after 0ms : "getData: behavior type '@ours/session-write' is outside the active dispatch scope"
11:27:53 Handler error: "capability.getData" for worker "lab-worker-2" after 0ms : "getData: behavior type '@ours/session-write' is outside the active dispatch scope"
11:27:53 Handler error: "capability.getData" for worker "lab-worker-2" after 0ms : "getData: behavior type '@ours/session-write' is outside the active dispatch scope"
11:27:53 Handler error: "capability.getData" for worker "lab-worker-2" after 0ms : "getData: behavior type '@ours/session-write' is outside the active dispatch scope"
11:27:53 [s3-sync] pullChanged.fastpath 20ms hit
11:27:53 [s3-sync] preparePush.fastpath 0ms miss
11:27:53 [s3-sync] preparePush.shardAssembly 270ms scoped=35
11:27:54 [s3-sync] preparePush.walk 1296ms toPush=71 toDelete=0
11:27:55 [s3-sync] preparePush.upload 223ms pushed=44
11:27:55 [s3-sync] preparePush.delete 0ms deleted=0
11:27:55 [s3-sync] commitPush.shards 378ms dirty=10
11:27:55 [s3-sync] commitPush.localIndex 124ms
11:27:55 [s3-sync] commitPush 530ms shard-first changes=44
11:28:01 [s3-sync] pullChanged.fastpath 3011ms hit
11:28:13 [s3-sync] pullChanged.fastpath 50ms hit
11:28:14 [s3-sync] preparePush.fastpath 0ms miss
11:28:15 [s3-sync] preparePush.shardAssembly 310ms scoped=20
11:28:16 [s3-sync] preparePush.walk 1057ms toPush=41 toDelete=0
11:28:16 [s3-sync] preparePush.upload 717ms pushed=29
11:28:16 [s3-sync] preparePush.delete 0ms deleted=0
11:28:17 [s3-sync] commitPush.shards 342ms dirty=10
11:28:17 [s3-sync] commitPush.localIndex 129ms
11:28:17 [s3-sync] commitPush 506ms shard-first changes=29
11:28:25 [s3-sync] pullChanged.fastpath 1601ms hit
11:28:28 [s3-sync] pullChanged.fastpath 30ms hit
11:28:29 [s3-sync] pullChanged.fastpath 29ms hit
11:28:53 [s3-sync] pullChanged.fastpath 36ms hit
11:28:54 [s3-sync] pullChanged.fastpath 29ms hit
11:29:08 [s3-sync] preparePush.fastpath 0ms miss
11:29:09 [s3-sync] preparePush.shardAssembly 442ms scoped=17
11:29:09 [s3-sync] preparePush.walk 1150ms toPush=35 toDelete=0
11:29:09 [s3-sync] preparePush.upload 171ms pushed=26
11:29:09 [s3-sync] preparePush.delete 0ms deleted=0
11:29:10 [s3-sync] commitPush.shards 335ms dirty=10
11:29:10 [s3-sync] commitPush.localIndex 153ms
11:29:10 [s3-sync] commitPush 514ms shard-first changes=26
11:29:24 [s3-sync] pullChanged.fastpath 48ms hit
11:29:25 [s3-sync] pullChanged.fastpath 22ms hit
11:29:25 [s3-sync] pullChanged.fastpath 28ms hit
11:29:27 Handler error: "capability.getData" for worker "lab-worker-2" after 1ms : "getData: behavior type '@ours/session-write' is outside the active dispatch scope"
11:29:27 Handler error: "capability.getData" for worker "lab-worker-2" after 0ms : "getData: behavior type '@ours/session-write' is outside the active dispatch scope"
11:29:27 [s3-sync] preparePush.fastpath 0ms miss
11:29:28 [s3-sync] preparePush.shardAssembly 561ms scoped=26
11:29:29 [s3-sync] preparePush.walk 1412ms toPush=53 toDelete=0
11:29:29 [s3-sync] preparePush.upload 233ms pushed=35
11:29:29 [s3-sync] preparePush.delete 0ms deleted=0
11:29:30 [s3-sync] commitPush.shards 336ms dirty=10
11:29:30 [s3-sync] commitPush.localIndex 137ms
11:29:30 [s3-sync] commitPush 491ms shard-first changes=35
11:29:49 [s3-sync] pushChanged.fastpath 0ms miss
11:29:51 [s3-sync] pushChanged.shardAssembly 2165ms scoped=3
11:29:51 [s3-sync] pushChanged.walk 373ms toPush=7 toDelete=0
11:29:52 [s3-sync] pushChanged.upload 90ms pushed=7
11:29:52 [s3-sync] pushChanged.delete 0ms deleted=0
11:29:52 [s3-sync] pushChanged.v2writeback 303ms
11:29:54 [s3-sync] pullChanged.fastpath 23ms hit
11:29:55 [s3-sync] pullChanged.fastpath 27ms hit
11:30:24 [s3-sync] pullChanged.fastpath 59ms hit
11:30:27 [s3-sync] pullChanged.fastpath 80ms hit
11:30:57 [s3-sync] pullChanged.fastpath 42ms hit
11:31:03 [s3-sync] pullChanged.fastpath 3077ms hit
11:31:29 [s3-sync] pullChanged.fastpath 155ms hit
11:31:35 [s3-sync] pullChanged.fastpath 3004ms hit
11:31:36 [s3-sync] pullChanged.fastpath 32ms hit
11:31:36 [s3-sync] pullChanged.fastpath 19ms hit
11:31:40 [s3-sync] preparePush.fastpath 0ms miss
11:31:41 [s3-sync] preparePush.shardAssembly 676ms scoped=31
11:31:45 [s3-sync] preparePush.fastpath 0ms miss
11:31:45 [s3-sync] preparePush.shardAssembly 673ms scoped=45
11:31:46 [s3-sync] preparePush.walk 5723ms toPush=61 toDelete=0
11:31:49 [s3-sync] preparePush.upload 2455ms pushed=49
11:31:49 [s3-sync] preparePush.delete 0ms deleted=0
11:31:49 [s3-sync] preparePush.walk 4630ms toPush=91 toDelete=0
11:31:49 [s3-sync] commitPush.shards 585ms dirty=14
11:31:50 [s3-sync] commitPush.localIndex 127ms
11:31:50 [s3-sync] commitPush 756ms shard-first changes=49
11:31:50 [s3-sync] preparePush.upload 602ms pushed=67
11:31:50 [s3-sync] preparePush.delete 0ms deleted=0
11:32:06 [s3-sync] commitPush.shards 10586ms dirty=16
11:32:06 [s3-sync] commitPush.localIndex 130ms
11:32:06 [s3-sync] commitPush 10748ms shard-first changes=67
11:32:12 [s3-sync] pullChanged.fastpath 5015ms hit
11:32:13 [s3-sync] pullChanged.fastpath 27ms hit
11:32:15 [s3-sync] pullChanged.fastpath 62ms hit
11:32:24 [s3-sync] pullChanged.fastpath 23ms hit
11:32:24 [s3-sync] preparePush.fastpath 0ms miss
11:32:24 [s3-sync] preparePush.shardAssembly 328ms scoped=40
11:32:25 [s3-sync] preparePush.walk 1289ms toPush=80 toDelete=0
11:32:26 [s3-sync] preparePush.upload 287ms pushed=56
11:32:26 [s3-sync] preparePush.delete 0ms deleted=0
11:32:26 [s3-sync] commitPush.shards 327ms dirty=13
11:32:26 [s3-sync] commitPush.localIndex 127ms
11:32:26 [s3-sync] commitPush 493ms shard-first changes=56
11:32:27 [s3-sync] preparePush.fastpath 0ms miss
11:32:32 [s3-sync] preparePush.shardAssembly 5606ms scoped=33
11:32:44 [s3-sync] preparePush.walk 17552ms toPush=61 toDelete=0
11:32:45 [s3-sync] preparePush.upload 286ms pushed=37
11:32:45 [s3-sync] preparePush.delete 0ms deleted=0
11:32:45 [s3-sync] commitPush.shards 382ms dirty=12
11:32:45 [s3-sync] commitPush.localIndex 179ms
11:32:45 [s3-sync] commitPush 593ms shard-first changes=37
11:32:46 [s3-sync] pullChanged.fastpath 28ms hit
11:32:48 [s3-sync] pullChanged.fastpath 2220ms hit
11:32:54 [s3-sync] pullChanged.fastpath 3260ms hit
11:33:03 [s3-sync] pullChanged.fastpath 51ms hit
11:33:03 [s3-sync] pullChanged.fastpath 26ms hit
11:33:04 [s3-sync] pullChanged.fastpath 19ms hit
11:33:07 [s3-sync] preparePush.fastpath 0ms miss
11:33:08 [s3-sync] preparePush.shardAssembly 291ms scoped=25
11:33:08 [s3-sync] preparePush.walk 909ms toPush=49 toDelete=0
11:33:08 [s3-sync] preparePush.upload 183ms pushed=40
11:33:08 [s3-sync] preparePush.delete 0ms deleted=0
11:33:09 [s3-sync] commitPush.shards 363ms dirty=13
11:33:09 [s3-sync] commitPush.localIndex 146ms
11:33:09 [s3-sync] commitPush 543ms shard-first changes=40
11:33:24 [s3-sync] preparePush.fastpath 0ms miss
11:33:25 [s3-sync] preparePush.shardAssembly 682ms scoped=18
11:33:26 [s3-sync] preparePush.walk 1437ms toPush=31 toDelete=0
11:33:26 [s3-sync] preparePush.upload 160ms pushed=22
11:33:26 [s3-sync] preparePush.delete 0ms deleted=0
11:33:29 [s3-sync] commitPush.shards 462ms dirty=12
11:33:29 [s3-sync] commitPush.localIndex 152ms
11:33:29 [s3-sync] commitPush 3049ms shard-first changes=22
11:33:35 [s3-sync] pullChanged.fastpath 35ms hit
11:33:35 [s3-sync] pullChanged.fastpath 24ms hit
11:33:37 [s3-sync] pullChanged.fastpath 63ms hit
11:33:44 [s3-sync] pullChanged.fastpath 26ms hit
11:33:46 [s3-sync] preparePush.fastpath 0ms miss
11:33:46 [s3-sync] preparePush.shardAssembly 290ms scoped=31
11:33:47 [s3-sync] preparePush.walk 1164ms toPush=61 toDelete=0
11:33:47 [s3-sync] preparePush.upload 254ms pushed=46
11:33:47 [s3-sync] preparePush.delete 0ms deleted=0
11:33:48 [s3-sync] commitPush.shards 449ms dirty=13
11:33:48 [s3-sync] commitPush.localIndex 129ms
11:33:48 [s3-sync] commitPush 628ms shard-first changes=46
11:34:03 [s3-sync] preparePush.fastpath 0ms miss
11:34:05 [s3-sync] preparePush.shardAssembly 2697ms scoped=24
11:34:06 [s3-sync] preparePush.walk 3420ms toPush=43 toDelete=0
11:34:06 [s3-sync] preparePush.upload 186ms pushed=28
11:34:06 [s3-sync] preparePush.delete 0ms deleted=0
11:34:07 [s3-sync] commitPush.shards 389ms dirty=12
11:34:07 [s3-sync] commitPush.localIndex 169ms
11:34:07 [s3-sync] commitPush 595ms shard-first changes=28
11:34:07 [s3-sync] pullChanged.fastpath 25ms hit
11:34:10 [s3-sync] pullChanged.fastpath 64ms hit
11:34:15 [s3-sync] pullChanged.fastpath 2936ms hit
11:34:21 [s3-sync] preparePush.fastpath 0ms miss
11:34:21 [s3-sync] preparePush.shardAssembly 416ms scoped=20
11:34:24 [s3-sync] preparePush.walk 3465ms toPush=41 toDelete=0
11:34:25 [s3-sync] preparePush.upload 190ms pushed=29
11:34:25 [s3-sync] preparePush.delete 0ms deleted=0
11:34:25 [s3-sync] commitPush.shards 389ms dirty=10
11:34:25 [s3-sync] commitPush.localIndex 150ms
11:34:25 [s3-sync] commitPush 562ms shard-first changes=29
11:34:25 [s3-sync] pullChanged.fastpath 32ms hit
11:34:42 [s3-sync] pullChanged.fastpath 6725ms hit
11:34:42 [s3-sync] preparePush.fastpath 0ms miss
11:34:43 [s3-sync] preparePush.shardAssembly 327ms scoped=21
11:34:43 [s3-sync] preparePush.walk 1118ms toPush=41 toDelete=0
11:34:44 [s3-sync] preparePush.upload 164ms pushed=29
11:34:44 [s3-sync] preparePush.delete 0ms deleted=0
11:34:44 [s3-sync] commitPush.shards 335ms dirty=10
11:34:44 [s3-sync] commitPush.localIndex 185ms
11:34:44 [s3-sync] commitPush 551ms shard-first changes=29
11:34:44 [s3-sync] pullChanged.fastpath 27ms hit
11:34:50 [s3-sync] pullChanged.fastpath 4815ms hit
11:34:59 [s3-sync] pullChanged.fastpath 28ms hit
11:35:05 [s3-sync] pullChanged.fastpath 29ms hit
11:35:06 [s3-sync] pullChanged.fastpath 23ms hit
11:35:10 [s3-sync] preparePush.fastpath 0ms miss
11:35:10 [s3-sync] preparePush.shardAssembly 630ms scoped=24
11:35:11 [s3-sync] preparePush.walk 1284ms toPush=49 toDelete=0
11:35:11 [s3-sync] preparePush.upload 184ms pushed=40
11:35:11 [s3-sync] preparePush.delete 0ms deleted=0
11:35:12 [s3-sync] commitPush.shards 374ms dirty=13
11:35:12 [s3-sync] commitPush.localIndex 135ms
11:35:12 [s3-sync] commitPush 540ms shard-first changes=40
11:35:29 [s3-sync] preparePush.fastpath 0ms miss
11:35:30 [s3-sync] preparePush.shardAssembly 375ms scoped=18
11:35:30 [s3-sync] preparePush.walk 1036ms toPush=31 toDelete=0
11:35:31 [s3-sync] preparePush.upload 179ms pushed=22
11:35:31 [s3-sync] preparePush.delete 0ms deleted=0
11:35:33 [s3-sync] commitPush.shards 2625ms dirty=12
11:35:34 [s3-sync] commitPush.localIndex 134ms
11:35:34 [s3-sync] commitPush 2788ms shard-first changes=22
11:35:34 [s3-sync] pullChanged.fastpath 30ms hit
11:35:39 [s3-sync] pullChanged.fastpath 3256ms hit
11:35:41 [s3-sync] pullChanged.fastpath 184ms hit
11:35:41 Handler error: "capability.getData" for worker "lab-worker-2" after 0ms : "getData: behavior type '@ours/session-write' is outside the active dispatch scope"
11:35:41 Handler error: "capability.getData" for worker "lab-worker-2" after 1ms : "getData: behavior type '@ours/session-write' is outside the active dispatch scope"
11:35:41 Handler error: "capability.getData" for worker "lab-worker-2" after 0ms : "getData: behavior type '@ours/session-write' is outside the active dispatch scope"
11:35:41 Handler error: "capability.getData" for worker "lab-worker-2" after 1ms : "getData: behavior type '@ours/session-write' is outside the active dispatch scope"
11:35:46 Handler error: "capability.getData" for worker "lab-worker-2" after 1ms : "getData: behavior type '@ours/session-write' is outside the active dispatch scope"
11:35:46 Handler error: "capability.getData" for worker "lab-worker-2" after 0ms : "getData: behavior type '@ours/session-write' is outside the active dispatch scope"
11:35:46 Handler error: "capability.getData" for worker "lab-worker-2" after 0ms : "getData: behavior type '@ours/session-write' is outside the active dispatch scope"
11:35:46 Handler error: "capability.getData" for worker "lab-worker-2" after 0ms : "getData: behavior type '@ours/session-write' is outside the active dispatch scope"
11:35:49 [s3-sync] pullChanged.fastpath 29ms hit
11:35:50 [s3-sync] preparePush.fastpath 0ms miss
11:35:50 [s3-sync] preparePush.shardAssembly 532ms scoped=42
11:35:56 Handler error: "capability.getData" for worker "lab-worker-2" after 0ms : "getData: behavior type '@ours/session-write' is outside the active dispatch scope"
11:35:56 Handler error: "capability.getData" for worker "lab-worker-2" after 1ms : "getData: behavior type '@ours/session-write' is outside the active dispatch scope"
11:35:56 Handler error: "capability.getData" for worker "lab-worker-2" after 0ms : "getData: behavior type '@ours/session-write' is outside the active dispatch scope"
11:35:56 Handler error: "capability.getData" for worker "lab-worker-2" after 0ms : "getData: behavior type '@ours/session-write' is outside the active dispatch scope"
11:35:57 [s3-sync] preparePush.walk 7307ms toPush=85 toDelete=0
11:35:57 Handler error: "capability.getData" for worker "lab-worker-2" after 0ms : "getData: behavior type '@ours/session-write' is outside the active dispatch scope"
11:35:57 Handler error: "capability.getData" for worker "lab-worker-2" after 0ms : "getData: behavior type '@ours/session-write' is outside the active dispatch scope"
11:35:57 Handler error: "capability.getData" for worker "lab-worker-2" after 0ms : "getData: behavior type '@ours/session-write' is outside the active dispatch scope"
11:35:57 Handler error: "capability.getData" for worker "lab-worker-2" after 0ms : "getData: behavior type '@ours/session-write' is outside the active dispatch scope"
11:35:57 [s3-sync] preparePush.upload 370ms pushed=58
11:35:57 [s3-sync] preparePush.delete 0ms deleted=0
11:35:58 [s3-sync] commitPush.shards 388ms dirty=13
11:35:58 [s3-sync] commitPush.localIndex 143ms
11:35:58 [s3-sync] commitPush 575ms shard-first changes=58
11:35:58 [s3-sync] preparePush.fastpath 0ms miss
11:35:59 [s3-sync] preparePush.shardAssembly 257ms scoped=35
11:36:11 [s3-sync] preparePush.walk 12512ms toPush=66 toDelete=0
11:36:13 [s3-sync] preparePush.upload 2358ms pushed=39
11:36:13 [s3-sync] preparePush.delete 0ms deleted=0
11:36:34 [s3-sync] commitPush.shards 15281ms dirty=11
11:36:35 [s3-sync] commitPush.localIndex 549ms
11:36:35 [s3-sync] commitPush 15948ms shard-first changes=39
11:36:35 [s3-sync] pullChanged.fastpath 24ms hit
11:36:41 [s3-sync] pullChanged.fastpath 3014ms hit
11:36:43 [s3-sync] pushChanged.fastpath 0ms miss
11:36:49 [s3-sync] pushChanged.shardAssembly 5600ms scoped=4
11:36:56 [s3-sync] pushChanged.walk 7567ms toPush=8 toDelete=0
11:36:56 [s3-sync] pushChanged.upload 88ms pushed=8
11:36:56 [s3-sync] pushChanged.delete 0ms deleted=0
11:36:57 [s3-sync] pushChanged.v2writeback 316ms
11:36:59 [s3-sync] pullChanged.fastpath 1924ms hit
11:37:13 [s3-sync] pullChanged.fastpath 101ms hit
11:37:16 [s3-sync] pullChanged.fastpath 108ms hit
11:37:31 [s3-sync] pushChanged.fastpath 0ms miss
11:37:34 [s3-sync] pushChanged.shardAssembly 2536ms scoped=3
11:37:34 [s3-sync] pushChanged.walk 61ms toPush=7 toDelete=0
11:37:34 [s3-sync] pushChanged.upload 103ms pushed=7
11:37:34 [s3-sync] pushChanged.delete 0ms deleted=0
11:37:34 [s3-sync] pushChanged.v2writeback 352ms
11:37:43 [s3-sync] pullChanged.fastpath 31ms hit
11:37:46 [s3-sync] pullChanged.fastpath 40ms hit
11:38:13 [s3-sync] pullChanged.fastpath 25ms hit
11:38:14 [s3-sync] pullChanged.fastpath 28ms hit
11:38:44 [s3-sync] pullChanged.fastpath 22ms hit
11:38:45 [s3-sync] pullChanged.fastpath 23ms hit
11:39:14 [s3-sync] pullChanged.fastpath 26ms hit
11:39:15 [s3-sync] pullChanged.fastpath 66ms hit
11:39:45 [s3-sync] pullChanged.fastpath 21ms hit
11:39:46 [s3-sync] pullChanged.fastpath 48ms hit
11:40:15 [s3-sync] pullChanged.fastpath 43ms miss
11:40:16 [s3-sync] pullChanged.pullIndex 1338ms
11:40:17 [s3-sync] pullChanged.remoteListing 77ms keys=291
11:40:17 [s3-sync] pullChanged.walk 160ms toPull=0
11:40:17 [s3-sync] pullChanged.download 0ms pulled=0
11:40:17 [s3-sync] pullChanged.fastpath 22ms miss
11:40:18 [s3-sync] pullChanged.pullIndex 1118ms
11:40:20 [s3-sync] pullChanged.remoteListing 1387ms keys=10549
11:40:21 [s3-sync] pullChanged.walk 1507ms toPull=0
11:40:21 [s3-sync] pullChanged.download 0ms pulled=0
11:40:45 [s3-sync] pullChanged.fastpath 38ms miss
11:40:46 [s3-sync] pullChanged.pullIndex 1294ms
11:40:47 [s3-sync] pullChanged.remoteListing 54ms keys=291
11:40:47 [s3-sync] pullChanged.walk 164ms toPull=0
11:40:47 [s3-sync] pullChanged.download 0ms pulled=0
11:40:47 [s3-sync] pullChanged.fastpath 32ms miss
11:40:48 [s3-sync] pullChanged.pullIndex 1097ms
11:40:50 [s3-sync] pullChanged.remoteListing 1431ms keys=10549
11:40:51 [s3-sync] pullChanged.walk 1351ms toPull=0
11:40:51 [s3-sync] pullChanged.download 0ms pulled=0
11:41:15 [s3-sync] pullChanged.fastpath 42ms miss
11:41:16 [s3-sync] pullChanged.pullIndex 1175ms
11:41:16 [s3-sync] pullChanged.remoteListing 60ms keys=291
11:41:17 [s3-sync] pullChanged.walk 178ms toPull=0
11:41:17 [s3-sync] pullChanged.download 0ms pulled=0
11:41:17 [s3-sync] pullChanged.fastpath 22ms miss
11:41:18 [s3-sync] pullChanged.pullIndex 1146ms
11:41:20 [s3-sync] pullChanged.remoteListing 1222ms keys=10549
11:41:21 [s3-sync] pullChanged.walk 1386ms toPull=0
11:41:21 [s3-sync] pullChanged.download 0ms pulled=0
11:41:52 [s3-sync] pullChanged.fastpath 3254ms miss
11:41:58 [s3-sync] pullChanged.pullIndex 6658ms
11:41:59 [s3-sync] pullChanged.remoteListing 89ms keys=291
11:41:59 [s3-sync] pullChanged.walk 281ms toPull=0
11:41:59 [s3-sync] pullChanged.download 0ms pulled=0
11:41:59 [s3-sync] pullChanged.fastpath 28ms miss
11:41:59 [s3-sync] pullChanged.pullIndex 99ms
11:42:04 [s3-sync] pullChanged.remoteListing 5330ms keys=37421
11:42:10 [s3-sync] pullChanged.walk 5787ms toPull=0
11:42:10 [s3-sync] pullChanged.download 0ms pulled=0
11:42:12 [s3-sync] pullChanged.fastpath 36ms miss
11:42:14 [s3-sync] pullChanged.pullIndex 1852ms
11:42:20 [s3-sync] pullChanged.remoteListing 5628ms keys=10550
11:42:23 [s3-sync] pullChanged.walk 3525ms toPull=3
11:42:23 [s3-sync] pullChanged.download 48ms pulled=3
11:42:24 [s3-sync] preparePush.fastpath 0ms miss
11:42:24 [s3-sync] preparePush.shardAssembly 295ms scoped=22
11:42:25 [s3-sync] preparePush.walk 993ms toPush=41 toDelete=0
11:42:25 [s3-sync] preparePush.upload 191ms pushed=29
11:42:25 [s3-sync] preparePush.delete 0ms deleted=0
11:42:27 [s3-sync] commitPush.shards 308ms dirty=10
11:42:27 [s3-sync] commitPush.localIndex 129ms
11:42:27 [s3-sync] commitPush 471ms shard-first changes=29
11:42:30 [s3-sync] pullChanged.fastpath 1859ms miss
11:42:42 [s3-sync] pullChanged.pullIndex 12092ms
11:42:44 [s3-sync] pullChanged.remoteListing 76ms keys=291
11:42:45 [s3-sync] pullChanged.walk 549ms toPull=0
11:42:45 [s3-sync] pullChanged.download 0ms pulled=0
11:42:45 [s3-sync] pullChanged.fastpath 32ms miss
11:42:45 [s3-sync] pullChanged.pullIndex 53ms
11:42:48 [s3-sync] pullChanged.remoteListing 2468ms keys=37447
11:42:54 [s3-sync] pullChanged.walk 6358ms toPull=1
11:42:54 [s3-sync] pullChanged.download 43ms pulled=1
11:42:54 [s3-sync] pullChanged.fastpath 28ms miss
11:42:56 [s3-sync] pullChanged.pullIndex 1479ms
11:42:56 [s3-sync] pullChanged.remoteListing 78ms keys=291
11:42:57 [s3-sync] pullChanged.walk 353ms toPull=0
11:42:57 [s3-sync] pullChanged.download 0ms pulled=0
11:42:59 [s3-sync] pullChanged.fastpath 40ms miss
11:43:00 [s3-sync] pullChanged.pullIndex 1547ms
11:43:02 [s3-sync] pullChanged.remoteListing 1455ms keys=10571
11:43:03 [s3-sync] pullChanged.walk 1325ms toPull=3
11:43:03 [s3-sync] pullChanged.download 25ms pulled=3
11:43:03 [s3-sync] preparePush.fastpath 0ms miss
11:43:03 [s3-sync] preparePush.shardAssembly 238ms scoped=32
11:43:04 [s3-sync] preparePush.walk 1180ms toPush=63 toDelete=0
11:43:05 [s3-sync] preparePush.upload 233ms pushed=39
11:43:05 [s3-sync] preparePush.delete 0ms deleted=0
11:43:05 [s3-sync] commitPush.shards 337ms dirty=8
11:43:05 [s3-sync] commitPush.localIndex 134ms
11:43:05 [s3-sync] commitPush 503ms shard-first changes=39
11:43:20 [s3-sync] pullChanged.fastpath 55ms miss
11:43:22 [s3-sync] pullChanged.pullIndex 1458ms
11:43:22 [s3-sync] pullChanged.remoteListing 87ms keys=291
11:43:24 [s3-sync] pullChanged.walk 2219ms toPull=0
11:43:24 [s3-sync] pullChanged.download 0ms pulled=0
11:43:24 [s3-sync] pullChanged.fastpath 66ms miss
11:43:24 [s3-sync] pullChanged.pullIndex 66ms
11:43:29 [s3-sync] pullChanged.remoteListing 5270ms keys=37483
11:43:34 [s3-sync] pullChanged.walk 4444ms toPull=1
11:43:34 [s3-sync] pullChanged.download 30ms pulled=1
11:43:34 [s3-sync] pullChanged.fastpath 26ms miss
11:43:35 [s3-sync] pullChanged.pullIndex 1140ms
11:43:37 [s3-sync] pullChanged.remoteListing 1286ms keys=10604
11:43:38 [s3-sync] pullChanged.walk 990ms toPull=1
11:43:38 [s3-sync] pullChanged.download 27ms pulled=1
11:43:45 [s3-sync] preparePush.fastpath 0ms miss
11:43:45 [s3-sync] preparePush.shardAssembly 288ms scoped=17
11:43:46 [s3-sync] preparePush.walk 1018ms toPush=34 toDelete=0
11:43:46 [s3-sync] preparePush.upload 150ms pushed=25
11:43:46 [s3-sync] preparePush.delete 0ms deleted=0
11:43:46 [s3-sync] commitPush.shards 325ms dirty=9
11:43:46 [s3-sync] commitPush.localIndex 137ms
11:43:46 [s3-sync] commitPush 499ms shard-first changes=25
11:43:52 [s3-sync] pullChanged.fastpath 31ms miss
11:44:03 # Fatal JavaScript out of memory: Ineffective mark-compacts near heap limit
11:44:17 [s3-sync] pullChanged.fastpath 40ms miss
11:44:19 [s3-sync] pullChanged.pullIndex 1541ms
11:44:22 [s3-sync] pullChanged.remoteListing 3034ms keys=37503
11:44:25 [s3-sync] pullChanged.walk 3436ms toPull=1
11:44:25 [s3-sync] pullChanged.download 32ms pulled=1
11:44:43 Handler slow: "capability.queryData" for worker "lab-worker-2" took 5722ms
11:44:57 [s3-sync] pullChanged.fastpath 28ms hit
11:44:58 [s3-sync] pullChanged.fastpath 27ms hit
11:45:27 [s3-sync] pullChanged.fastpath 35ms hit
11:45:28 [s3-sync] pullChanged.fastpath 21ms hit
11:45:40 [s3-sync] pushChanged.fastpath 0ms miss
11:45:45 [s3-sync] pushChanged.shardAssembly 4830ms scoped=9
11:45:46 [s3-sync] pushChanged.walk 817ms toPush=23 toDelete=0
11:45:46 [s3-sync] pushChanged.upload 160ms pushed=23
11:45:46 [s3-sync] pushChanged.delete 0ms deleted=0
11:45:46 [s3-sync] pushChanged.v2writeback 530ms
11:45:59 [s3-sync] pullChanged.fastpath 42ms hit
11:46:00 [s3-sync] pullChanged.fastpath 37ms hit
11:46:29 [s3-sync] pullChanged.fastpath 19ms hit
11:46:30 [s3-sync] pullChanged.fastpath 26ms hit
11:46:59 [s3-sync] pullChanged.fastpath 40ms hit
11:47:00 [s3-sync] pullChanged.fastpath 20ms hit
11:47:29 [s3-sync] pullChanged.fastpath 27ms hit
11:47:33 [s3-sync] pullChanged.fastpath 2059ms hit
11:48:04 [s3-sync] pullChanged.fastpath 3050ms hit
11:48:06 [s3-sync] pullChanged.fastpath 28ms hit
11:48:06 [s3-sync] pullChanged.fastpath 16ms hit
11:48:09 [s3-sync] preparePush.fastpath 0ms miss
11:48:10 [s3-sync] preparePush.shardAssembly 775ms scoped=22
11:48:13 [s3-sync] preparePush.walk 3699ms toPush=42 toDelete=0
11:48:14 [s3-sync] preparePush.upload 663ms pushed=30
11:48:14 [s3-sync] preparePush.delete 0ms deleted=0
11:48:14 [s3-sync] commitPush.shards 430ms dirty=11
11:48:14 [s3-sync] commitPush.localIndex 139ms
11:48:14 [s3-sync] commitPush 607ms shard-first changes=30
11:48:30 [s3-sync] pullChanged.fastpath 23ms hit
11:48:31 [s3-sync] pullChanged.fastpath 28ms hit
11:48:37 [s3-sync] pullChanged.fastpath 29ms hit
11:48:42 [s3-sync] preparePush.fastpath 0ms miss
11:48:42 [s3-sync] preparePush.shardAssembly 327ms scoped=32
11:48:43 [s3-sync] preparePush.walk 1238ms toPush=65 toDelete=0
11:48:43 [s3-sync] preparePush.upload 253ms pushed=41
11:48:43 [s3-sync] preparePush.delete 0ms deleted=0
11:48:46 [s3-sync] commitPush.shards 2333ms dirty=10
11:48:46 [s3-sync] commitPush.localIndex 382ms
11:48:46 [s3-sync] commitPush 2749ms shard-first changes=41
11:49:02 [s3-sync] pullChanged.fastpath 34ms hit
11:49:03 [s3-sync] pullChanged.fastpath 40ms hit
11:49:03 [s3-sync] pullChanged.fastpath 34ms hit
11:49:11 [s3-sync] preparePush.fastpath 0ms miss
11:49:11 [s3-sync] preparePush.shardAssembly 353ms scoped=17
11:49:12 [s3-sync] preparePush.walk 1137ms toPush=35 toDelete=0
11:49:12 [s3-sync] preparePush.upload 167ms pushed=26
11:49:12 [s3-sync] preparePush.delete 0ms deleted=0
11:49:13 [s3-sync] commitPush.shards 385ms dirty=10
11:49:13 [s3-sync] commitPush.localIndex 120ms
11:49:13 [s3-sync] commitPush 548ms shard-first changes=26
11:49:26 [s3-sync] pullChanged.fastpath 45ms hit
11:49:30 [s3-sync] preparePush.fastpath 0ms miss
11:49:30 [s3-sync] preparePush.shardAssembly 269ms scoped=23
11:49:31 [s3-sync] preparePush.walk 1122ms toPush=47 toDelete=0
11:49:32 [s3-sync] preparePush.upload 193ms pushed=32
11:49:32 [s3-sync] preparePush.delete 0ms deleted=0
11:49:32 [s3-sync] commitPush.shards 343ms dirty=10
11:49:32 [s3-sync] commitPush.localIndex 131ms
11:49:32 [s3-sync] commitPush 495ms shard-first changes=32
11:49:35 [s3-sync] pullChanged.fastpath 1890ms hit
11:49:38 [s3-sync] pullChanged.fastpath 18ms hit
11:49:49 [s3-sync] pullChanged.fastpath 42ms hit
11:49:49 [s3-sync] preparePush.fastpath 0ms miss
11:49:50 [s3-sync] pushChanged.fastpath 0ms miss
11:49:50 [s3-sync] preparePush.shardAssembly 242ms scoped=9
11:49:50 [s3-sync] preparePush.walk 397ms toPush=15 toDelete=0
11:49:50 [s3-sync] pushChanged.shardAssembly 337ms scoped=9
11:49:50 [s3-sync] pushChanged.walk 50ms toPush=15 toDelete=0
11:49:50 [s3-sync] pushChanged.upload 74ms pushed=15
11:49:50 [s3-sync] pushChanged.delete 0ms deleted=0
11:49:50 [s3-sync] pushChanged.v2writeback 335ms
11:49:51 [s3-sync] preparePush.upload 739ms pushed=9
11:49:51 [s3-sync] preparePush.delete 0ms deleted=0
11:49:54 [s3-sync] pullChanged.fastpath 29ms hit
11:50:25 [s3-sync] pullChanged.fastpath 35ms hit
11:50:26 [s3-sync] pullChanged.fastpath 31ms hit
11:50:57 [s3-sync] pullChanged.fastpath 61ms hit
11:50:58 [s3-sync] pullChanged.fastpath 25ms hit
11:51:27 [s3-sync] pullChanged.fastpath 46ms hit
11:51:28 [s3-sync] pullChanged.fastpath 32ms hit
11:51:57 [s3-sync] pullChanged.fastpath 30ms hit
11:51:58 [s3-sync] pullChanged.fastpath 21ms hit
11:52:19 # Fatal JavaScript out of memory: Ineffective mark-compacts near heap limit
11:52:32 [s3-sync] pullChanged.fastpath 43ms hitvcjdeboer commented 10/31/2026, 9:44:49 PM
Off-lab reproduction of this issue's error, with the pinned versions (no AWS)
Following up on the trace we posted: we could reproduce RPC call 'capability.queryData' timed out after 30000ms locally, with swamp a2f815f3, @swamp/s3-datastore 2026.10.24.2, MinIO and one remote worker. The scripts are in the next ripple as a self-extracting bundle.
The failure
A step on the remote worker fails with RPC call 'capability.queryData' timed out after 30000ms. At serve, that queryData is answered only after a per-model datastore pull (Syncing behavior … from datastore) for the next dispatch has listed and walked the data prefix. See results/queryData-78a8186d-window.log: handler start 21:11:13.4; remoteListing 29395ms keys=11088; walk 5700ms; the worker's dispatch fails at 33.5 s; serve answers at 42,976 ms.
Cases (each = two runs started together, so they run serially on the capacity-1 worker; a client polls as described below)
| case | S3 latency (netem on MinIO) | second writer commits | runs failed (queryData RPC timeout) | serve queryData max |
largest listing |
|---|---|---|---|---|---|
| c: no latency | 0 ms | on | 0/2 (also 0/1 in a one-run stage) | 26 s | 22.7 s |
| d: failing | 30 ms | on | 2/2 (place, fill) |
43.0 s | 29.4 s |
| d2: repeat | 30 ms | on | 1/2 (fill) |
42.2 s | 31.3 s |
| d3: no second writer | 30 ms | off | 0/2 | 22.3 s | 29.4 s |
So it repeats with latency plus foreign commits, and was absent in the one comparison without foreign commits. With two runs per case that is consistent, not proven: d3 still had 29 s listings.
Setup (what makes it slow)
- Storage: ~12k objects under prefix
lab/(data/11.2k,outputs/0.6k,_index/64),.datastore-index.json3.86 MB. Seeded through real method runs (gen_seed.py): one name with many versions (a heartbeat-like record), several hundred one-version names, and per-member models with ~100 names each. - Run (
gen_analysis.py): 9 steps.- A ~630 KB octet-stream input file, passed as a base64 workflow input and written as a file record.
- Two octet-stream outputs (250 + 200 KB) and small JSON resources.
- Steps 2–9 call
context.queryData('modelName == "<member model>" && specName == "rec"'). Each call returns 92–119 hydrated rows, 0.7–6 MB.
- Second writer: a copy of the repo with its own
SWAMP_HOME(so its own cache). It makes one small method run against the same bucket about once a minute during the runs, which keeps serve's subdir-scoped pollers on the slow path (fast-path misses in the trace). - Client (
driver.py):workflow history getevery 2 s untilexecute-parseruns, then every 10 s;history searchevery ~4 s; afterwards aversion >= 0tag query and 3 paralleldata get --json. Serve runs withSWAMP_LOG_LEVEL=debug SWAMP_S3_SYNC_TRACE=1.
Run it (Docker on arm64, needs a swamp-club collective key with scopes serve:*,datastore:*)
mkdir key && printf %s "$SWAMP_API_KEY" > key/club-key # optional: key/signin with SWAMP_SIGNIN_TOKEN
zsh fetch-swamp.sh && zsh build-minio.sh && zsh up.sh
zsh setup-repo.sh # seeding: we stopped it after 30 min (~10k objects)
zsh serve-start.sh 1 && zsh worker-start.sh && (zsh sampler.sh &)
zsh run-case.sh c-nolat 0 on # comparison: no latency
zsh run-case.sh d-fail 30 on # failing case
zsh run-case.sh d-noforeign 30 off
python3 timeline.py work/probe work/probe/w/serve-1.log 10 # peak windows and overlapping operationsResults go to work/probe/requests.csv, samples.csv and w/serve-1.log. Clean up with docker rm -f r4102-serve r4102-worker r4102-minio; docker network rm r4102-net; docker image rm r4102-base:1; rm -rf work.
Limits (read before relying on numbers)
- netem exaggerates listings. netem delays every packet, so multi-page
ListObjectsV2took 29 s for 11k keys. Against real S3 our production trace shows 1.2–2.7 s for 10.5k keys. The repro reproduces the wait chain and the error, with a slower listing than real S3. In production the same 30 s limit was crossed with faster listings but ~37k objects and real step workloads. - Small counts: two runs per case.
- Hardware: Apple M-series under Docker, not Graviton.
- Seed not completed: the heartbeat-like record has ~161 versions, not the 1,078 we see in production.
- The scripts are the ones that ran, with paths and names made generic afterwards; they were not re-run after that edit.
Serve log for the failing call (request 78a8186d, case d; per-row CEL skip lines removed)
2026-10-31T21:11:13.418Z system │ Handler start: "capability.queryData" for worker "w1" (request "78a8186d-c354-429b-9627-e95beaf3fe4e")
2026-10-31T21:11:13.459Z [s3-sync] commitPush.shards 680ms dirty=11
2026-10-31T21:11:13.602Z [s3-sync] commitPush.localIndex 143ms
2026-10-31T21:11:13.606Z [s3-sync] commitPush 864ms shard-first changes=39
2026-10-31T21:11:13.606Z system │ Committed 39 file(s) to datastore index
2026-10-31T21:11:15.680Z [s3-sync] pullChanged.fastpath 547ms miss
2026-10-31T21:11:18.950Z system │ WebSocket token received via "bearer" from "127.0.0.1"
2026-10-31T21:11:18.957Z system │ WebSocket token received via "bearer" from "127.0.0.1"
2026-10-31T21:11:19.020Z [s3-sync] pullChanged.pullIndex 3410ms
2026-10-31T21:11:19.123Z system │ Authenticated token "adm" as "user:admin"
2026-10-31T21:11:19.123Z system │ Authenticated token "adm" as "user:admin"
2026-10-31T21:11:19.356Z system │ Acquiring lock for "datastore"
2026-10-31T21:11:19.423Z [s3-sync] pullChanged.remoteListing 191ms keys=9
2026-10-31T21:11:19.516Z [s3-sync] pullChanged.walk 93ms toPull=0
2026-10-31T21:11:19.517Z [s3-sync] pullChanged.download 0ms pulled=0
2026-10-31T21:11:19.627Z system │ Syncing behavior @repro/probe/b1188b4d-fdd7-4296-8d7e-5339bb4b8d83 from datastore...
2026-10-31T21:11:19.669Z [s3-sync] pullChanged.fastpath 40ms miss
2026-10-31T21:11:19.908Z [s3-sync] pullChanged.pullIndex 240ms
2026-10-31T21:11:27.908Z system │ WebSocket token received via "bearer" from "127.0.0.1"
2026-10-31T21:11:27.912Z system │ WebSocket token received via "bearer" from "127.0.0.1"
2026-10-31T21:11:27.984Z system │ Authenticated token "adm" as "user:admin"
2026-10-31T21:11:27.985Z system │ Authenticated token "adm" as "user:admin"
2026-10-31T21:11:43.769Z system │ Dispatch "09432c02-dace-40fe-9720-03081b43591a" on worker "w1" "failed" in 33461ms
2026-10-31T21:11:43.905Z system │ Runtime expression resolution: 0 total expressions, 0 runtime (vault/env)
2026-10-31T21:11:44.405Z system │ Runtime expression resolution: 0 total expressions, 0 runtime (vault/env)
2026-10-31T21:11:46.730Z system │ WebSocket token received via "bearer" from "127.0.0.1"
2026-10-31T21:11:46.736Z system │ WebSocket token received via "bearer" from "127.0.0.1"
2026-10-31T21:11:46.870Z system │ Authenticated token "adm" as "user:admin"
2026-10-31T21:11:46.871Z system │ Authenticated token "adm" as "user:admin"
2026-10-31T21:11:49.306Z [s3-sync] pullChanged.remoteListing 29395ms keys=11088
2026-10-31T21:11:50.793Z system │ WebSocket token received via "bearer" from "127.0.0.1"
2026-10-31T21:11:50.798Z system │ WebSocket token received via "bearer" from "127.0.0.1"
2026-10-31T21:11:50.890Z system │ Authenticated token "adm" as "user:admin"
2026-10-31T21:11:50.898Z system │ Authenticated token "adm" as "user:admin"
2026-10-31T21:11:55.009Z [s3-sync] pullChanged.walk 5700ms toPull=8
2026-10-31T21:11:55.352Z [s3-sync] pullChanged.download 345ms pulled=8
2026-10-31T21:11:55.439Z system │ Runtime expression resolution: 0 total expressions, 0 runtime (vault/env)
2026-10-31T21:11:55.485Z system │ Preparing push (uploading files)...
2026-10-31T21:11:55.485Z [s3-sync] preparePush.fastpath 0ms miss
2026-10-31T21:11:55.514Z system │ Dispatching "@repro/probe"."step" (step "place") to worker "w1" ["440c86b8-6e2b-4e19-8bcd-02059ab89205"]
2026-10-31T21:11:55.532Z system │ Runtime expression resolution: 0 total expressions, 0 runtime (vault/env)
2026-10-31T21:11:56.395Z system │ Handler slow: "capability.queryData" for worker "w1" took 42976ms (request "78a8186d-c354-429b-9627-e95beaf3fe4e")If it helps: from the log it looks as if the worker's queryData cannot be answered while a model pull holds the datastore lock, and that pull's listing covers the whole data/ prefix (11,088 keys) although it is for one model. We have not confirmed this in the code.
vcjdeboer commented 10/31/2026, 9:44:56 PM
Reproduction bundle (sh bundle.sh in an empty directory writes the 19 scripts; usage in the previous ripple, "Run it")
#!/bin/sh
# swamp-club #4102 reproduction bundle: sh this file in an empty directory; it writes the scripts (see README in the ripple).
set -e; mkdir -p image
cat > env.sh <<'R4102_EOF'
# swamp-club #4102 reproduction: common settings. Everything lives under $R/work (state) and $R/key (your swamp-club key).
R=${0:A:h}
J=$R/work
P=$J/probe # state: minio data, serve /w, worker cache, logs
BIN=$P/bin/swamp-a2f815f3 # swamp 20261030.160426.0-sha.a2f815f3 linux/aarch64 (sha256 5e695454...)
KEY=$R/key # club-key (SWAMP_API_KEY value, scopes serve:* datastore:*), optional signin (SWAMP_SIGNIN_TOKEN)
H=$R
IMG=r4102-base:1
MINIO_IMG=none # MinIO is a binary: $P/bin/minio (build-minio.sh)
NET=r4102-net; MN=r4102-minio; SV=r4102-serve; WK=r4102-worker
LBL=r4102=1
ts() { date -u +%H:%M:%S; }
ms() { python3 -c 'import time;print(int(time.time()*1000))'; }
R4102_EOF
cat > up.sh <<'R4102_EOF'
#!/bin/zsh
# Network, base image, MinIO (pinned release), serve container. Usage: up.sh
set -eu
. ${0:A:h}/env.sh
mkdir -p $P/bin $P/minio $P/w/secrets $P/w/client/config $P/wk-cache $P/logs
docker network inspect $NET >/dev/null 2>&1 || docker network create --label $LBL $NET >/dev/null
docker image inspect $IMG >/dev/null 2>&1 || docker build -q --label $LBL -t $IMG $H/image >/dev/null
# MinIO RELEASE.2025-09-07T16-13-09Z built from source (commit 07c3a429; docker hub, quay and dl.min.io no longer serve it)
[[ -f $P/minio.env ]] || { umask 077; printf 'MINIO_ROOT_USER=p%s\nMINIO_ROOT_PASSWORD=%s\n' $(openssl rand -hex 6) $(openssl rand -hex 20) > $P/minio.env; }
docker run -d --name $MN --label $LBL --network $NET --memory 1g --cpus 2 --cap-add NET_ADMIN --env-file $P/minio.env -v $P/bin/minio:/usr/local/bin/minio:ro -v $P/minio:/data $IMG minio server /data >/dev/null
. $P/minio.env
for i in {1..60}; do docker exec $MN curl -sf http://127.0.0.1:9000/minio/health/live >/dev/null 2>&1 && break; sleep 1; done
docker exec $MN curl -sf -X PUT --aws-sigv4 aws:amz:us-east-1:s3 --user "$MINIO_ROOT_USER:$MINIO_ROOT_PASSWORD" http://127.0.0.1:9000/repro >/dev/null
echo "$(ts) minio ready, bucket repro"
docker run -d --name $SV --label $LBL --network $NET --memory 8g --memory-swap 8g --cpus 2 \
-v $P/bin/swamp-a2f815f3:/usr/local/bin/swamp-real:ro -v $P/bin/swamp:/usr/local/bin/swamp:ro -v $KEY:/run/key:ro -v $H:/h:ro -v $P/w:/w \
-e HOME=/w -e SWAMP_HOME=/w/swamp-home -e SWAMP_CONFIG_DIR=/w/config \
-e SWAMP_API_KEY_FILE=/run/key/club-key \
-e AWS_ACCESS_KEY_ID=$MINIO_ROOT_USER -e AWS_SECRET_ACCESS_KEY=$MINIO_ROOT_PASSWORD -e AWS_REGION=us-east-1 \
-e SWAMP_NO_TELEMETRY=1 -e SWAMP_NO_INBOX_NOTICE=1 $IMG sleep infinity >/dev/null
echo "$(ts) serve container up"
R4102_EOF
cat > fetch-swamp.sh <<'R4102_EOF'
#!/bin/zsh
# swamp a2f815f3 linux/aarch64 + a wrapper that exports SWAMP_SIGNIN_TOKEN from $KEY/signin if present
set -eu
. ${0:A:h}/env.sh
mkdir -p $P/bin; V=20261030.160426.0-sha.a2f815f3
curl -sfL -o $P/bin/a.tar.gz https://artifacts.swamp-club.com/swamp/$V/binary/linux/aarch64/swamp-$V-binary-linux-aarch64.tar.gz
echo "a0990baeab11c5ac15a345ee58842e1c505d72855088e0711d9f668f0c3fdf13 $P/bin/a.tar.gz" | shasum -a 256 -c
tar xzf $P/bin/a.tar.gz -C $P/bin && rm $P/bin/a.tar.gz && mv $P/bin/swamp $BIN
echo "5e695454df72ac7f10ff6379b6e2556fd7c94339c62eb5927a1f23f1a5997ddd $BIN" | shasum -a 256 -c
printf '#!/bin/sh\n[ -r /run/key/signin ] && SWAMP_SIGNIN_TOKEN=$(cat /run/key/signin) && export SWAMP_SIGNIN_TOKEN\nexec /usr/local/bin/swamp-real "$@"\n' > $P/bin/swamp; chmod 755 $P/bin/swamp
R4102_EOF
cat > build-minio.sh <<'R4102_EOF'
#!/bin/zsh
# MinIO RELEASE.2025-09-07T16-13-09Z (the release datastore/benchmarks/docker-compose.yml pins) is no longer
# published as an image or binary, so build it from its tag (commit 07c3a429) into $P/bin/minio.
set -eu
. ${0:A:h}/env.sh
mkdir -p $P/bin $J/gobuild/{gomod,gocache}
[[ -d $J/gobuild/src/.git ]] || { git init -q $J/gobuild/src; git -C $J/gobuild/src fetch -q --depth 1 https://github.com/minio/minio refs/tags/RELEASE.2025-09-07T16-13-09Z; git -C $J/gobuild/src checkout -q FETCH_HEAD; }
docker run --rm -v $J/gobuild/src:/src -v $J/gobuild/gomod:/go/pkg/mod -v $J/gobuild/gocache:/root/.cache/go-build -v $P/bin:/out -w /src \
-e CGO_ENABLED=0 -e GOFLAGS=-buildvcs=false golang:1.24-bookworm sh -c 'go build -trimpath -tags kqueue -ldflags "-s -w -X github.com/minio/minio/cmd.ReleaseTag=RELEASE.2025-09-07T16-13-09Z -X github.com/minio/minio/cmd.CommitID=07c3a429bfed433e49018cb0f78a52145d4bedeb" -o /out/minio .'
chmod -R u+w $J/gobuild; rm -rf $J/gobuild
echo "minio built: $(shasum -a 256 $P/bin/minio)"
R4102_EOF
cat > setup-repo.sh <<'R4102_EOF'
#!/bin/zsh
# Repo in the serve container: @swamp/s3-datastore 2026.10.24.2 on MinIO, probe model, seed (~10k objects), workflows,
# logins, and a second repo copy with its own SWAMP_HOME (own cache) that plays a second writer.
set -eu
. ${0:A:h}/env.sh
mkdir -p $P/w/repo/extensions/models/probe $P/w/repo/workflows
docker exec $SV sh -c 'cd /w && swamp repo init /w/repo --tool none >/dev/null'
docker exec -w /w/repo $SV swamp extension pull @swamp/s3-datastore@2026.10.24.2 >/dev/null
docker exec -w /w/repo $SV sh -c 'grep -q 5b23db5329ba342bd74f983d2a6c10b78aeff4718b7cc0baac8640343288b7e2 extensions/models/upstream_extensions.json && echo "s3-datastore 2026.10.24.2 checksum ok"'
docker exec -w /w/repo $SV swamp datastore setup extension @swamp/s3-datastore --config '{"bucket":"repro","prefix":"lab","region":"us-east-1","endpoint":"http://r4102-minio:9000","forcePathStyle":true}' --skip-migration --json >/dev/null
cp $R/probe.ts $P/w/repo/extensions/models/probe/probe.ts
cp $R/serve.yaml $P/w/repo/.swamp/serve.yaml
MODELS=$(python3 $R/gen_seed.py $P/w/repo/workflows); python3 $R/gen_analysis.py $P/w/repo/workflows
docker exec -w /w/repo $SV sh -c "for m in $MODELS fw; do swamp model create @repro/probe \$m >/dev/null; done"
s=$(date +%s); docker exec -w /w/repo $SV swamp workflow run seed --json > $P/seed.json 2>&1 || true # long; we stopped it after 30 min (state-main ~161 of 1078 versions)
echo "seed: $(( $(date +%s)-s )) s"
docker exec $SV sh -c 'cp -a /w/repo /w/repo2 && mkdir -p /w/home2'
docker exec $SV sh /h/logins.sh
R4102_EOF
cat > logins.sh <<'R4102_EOF'
#!/bin/sh
# inside r4102-serve, before serve starts: throwaway logins + worker enrolment token
set -eu
cd /w/repo; umask 077
mint() { swamp access token mint $1 --principal $2 --duration 2d --json >/dev/null; swamp access token reveal $1 --yes --json | python3 -I -c 'import json,sys; sys.stdout.write(json.load(sys.stdin)["token"])' > /w/secrets/$1; }
mint adm user:admin; mint testa user:testa; mint testb user:testb; mint wlogin worker:w1
swamp worker token create w1 --duration 2d --max-enrollments 1 --json | python3 -I -c 'import json,sys; sys.stdout.write(json.load(sys.stdin)["token"])' > /w/secrets/enrol
ls /w/secrets
R4102_EOF
cat > serve-start.sh <<'R4102_EOF'
#!/bin/zsh
# (Re)start swamp serve in r4102-serve at debug with the s3 sync trace (st-166's unit env). Usage: serve-start.sh <n>
set -u
. ${0:A:h}/env.sh
old=$(docker exec $SV cat /w/serve.pid 2>/dev/null)
if [[ -n $old ]] && docker exec $SV kill -0 $old 2>/dev/null; then docker exec $SV kill -TERM $old; for i in {1..60}; do docker exec $SV kill -0 $old 2>/dev/null || break; sleep 1; done; fi
docker exec -d -e SWAMP_LOG_LEVEL=debug -e SWAMP_S3_SYNC_TRACE=1 $SV sh -c "swamp serve --repo-dir /w/repo 2>&1 | python3 -u -I /h/stamp.py >> /w/serve-$1.log"
for i in {1..20}; do p=$(docker exec $SV pgrep -f 'swamp-real serve' | head -1); [[ -n $p ]] && break; sleep 0.5; done
docker exec $SV sh -c "echo $p > /w/serve.pid"
docker exec -d $SV sh -c 'pkill socat; ip=$(hostname -i | cut -d" " -f1); exec socat TCP-LISTEN:9190,bind=$ip,reuseaddr,fork TCP:127.0.0.1:9190'
for i in {1..180}; do docker exec $SV curl -sf -m 2 http://127.0.0.1:9190/ready >/dev/null 2>&1 && { echo "$(ts) serve $1 ready, pid $(docker exec $SV cat /w/serve.pid)"; exit 0; }; sleep 1; done
echo "$(ts) serve $1 NOT ready"; docker exec $SV tail -20 /w/serve-$1.log; exit 1
R4102_EOF
cat > worker-start.sh <<'R4102_EOF'
#!/bin/zsh
# the one remote worker (capacity 1)
set -u
. ${0:A:h}/env.sh
docker rm -f $WK >/dev/null 2>&1
SIP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' $SV)
docker run -d --name $WK --add-host host.docker.internal:$SIP --label $LBL --network $NET --memory 2g --cpus 2 \
-v $P/bin/swamp-a2f815f3:/usr/local/bin/swamp-real:ro -v $P/bin/swamp:/usr/local/bin/swamp:ro -v $KEY:/run/key:ro -v $P/w/secrets:/s:ro -v $P/wk-cache:/c \
-e HOME=/c/home -e SWAMP_HOME=/c/home -e SWAMP_CONFIG_DIR=/c/config -e SWAMP_API_KEY_FILE=/run/key/club-key \
-e SWAMP_LOG_LEVEL=debug -e SWAMP_NO_TELEMETRY=1 -e SWAMP_NO_INBOX_NOTICE=1 $IMG \
swamp worker connect ws://host.docker.internal:9190 --token-file /s/enrol --server-token-file /s/wlogin --label pool=probe --cache-dir /c/worker >/dev/null
sleep 12; docker logs --tail 5 $WK 2>&1
R4102_EOF
cat > sampler.sh <<'R4102_EOF'
#!/bin/zsh
# every 5 s: serve pid, VmRSS, VmHWM since the previous sample (reset via clear_refs), cgroup current/peak,
# worker RSS, established :9190 connections, stage label. CSV $P/samples.csv. Stop: touch $P/stop-sampler
. ${0:A:h}/env.sh
[[ -f $P/samples.csv ]] || echo "epoch_ms,utc,pid,rss_mb,win_peak_rss_mb,cg_mb,cg_peak_mb,wk_cg_mb,ws_estab,stage" > $P/samples.csv
rm -f $P/stop-sampler
while [[ ! -f $P/stop-sampler ]]; do
l=$(docker exec $SV sh -c 'p=$(cat /w/serve.pid 2>/dev/null); r=0; h=0; if [ -n "$p" ] && [ -r /proc/$p/status ]; then r=$(awk "/VmRSS/{print int(\$2/1024)}" /proc/$p/status); h=$(awk "/VmHWM/{print int(\$2/1024)}" /proc/$p/status); echo 5 > /proc/$p/clear_refs; else p=dead; fi; c=$(( $(cat /sys/fs/cgroup/memory.current) / 1048576 )); k=$(( $(cat /sys/fs/cgroup/memory.peak 2>/dev/null || echo 0) / 1048576 )); e=$(ss -Htn state established "( sport = :9190 )" | wc -l); echo "$p,$r,$h,$c,$k,$e"' 2>/dev/null)
w=$(docker exec $WK sh -c 'echo $(( $(cat /sys/fs/cgroup/memory.current) / 1048576 ))' 2>/dev/null)
IFS=, read p r h c k e <<< "$l"
echo "$(ms),$(ts),$p,$r,$h,$c,$k,${w:-0},$e,$(cat $P/stage 2>/dev/null)" >> $P/samples.csv
sleep 5
done
R4102_EOF
cat > watchdog.sh <<'R4102_EOF'
#!/bin/zsh
# Disk watchdog: every 15 s, du of work/ + images; >= 8 GB -> STOP-disk, stops the load.
. ${0:A:h}/env.sh
LIMIT_KB=$((8*1024*1024))
[[ -f $P/disk.csv ]] || echo "utc,jobtmp_kb,images_kb,total_kb" > $P/disk.csv
while [[ ! -f $P/stop-watchdog ]]; do
d=$(du -sk $J 2>/dev/null | awk '{print $1}')
i=0; for im in $IMG $MINIO_IMG golang:1.24-bookworm; do s=$(docker image inspect -f '{{.Size}}' $im 2>/dev/null) && i=$((i + s/1024)); done
t=$((d+i)); echo "$(ts),$d,$i,$t" >> $P/disk.csv
if (( t >= LIMIT_KB )) && [[ ! -f $P/STOP-disk ]]; then
echo "$(ts) total ${t} KB" > $P/STOP-disk
pkill -f 'driver.py' ; pkill -f 'cdp.js'
fi
sleep 15
done
R4102_EOF
cat > run-case.sh <<'R4102_EOF'
#!/bin/zsh
# One case = two concurrent analysis runs (testa, testb) with the client polling pattern. Usage:
# run-case.sh <name> <latency ms: 0|30> <foreign: on|off>
# A name must start with c or d (the driver's two-run stage). Failing case: d-fail 30 on; comparisons: c-nolat 0 on, d-noforeign 30 off.
set -u
. ${0:A:h}/env.sh
docker exec $MN tc qdisc del dev eth0 root 2>/dev/null
(( $2 > 0 )) && docker exec $MN tc qdisc add dev eth0 root netem delay ${2}ms
f=(); [[ $3 == off ]] && f=(--no-foreign)
python3 -I $R/driver.py $1 $f > $P/driver-$1.log 2>&1
grep -E "^$1,(discovered|runs-ended)" $P/requests.csv
grep -E 'Handler slow: "capability.queryData"' $P/w/serve-1.log | tail -8 | sed -E 's/.*took ([0-9]+)ms.*/queryData \1 ms/'
R4102_EOF
cat > serve.yaml <<'R4102_EOF'
# serve config: token auth, remote-only, one worker reaching serve through a socat relay
host: 127.0.0.1
port: 9190
schedule: false
grant-reload: manual
remote-only: true
dispatch-env-allow: []
verify-on-enroll: true
max-runs-per-principal: 2
max-run-duration: 2h
dashboard: false
enable-internal-api: false
auto-resume: false
auth:
mode: token
admins:
- "user:admin"
- "user:testa"
- "user:testb"
R4102_EOF
cat > stamp.py <<'R4102_EOF'
import sys, datetime
for line in sys.stdin:
sys.stdout.write(datetime.datetime.now(datetime.timezone.utc).strftime("%Y-%m-%dT%H:%M:%S.%f")[:-3] + "Z " + line); sys.stdout.flush()
R4102_EOF
cat > gen_seed.py <<'R4102_EOF'
"""Seed workflow: the lab's record classes from the upgrade.log retention plan (2026-10-31 14:13).
Each lane is one model (own lock), sequential; lanes run in parallel. Version k of a name = forEach item k."""
import sys, uuid
d = sys.argv[1]
# lane: (model, iterations, names, bytes, files, fileBytes)
LANES = [
("wstate", 1078, 1, 900, 0, 0), # swamp/worker state-main: 1078 versions on one name
("fleet", 106, 1, 1500, 0, 0), # swamp/fleet-probe probe-result
("p", 1, 100, 2000, 0, 0), # P: 100 names / 100 versions
("t", 1, 637, 1500, 0, 0), # T: 637 one-version names
("u3", 1, 214, 3000, 0, 0), ("u3b", 1, 59, 3000, 0, 0), # U3: 273 versions in 214(+59) names
("r", 14, 58, 4000, 0, 0), # R: ~834 versions in 58 names
("wa", 1, 114, 12000, 0, 0), ("wb", 1, 114, 12000, 0, 0), # session-write 228
("ea", 1, 115, 20000, 25, 300000), ("eb", 1, 115, 20000, 25, 300000), # session-execute 280 (230 json + 50 files)
("va", 1, 88, 2500, 0, 0), ("vb", 1, 88, 2500, 0, 0), # session-witness 176
]
steps = []
chunks = []
for m, it, n, b, f, fb in LANES:
for c, lo in enumerate(range(0, it, 900)): # CEL maxListElements 1000
chunks.append((m, c, list(range(lo, min(it, lo + 900))), n, b, f, fb))
for m, c, rng, n, b, f, fb in chunks:
items = "[" + ",".join(str(k) for k in rng) + "]"
dep = " dependsOn: []\n"
steps.append(f""" - name: {m}-c{c}-${{{{ self.k }}}}
forEach: {{ item: k, in: "${{{{ {items} }}}}" }}
concurrency: 1
task:
type: model_method
modelIdOrName: {m}
methodName: seed
inputs:
prefix: {m}
names: {n}
versions: 1
bytes: {b}
files: {f}
fileBytes: {fb}
{dep} weight: 0
allowFailure: false
""")
i = str(uuid.uuid5(uuid.NAMESPACE_URL, "r4102/seed"))
open(f"{d}/workflow-{i}.yaml", "w").write(f"id: {i}\nname: seed\ndescription: seed the lab's record classes (local)\njobs:\n - name: main\n steps:\n" + "".join(steps))
print(" ".join(l[0] for l in LANES))
R4102_EOF
cat > gen_analysis.py <<'R4102_EOF'
"""analysis: 9 steps on the worker shaped like a real analysis workflow: per run a ~630 KB octet-stream input file (also a base64 workflow input), 2 octet-stream outputs (250 + 200 KB), JSON resources; steps 2-9 call queryData on the member model.
Per run: input file ~630 KB (octet-stream, from the base64 workflow input), 2 Parquet-like
octet-stream files (250 + 200 KB), JSON resources ~150 KB; queryData like the batched catalog read."""
import sys, uuid
d = sys.argv[1]
M = "${{ inputs.m }}"
WR = "${{ run.id.lowerAscii() }}"
def q(model): return f"modelName == \\\"{model}{M}\\\" && specName == \\\"rec\\\""
STEPS = [ # name, model, query-model, json, jsonBytes, files, fileBytes, input
("intake", "e", "", 1, 1000, 0, 0, True),
("fill", "w", "w", 2, 20000, 0, 0, False),
("place", "e", "e", 1, 5000, 0, 0, False),
("execute-parse", "e", "w", 1, 30000, 1, 250000, False),
("seal-parse", "v", "e", 2, 2000, 0, 0, False),
("xf-fill", "w", "", 1, 20000, 0, 0, False),
("execute-xf", "e", "w", 1, 30000, 1, 200000, False),
("seal-xf", "v", "e", 2, 2000, 0, 0, False),
("verify", "v", "v", 2, 1000, 0, 0, False),
]
body = "jobs:\n - name: main\n steps:\n"
prev = None
for name, m, qm, j, jb, f, fb, inp in STEPS:
body += f""" - name: {name}
labels: {{ pool: probe }}
task:
type: model_method
modelIdOrName: "{m}{M}"
methodName: step
inputs:
label: {name}
wr: "{WR}"
query: "{q(qm) if qm else ''}"
json: {j}
jsonBytes: {jb}
files: {f}
fileBytes: {fb}
"""
if inp: body += " inputB64: ${{ inputs.instrumentFile }}\n"
body += (f" dependsOn:\n - step: {prev}\n condition: {{ type: succeeded }}\n" if prev else " dependsOn: []\n")
body += " weight: 0\n allowFailure: false\n"
prev = name
i = str(uuid.uuid5(uuid.NAMESPACE_URL, "r4102/analysis"))
open(f"{d}/workflow-{i}.yaml", "w").write(f"""id: {i}
name: analysis
description: analysis-shaped probe run
inputs:
properties:
m: {{ type: string }}
instrumentFile: {{ type: string }}
required: ["m", "instrumentFile"]
{body}""")
R4102_EOF
cat > driver.py <<'R4102_EOF'
"""Load driver for the swamp-club #4102 reproduction.
Usage: python3 -I r4102-driver.py <stage: a|b|c|d> [--no-foreign] [--no-observer] [--no-query]
Every request -> requests.csv: stage,op,identity,start_ms,end_ms,exit,bytes,note"""
import subprocess, sys, time, json, threading, os, random, base64, csv
P = os.environ.get("REPRO_STATE") or os.path.join(os.path.dirname(os.path.abspath(__file__)), "work", "probe")
SV = "r4102-serve"
args = sys.argv[1:]; STAGE = args[0]
FOREIGN = "--no-foreign" not in args; OBSERVER = "--no-observer" not in args
lock = threading.Lock(); stop = threading.Event()
REQ = os.path.join(P, "requests.csv")
if not os.path.exists(REQ):
open(REQ, "w").write("stage,op,identity,start_ms,end_ms,exit,bytes,note\n")
def now(): return int(time.time() * 1000)
def stopped(): return os.path.exists(os.path.join(P, "STOP-disk"))
def log(op, ident, t0, t1, rc, nbytes, note=""):
with lock:
with open(REQ, "a", newline="") as f: csv.writer(f).writerow([STAGE, op, ident, t0, t1, rc, nbytes, note])
def cli(ident, *a, stdin=None, op=None, note=""):
if stopped(): raise SystemExit("STOP-disk")
cmd = ["docker", "exec"] + (["-i"] if stdin is not None else []) + ["-e", "SWAMP_HOME=/w/client", "-e", "SWAMP_CONFIG_DIR=/w/client/config", SV,
"swamp", *a, "--server", "ws://127.0.0.1:9190", "--token-file", f"/w/secrets/{ident}"]
t0 = now(); r = subprocess.run(cmd, input=stdin, capture_output=True); t1 = now()
log(op or a[0] + " " + a[1], ident, t0, t1, r.returncode, len(r.stdout), note)
return r
def foreign(n):
"""second writer: its own repo copy + SWAMP_HOME (own cache), one small commit to the bucket"""
t0 = now()
r = subprocess.run(["docker", "exec", "-w", "/w/repo2", "-e", "SWAMP_HOME=/w/home2", SV, "swamp", "model", "method", "run", "fw", "seed",
"--input", f"prefix=fw{n}", "--input", "names=1", "--input", "versions=1", "--input", "bytes=500", "--json"], capture_output=True)
log("foreign-commit", "second-writer", t0, now(), r.returncode, len(r.stdout), f"fw{n}")
def jparse(b):
t = b.decode(errors="replace"); i = t.find("{")
try: return json.loads(t[i:]) if i >= 0 else {}
except Exception: return {}
INSTR = None
def instrument():
global INSTR
if INSTR is None:
INSTR = base64.b64encode(os.urandom(630_000)).decode()
return INSTR
def start_run(ident, m, out):
inp = json.dumps({"m": m, "instrumentFile": instrument()}).encode()
r = cli(ident, "workflow", "run", "analysis", "--stdin", "--json", stdin=inp, op="workflow run (blocking)")
out[ident] = (r.returncode, r.stdout[-400:])
def find_run(ident, m, known):
for _ in range(90):
r = cli("adm", "workflow", "history", "search", "--workflow", "analysis", "--json", op="history search", note="discovery")
d = jparse(r.stdout)
for x in d.get("results") or d.get("runs") or []:
rid = x.get("runId") or x.get("id")
inp = x.get("inputs") or {}
if rid and rid not in known and isinstance(inp, dict) and inp.get("m") == m:
return rid
time.sleep(2)
return None
def observe(rid, ident, t_end_flag):
"""client polling pattern: 2 s history get until execute-parse is seen running (restart wait), then 10 s history get; search every ~4 s"""
phase = "restart-wait"; last_search = 0
while not stop.is_set() and not stopped():
r = cli("adm", "workflow", "history", "get", rid, "--json", op="history get", note=phase)
d = jparse(r.stdout); st = d.get("status") or (d.get("run") or {}).get("status")
steps = {s.get("name"): s.get("status") for j in (d.get("jobs") or []) for s in (j.get("steps") or [])}
if phase == "restart-wait" and (steps.get("execute-parse") in ("running", "succeeded") or steps.get("seal-parse")): phase = "observe"
if time.time() - last_search > 4:
cli("adm", "workflow", "history", "search", "--workflow", "analysis", "--json", op="history search", note="observe"); last_search = time.time()
if st in ("succeeded", "failed", "interrupted", "cancelled"): return st
time.sleep(2 if phase == "restart-wait" else 10)
return "stopped"
def collect(rid):
"""evidence collection: query by tag over all versions, then 3 data gets in flight"""
r = cli("adm", "data", "query", f'tags.workflowRunId == "{rid.lower()}" && version >= 0', "--select", "{\"m\": modelName, \"n\": name, \"v\": version, \"s\": size}", "--json", op="data query by-tag")
d = jparse(r.stdout); recs = d.get("results") or []
sizes = sorted(((x.get("s") or 0) for x in recs if isinstance(x, dict)), reverse=True)
log("collect-summary", "adm", now(), now(), 0, sum(sizes), f"records={len(recs)} top={sizes[:3]}")
todo = [(x.get("m"), x.get("n")) for x in recs if isinstance(x, dict) and x.get("m")]
def worker():
while todo and not stopped():
with lock:
if not todo: return
m, n = todo.pop()
cli("adm", "data", "get", m, n, "--json", op="data get --json")
ts = [threading.Thread(target=worker) for _ in range(3)]
[t.start() for t in ts]; [t.join() for t in ts]
def stage_label(s):
open(os.path.join(P, "stage"), "w").write(s); log("stage", "-", now(), now(), 0, 0, s)
def runs(idents):
known = set()
r = cli("adm", "workflow", "history", "search", "--workflow", "analysis", "--json", op="history search", note="before")
for x in (jparse(r.stdout).get("results") or []): known.add(x.get("runId") or x.get("id"))
out = {}; th = []
for ident, m in idents:
t = threading.Thread(target=start_run, args=(ident, m, out)); t.start(); th.append(t)
rids = {ident: find_run(ident, m, known) for ident, m in idents}
log("discovered", "-", now(), now(), 0, 0, json.dumps(rids))
obs = []
if OBSERVER:
res = {}
for ident, rid in rids.items():
if rid:
t = threading.Thread(target=lambda i=ident, r=rid: res.__setitem__(i, observe(r, i, None))); t.start(); obs.append(t)
fc = 0
while any(t.is_alive() for t in th) and not stopped():
time.sleep(5)
if FOREIGN and fc < 3 and int(time.time()) % 60 < 5: fc += 1; foreign(f"{STAGE}{fc}")
[t.join() for t in th]; [t.join() for t in obs]
log("runs-ended", "-", now(), now(), 0, 0, json.dumps({k: v[0] for k, v in out.items()}))
for ident, rid in rids.items():
if rid and OBSERVER: collect(rid)
return rids
if STAGE == "a":
stage_label("a-idle")
t0 = time.time()
for at in (300, 600):
while time.time() - t0 < at and not stopped(): time.sleep(5)
if FOREIGN: foreign(f"a{at}")
while time.time() - t0 < 900 and not stopped(): time.sleep(5)
elif STAGE == "b":
stage_label("b-one-run"); print(runs([("testa", "a")]))
elif STAGE[0] in ("c", "d"):
stage_label(STAGE + "-two-runs"); print(runs([("testa", "a"), ("testb", "b")]))
stage_label(STAGE + "-end")
R4102_EOF
cat > timeline.py <<'R4102_EOF'
"""Join samples.csv (5 s windows, per-window peak RSS), requests.csv, serve's [s3-sync] and Heap: lines
into one timeline; print the top windows with every operation that overlapped them.
Usage: python3 -I timeline.py <probe dir> <serve log> [topN]"""
import csv, re, sys, datetime as dt
P, LOG = sys.argv[1], sys.argv[2]; TOP = int(sys.argv[3]) if len(sys.argv) > 3 else 10
def iso_ms(s):
return int(dt.datetime.fromisoformat(s.replace("Z", "+00:00")).timestamp() * 1000)
samples = [r for r in csv.DictReader(open(f"{P}/samples.csv")) if r["pid"] not in ("", "dead")]
reqs = list(csv.DictReader(open(f"{P}/requests.csv")))
ops = [(int(r["start_ms"]), int(r["end_ms"]), f'{r["op"]}[{r["identity"]}]{(" " + r["note"]) if r["note"] else ""}') for r in reqs if r["op"] not in ("stage",)]
sync, heap, hstart, handlers = [], [], {}, []
ts_re = re.compile(r"^(\d{4}-\d\d-\d\dT[\d:.]+Z)")
for line in open(LOG, errors="replace"):
m = ts_re.match(line)
if not m: continue
t = iso_ms(m.group(1))
s = re.search(r"\[s3-sync\] (\S+) (\d+)ms(.*)", line)
if s: sync.append((t - int(s.group(2)), t, f"sync {s.group(1)}{s.group(3)[:40]}")); continue
hs = re.search(r'Handler start: "([^"]+)".*\(request "([^"]+)"\)', line)
if hs: hstart[hs.group(2)] = (t, hs.group(1)); continue
hd = re.search(r'Handler (?:done|slow): "([^"]+)".*\(request "([^"]+)"\)', line)
if hd and hd.group(2) in hstart:
t0, name = hstart.pop(hd.group(2)); handlers.append((t0, t, f"handler {name}")); continue
h = re.search(r"Heap: rss=(\d+)MB heapUsed=(\d+)MB heapTotal=(\d+)MB external=(\d+)MB", line)
if h: heap.append((t, *map(int, h.groups())))
allops = ops + sync + handlers + [(t0, 10**15, f'handler {n} (no done)') for t0, n in hstart.values()]
rows = []
for i, s in enumerate(samples):
t1 = int(s["epoch_ms"]); t0 = int(samples[i - 1]["epoch_ms"]) if i else t1 - 5000
over = sorted({o[2] for o in allops if o[0] <= t1 and o[1] >= t0})
rows.append((int(s["win_peak_rss_mb"] or 0), s["utc"], s["stage"], int(s["rss_mb"] or 0), over))
print(f"samples {len(samples)}, requests {len(ops)}, sync phases {len(sync)}, handlers {len(handlers)}, heap lines {len(heap)}")
print("max heapUsed", max((h[2] for h in heap), default=None), "MB; max window peak RSS", max((r[0] for r in rows), default=None), "MB")
for r in sorted(rows, reverse=True)[:TOP]:
print(f"\n{r[1]} stage={r[2]} winPeakRSS={r[0]} MB rss={r[3]} MB overlapping {len(r[4])}:")
for o in r[4][:25]: print(" ", o)
R4102_EOF
cat > probe.ts <<'R4102_EOF'
// Probe model for the swamp-club #4102 reproduction (copy into <repo>/extensions/models/probe/probe.ts).
import { z } from "npm:zod@4";
const Any = z.record(z.string(), z.unknown());
const hex = (n: number) => {
const b = new Uint8Array(Math.ceil(n / 2));
for (let i = 0; i < b.length; i += 65536) crypto.getRandomValues(b.subarray(i, Math.min(i + 65536, b.length)));
return Array.from(b, (x) => x.toString(16).padStart(2, "0")).join("").slice(0, n);
};
const bytes = (n: number) => {
const b = new Uint8Array(n);
for (let i = 0; i < n; i += 65536) crypto.getRandomValues(b.subarray(i, Math.min(i + 65536, n)));
return b;
};
// a JSON "table" of about n bytes (rows of numbers and short strings, like an analysis output)
const table = (n: number) => {
const rows: unknown[] = [];
let size = 0;
while (size < n) { const r = { well: hex(4), t: Math.random() * 1000, ocr: Math.random() * 300, ecar: Math.random() * 100, note: hex(40) }; rows.push(r); size += 110; }
return rows;
};
type Ctx = {
writeResource: (spec: string, name: string, data: Record<string, unknown>, o?: { tags?: Record<string, string> }) => Promise<unknown>;
createFileWriter: (spec: string, name: string, o?: { tags?: Record<string, string> }) => { writeAll: (b: Uint8Array) => Promise<unknown> };
queryData?: (predicate: string, select?: string) => Promise<unknown[]>;
logger: { info: (m: string, p?: Record<string, unknown>) => void };
};
const SeedArgs = z.object({
prefix: z.string(), names: z.number().int().min(1), versions: z.number().int().min(1),
bytes: z.number().int().min(0), files: z.number().int().min(0).default(0), fileBytes: z.number().int().min(0).default(0),
wr: z.string().default(""),
});
const StepArgs = z.object({
label: z.string(), wr: z.string(),
query: z.string().default(""), // queryData predicate run first (worker capability), empty = none
json: z.number().int().min(0).default(0), jsonBytes: z.number().int().min(0).default(0),
files: z.number().int().min(0).default(0), fileBytes: z.number().int().min(0).default(0),
inputB64: z.string().default(""), // input file (base64) -> one octet-stream file record
});
export const model = {
type: "@repro/probe",
version: "2026.10.31.1",
resources: {
rec: { description: "probe JSON resource", schema: Any, lifetime: "infinite", garbageCollection: 5000 },
},
files: {
blob: { description: "probe binary file record", contentType: "application/octet-stream", lifetime: "infinite", garbageCollection: 5000 },
},
methods: {
seed: {
description: "write names x versions JSON records (and file records) of the given sizes",
arguments: SeedArgs,
execute: async (a: z.input<typeof SeedArgs>, ctx: Ctx) => {
const args = SeedArgs.parse(a); const handles: unknown[] = [];
const tags: Record<string, string> = args.wr ? { workflowRunId: args.wr, probe: args.prefix } : { probe: args.prefix };
for (let v = 0; v < args.versions; v++) {
for (let n = 0; n < args.names; n++) {
handles.push(await ctx.writeResource("rec", `${args.prefix}-${n}`, { v, n, rows: table(args.bytes) }, { tags }));
}
}
for (let f = 0; f < args.files; f++) handles.push(await ctx.createFileWriter("blob", `${args.prefix}-f${f}`, { tags }).writeAll(bytes(args.fileBytes)));
ctx.logger.info("seed {p}: {n} records", { p: args.prefix, n: handles.length });
return { dataHandles: handles };
},
},
step: {
description: "one analysis-shaped step: optional queryData, then file and JSON writes of given sizes",
arguments: StepArgs,
execute: async (a: z.input<typeof StepArgs>, ctx: Ctx) => {
const args = StepArgs.parse(a); const handles: unknown[] = [];
const tags = { workflowRunId: args.wr };
if (args.query && ctx.queryData) {
const t0 = Date.now();
const rows = await ctx.queryData(args.query);
const b = JSON.stringify(rows).length;
ctx.logger.info("PROBE queryData {label} start={t0} end={t1} rows={rows} bytes={b}", { label: args.label, t0, t1: Date.now(), rows: rows.length, b });
}
if (args.inputB64) {
const bin = Uint8Array.from(atob(args.inputB64), (c) => c.charCodeAt(0));
handles.push(await ctx.createFileWriter("blob", `${args.wr}-input`, { tags }).writeAll(bin));
}
for (let f = 0; f < args.files; f++) handles.push(await ctx.createFileWriter("blob", `${args.wr}-${args.label}-f${f}`, { tags }).writeAll(bytes(args.fileBytes)));
for (let j = 0; j < args.json; j++) handles.push(await ctx.writeResource("rec", `${args.wr}-${args.label}-j${j}`, { rows: table(args.jsonBytes) }, { tags }));
ctx.logger.info("PROBE step {label} wrote {n}", { label: args.label, n: handles.length });
return { dataHandles: handles };
},
},
},
};
R4102_EOF
cat > image/Dockerfile <<'R4102_EOF'
# Probe base image for swamp-club #4224 (throwaway; the swamp binary is bind-mounted, not baked)
FROM debian:bookworm-slim
RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates procps python3 iproute2 curl socat && rm -rf /var/lib/apt/lists/*
R4102_EOF
chmod +x *.sh; echo "written: $(ls | wc -l) files"vcjdeboer commented 11/1/2026, 9:17:12 AM
Where this stands (2026-11-01). A short summary, since the thread above got long. The traces and the reproduction scripts from the earlier ripples are now files in one gist: https://gist.github.com/vcjdeboer/a1c217deb645a94942399b6f3418f8a7
- Production. The 30 s
queryDatatimeout happened in two sittings on f96841b5 (42.6 s and 45.0 s). It has not recurred in the two sittings since on a2f815f3; in the traced one the slowest call was 7.9 s. - Reproduced off production (a2f815f3, MinIO, one worker). The worker's
queryDatais answered only after a per-model pull (Syncing behavior … from datastore) has listed and walked the wholedata/prefix: 11,088 keys for one model. So the worker gives up at 30 s, and serve answers at 43 s. - One correction. Our reproduction needed 30 ms S3 latency plus a second writer committing to the bucket. Production has no such steady second writer, so that trigger explains the reproduction, not production. The trace does show production on the slow path without one:
pullChanged.fastpathmissed 306 of 524 times, and listings of ~35k keys took 2.5–3 s, with walks of 4–6 s. - Not this issue: serve heap exhaustion in the same sittings is #4224 (large workflow inputs in
.runs-index.json). We checked whether it explains the slowqueryData. In its comparison (filesystem datastore) serve reached 4–5 GB, yet the 21queryDatacalls that ran all took under 0.6 s. So we don't think #4224 causes this, though a serve near its heap limit may add to it.
zack commented 11/2/2026, 4:38:18 PM
Thanks @vcjdeboer for reporting this! We shipped: Stop one serve falling permanently off the pull fast path and invalidating its catalog on its own writes. Overlapping pushes on one S3 or GCS sync service falsely report that another writer committed, so the sidecar commitSeq falls behind and scoped pulls never catch up. Each slow model-scoped pull then walks the whole in-memory index, overwrites unpushed files of other models, and reports downloads, which makes core invalidate the catalog so the next worker query waits for a full rebuild. Fix the false disarm, scope model pulls to their model as the README promises, and keep marked files in subdir-scoped pulls. Corrects the triage note that placed the whole fix in core: the trigger and overwrite are in these extensions, while the fixed 30 s RPC limit and full rebuild stay in core and are filed separately. Reproduced on 2026.10.01.1, 2026.10.24.2 and HEAD; not a regression. May not fully explain the 45 s the reporter measured.. The fix has been merged and a release is on its way. We appreciate your contribution to swamp.
Sign in to post a ripple.