fix(gateway): recover bounded pacing debt
Verify Data Plane / gateway (push) Failing after 3m59s

This commit is contained in:
sechmachine
2026-08-09 20:54:15 +07:00
parent 55afea72a1
commit 122080ab34
8 changed files with 334 additions and 3 deletions
@@ -12,6 +12,10 @@ The later Darwin non-sustained pre-CI invocation was not green and was not retri
Private Linux run 125 at the frozen v8 harness head is retained as failed evidence. Its exact 33-datagram gap between successful fixture writes and production `MediaIngress` equaled the Linux socket's 33 measured kernel UDP drops. The complete-frame queue, fair pacer, QUIC fragmentation, and public decoder were downstream and did not account for the loss.
The one authorized v8 Section 7 invocation at production candidate `55afea72a1487fa071501615d806e68efc0a436b` was consumed and failed. Its directory `gateway-rc10-55afea7` is retained byte-for-byte with two partial processing files and no manifest. The failure occurred at payload sequence 16801 after 16,834 provider frames had been recovered and enqueued; the provider queue reached 15 entries and dropped one valid frame while source-write and ingress accounting remained balanced at the diagnostic boundary. This attempt is failed evidence and is not eligible for retry or relabeling.
The production fair pacer previously limited instantaneous recovery to 5 ms by moving an overdue flow's schedule to `now-5ms`, but silently discarded every additional valid scheduling interval. Repeated host stalls therefore accumulated complete frames in the existing provider queue until its 250 ms residence horizon correctly expired one. The repair keeps the 5 ms instantaneous ceiling, carries only the remaining debt up to that existing horizon, and shortens later nominal intervals by at most one twenty-first. That 20/21 interval is exactly 5% above nominal rate; once the debt reaches zero, the flow returns to its unchanged nominal interval. Per-flow debt and the shared nominal fair-share calculation preserve the existing eight-flow fairness and rolling aggregate cap through the existing 25% and 50% capacity changes.
The ingress repair follows reviewed behavior rather than copying implementation source:
- Apollo `adc5c5a0bd80831ce495434bb16aee2cd4175fb8`, GPL-3.0, `src/stream.cpp:1463-1474,1573-1627`, supplies the 80%-of-1-Gbps raw-block pacing, 64-KiB/64-packet batch cap, and cross-frame send schedule used by the fixture.
@@ -37,9 +41,11 @@ The native provider therefore requests `2,048 * 1,072 = 2,195,456` bytes with `S
- Carry a deterministic frame index/pattern only in the generated payload bytes; no codec semantics are claimed.
- Keep the existing path/impairment/resource driver and change its unit from datagram payload to complete frame.
- Keep video decrypt/FEC single-threaded; only the bounded connected-socket drain is separated so crypto stalls cannot become unexplained kernel loss.
- Preserve valid scheduling debt after bounded host stalls instead of converting it into provider-queue residence; repay it within the existing fair pacer without a new queue, interface, or configured headroom.
## Risks / Trade-offs
- [Keyframes can exceed queue budget] → use the reviewed 1 MiB frame ceiling and production byte-bound queue.
- [Short smoke windows have rounding effects] → assert exact generated totals and report measured duration separately from normative ten-minute gates.
- [A stalled video processor exhausts the user-space pool] → keep draining into one fixed scratch buffer and attribute accepted-size overflow to existing ingress/drop counters rather than kernel loss or unbounded allocation.
- [Debt repayment creates a burst or aggregate oversubscription] → retain the 5 ms instantaneous ceiling and limit repayment to a 20/21 nominal interval per flow, with every rolling five-second aggregate window bounded to 105%.