fix(gateway): recover bounded pacing debt

openspec/changes/truthful-fixed-frame-qualification/design.md

@@ -12,6 +12,10 @@ The later Darwin non-sustained pre-CI invocation was not green and was not retri
 
 Private Linux run 125 at the frozen v8 harness head is retained as failed evidence. Its exact 33-datagram gap between successful fixture writes and production `MediaIngress` equaled the Linux socket's 33 measured kernel UDP drops. The complete-frame queue, fair pacer, QUIC fragmentation, and public decoder were downstream and did not account for the loss.
 
+The one authorized v8 Section 7 invocation at production candidate `55afea72a1487fa071501615d806e68efc0a436b` was consumed and failed. Its directory `gateway-rc10-55afea7` is retained byte-for-byte with two partial processing files and no manifest. The failure occurred at payload sequence 16801 after 16,834 provider frames had been recovered and enqueued; the provider queue reached 15 entries and dropped one valid frame while source-write and ingress accounting remained balanced at the diagnostic boundary. This attempt is failed evidence and is not eligible for retry or relabeling.
+
+The production fair pacer previously limited instantaneous recovery to 5 ms by moving an overdue flow's schedule to `now-5ms`, but silently discarded every additional valid scheduling interval. Repeated host stalls therefore accumulated complete frames in the existing provider queue until its 250 ms residence horizon correctly expired one. The repair keeps the 5 ms instantaneous ceiling, carries only the remaining debt up to that existing horizon, and shortens later nominal intervals by at most one twenty-first. That 20/21 interval is exactly 5% above nominal rate; once the debt reaches zero, the flow returns to its unchanged nominal interval. Per-flow debt and the shared nominal fair-share calculation preserve the existing eight-flow fairness and rolling aggregate cap through the existing 25% and 50% capacity changes.
+
 The ingress repair follows reviewed behavior rather than copying implementation source:
 
 - Apollo `adc5c5a0bd80831ce495434bb16aee2cd4175fb8`, GPL-3.0, `src/stream.cpp:1463-1474,1573-1627`, supplies the 80%-of-1-Gbps raw-block pacing, 64-KiB/64-packet batch cap, and cross-frame send schedule used by the fixture.
@@ -37,9 +41,11 @@ The native provider therefore requests `2,048 * 1,072 = 2,195,456` bytes with `S
 - Carry a deterministic frame index/pattern only in the generated payload bytes; no codec semantics are claimed.
 - Keep the existing path/impairment/resource driver and change its unit from datagram payload to complete frame.
 - Keep video decrypt/FEC single-threaded; only the bounded connected-socket drain is separated so crypto stalls cannot become unexplained kernel loss.
+- Preserve valid scheduling debt after bounded host stalls instead of converting it into provider-queue residence; repay it within the existing fair pacer without a new queue, interface, or configured headroom.
 
 ## Risks / Trade-offs
 
 - [Keyframes can exceed queue budget] → use the reviewed 1 MiB frame ceiling and production byte-bound queue.
 - [Short smoke windows have rounding effects] → assert exact generated totals and report measured duration separately from normative ten-minute gates.
 - [A stalled video processor exhausts the user-space pool] → keep draining into one fixed scratch buffer and attribute accepted-size overflow to existing ingress/drop counters rather than kernel loss or unbounded allocation.
+- [Debt repayment creates a burst or aggregate oversubscription] → retain the 5 ms instantaneous ceiling and limit repayment to a 20/21 nominal interval per flow, with every rolling five-second aggregate window bounded to 105%.

openspec/changes/truthful-fixed-frame-qualification/proposal.md

@@ -6,12 +6,15 @@ The v6 complete-frame fixture subsequently exposed a source-fidelity defect on o
 
 Private Linux run 125 then demonstrated a separate production-ingress defect after source pacing was corrected: 33 successful fixture writes missing from `MediaIngress` matched 33 measured kernel UDP drops while decrypt/FEC, queue, pacer, QUIC, and client counters remained downstream of the shortfall.
 
+The consumed v8 Section 7 attempt at `55afea72` subsequently failed after accumulated host scheduling delays exposed the production pacer's discarded schedule debt beyond its 5 ms instantaneous catch-up allowance. The retained partial evidence remains failed and supersedes `55afea72` as a final executable candidate.
+
 ## What Changes
 
 - Generate deterministic variable-size encoded frame units at the named frame rates and target bitrates, including bounded keyframes.
 - Traverse native Apollo recovery, production queues, the production pacer, QUIC framing, and independent reassembly.
 - Assert frame count/rate, bitrate, exact bytes and boundaries, clean loss attribution, latency, and resource bounds.
 - Decouple native video socket draining from the single decrypt/FEC processor with a fixed provider-scoped receive pool and request the source-backed video receive-buffer size.
+- Retain bounded valid per-flow pacing debt after a host stall and repay it at no more than 5% above nominal fair share.
 - Keep short smoke tests separate and leave all prior normative artifacts unchanged.
 
 ## Capabilities
@@ -26,4 +29,4 @@ None.
 
 ## Impact
 
-The qualification harness, its canonical specification, and native Apollo video ingress in `gateway/apollo_native.go`. Downstream complete-frame queues, audio/control ingress, codec/FEC formats, pacing, QUIC, dependencies, and public interfaces remain unchanged. No codec operation or normative rerun is included. Requirements: P3C-002, P3C-008, P3C-029, P3C-030, P3C-033, VER-009, VER-010, OPS-015.
+The qualification harness, its canonical specification, native Apollo video ingress in `gateway/apollo_native.go`, and the existing production fair pacer in `gateway/telemetry.go`. Downstream complete-frame queues, audio/control ingress, codec/FEC formats, QUIC, dependencies, and public interfaces remain unchanged. No codec operation or normative rerun is included. Requirements: P3C-002, P3C-008, P3C-029, P3C-030, P3C-033, VER-009, VER-010, OPS-015.

openspec/changes/truthful-fixed-frame-qualification/specs/gateway-qualification/spec.md

@@ -7,6 +7,8 @@ Within each complete frame the source fixture SHALL reproduce pinned Apollo's so
 
 Native Apollo video ingress SHALL request a 2,195,456-byte socket receive buffer before media ping or worker startup and SHALL drain the connected video socket into a fixed FIFO pool of exactly 2,048 slots before the existing single decrypt/FEC processor. Each slot and the saturation scratch buffer SHALL be `apolloMediaMaximumPacket + 1` bytes so oversized datagrams remain rejected. A full pool SHALL NOT stop socket draining: each successfully read accepted-size discard SHALL increment both media-ingress and media-drop telemetry without allocation, while oversized reads SHALL retain the existing rejection accounting. Socket closure SHALL cancel the video read, and video/audio channels SHALL close only after the audio reader, video drain, and video processor exit. Audio and control ingress SHALL remain unchanged.
 
+The production fair pacer SHALL retain its 5 ms instantaneous catch-up ceiling. When a flow resumes later than that ceiling, it SHALL carry the remaining valid schedule debt only within the existing 250 ms provider-queue horizon and SHALL repay that debt using an interval no shorter than 20/21 of its nominal equal-tier fair-share interval. It SHALL return to the nominal interval when the debt is repaid. Simultaneous debt across eight equal-tier flows and the existing 25% and 50% capacity changes SHALL preserve the existing share-error contract and SHALL NOT exceed 105% of configured aggregate capacity in any rolling five-second window.
+
 #### Scenario: Healthy fixed profile
 - **WHEN** a frozen candidate runs one fixed profile for the normative duration in the isolated qualification command
 - **THEN** the harness emits compressed raw frame/path and gateway-process resource samples plus a summary tied to the exact command, CPU scope, timing-overhead method, topology, source commit, immutable Protocol version, environment, and payload hash
@@ -26,3 +28,7 @@ Native Apollo video ingress SHALL request a 2,195,456-byte socket receive buffer
 #### Scenario: Video ingress pool saturates
 - **WHEN** all 2,048 fixed video slots are occupied
 - **THEN** accepted-size datagrams are deliberately discarded through the fixed scratch buffer and counted as ingress plus drops, oversized datagrams remain rejected, and cancellation closes every media worker without a race or leak
+
+#### Scenario: Repeated media-loop host stalls
+- **WHEN** three approximately 95 ms scheduling debts are introduced at separated completed-public-frame barriers while source recovery continues
+- **THEN** the pacer limits instantaneous catch-up to 5 ms, repays each remaining debt at no more than 5% above nominal fair share, preserves every frame in exact order and bytes without provider or gateway drops, stays within the existing queue bounds, and closes cleanly on cancellation

openspec/changes/truthful-fixed-frame-qualification/tasks.md

@@ -16,7 +16,7 @@
 
 ## 4. Frozen qualification
 
-- [x] 4.1 Run the single normative Section 7 qualification after immutable Protocol consumer resolution
+- [x] 4.1 Run the earlier single normative Section 7 qualification after immutable Protocol consumer resolution; later audit findings superseded that candidate
 
 ## 5. Pinned Apollo source-fidelity remediation
 
@@ -31,3 +31,10 @@
 - [x] 6.1 Preserve run 125 and reproduce its pre-decrypt shortfall with a public 662-shard blocked-AEAD regression
 - [x] 6.2 Add the video-only 2,195,456-byte socket-buffer request, fixed 2,048-slot drain, single processor, overflow accounting, and bounded cancellation tests
 - [ ] 6.3 Freeze the reviewed production repair through the still-open private Linux full-verification and artifact gate before any replacement normative run
+
+## 7. Fair-pacer schedule-debt remediation
+
+- [x] 7.1 Preserve the consumed failed `55afea72` v8 attempt and its two partial files without retry, relabeling, or modification
+- [x] 7.2 Reproduce repeated host-stall queue expiry through the public native path and add bounded one-flow/eight-flow debt, fairness, rolling-cap, and capacity-step regressions
+- [x] 7.3 Retain the 5 ms instantaneous ceiling, carry valid debt within the 250 ms queue horizon, and repay it at no more than 5% above nominal fair share
+- [ ] 7.4 Freeze and verify a new executable candidate on private Linux before any separately authorized replacement normative run