Files
VerseVDI-Data-Plane/openspec/changes/truthful-fixed-frame-qualification/design.md
T

73 lines
16 KiB
Markdown

## Context
The current harness sends one fixed 1,179-byte payload per logical sample. It reaches the production path but does not represent encoded frames at 60/120 FPS or exercise realistic fragmentation, reassembly, queue bytes, and keyframe pressure.
The complete-frame fixture also must preserve the pinned Apollo source schedule. For each frame it derives packets per millisecond from the raw UDP block size at 80% of 1 Gbps, limits source batches to both 64 KiB and 64 packets, and carries the next-send time into the following frame. Waiting is context-cancellable. This is qualification-fixture behavior only; production transport and queue behavior remain unchanged.
Because the bounded fixture uses loopback rather than a physical 1 Gbps link, v8 writes the first shard of a batch successfully, captures that actual monotonic emission start, and schedules the next batch no earlier than that start plus the current batch's raw-block serialization interval. The persistent schedule carries across frames. A delayed batch therefore remains late instead of collapsing overdue batches into a catch-up burst.
The retained v6 qualification run passed its then-current checks but is superseded because its tight-loop sender contradicted the pinned Apollo schedule. Private Linux runs 123 and 124 remain failed evidence. One local v8 sustained run passed on Darwin, but it is neither Linux proof nor normative Section 7 evidence.
The later Darwin non-sustained pre-CI invocation was not green and was not retried. Its 1440p120 profile delivered the exact 6,250,000 bytes in 120 frames plus all 6,483 source and warm-up shards with zero drops, but measured 46,973.13 kbps over an implied approximately 1.0644383 seconds and failed the 5% throughput gate. At that point, private Linux full verification/artifact retention and the replacement v10 normative run were still open.
Private Linux run 125 at the frozen v8 harness head is retained as failed evidence. Its exact 33-datagram gap between successful fixture writes and production `MediaIngress` equaled the Linux socket's 33 measured kernel UDP drops. The complete-frame queue, fair pacer, QUIC fragmentation, and public decoder were downstream and did not account for the loss.
The one authorized v8 Section 7 invocation at production candidate `55afea72a1487fa071501615d806e68efc0a436b` was consumed and failed. Its directory `gateway-rc10-55afea7` is retained byte-for-byte with two partial processing files and no manifest. The failure occurred at payload sequence 16801 after 16,834 provider frames had been recovered and enqueued; the provider queue reached 15 entries and dropped one valid frame while source-write and ingress accounting remained balanced at the diagnostic boundary. This attempt is failed evidence and is not eligible for retry or relabeling.
The production fair pacer previously limited instantaneous recovery to 5 ms by moving an overdue flow's schedule to `now-5ms`, but silently discarded every additional valid scheduling interval. Repeated host stalls therefore accumulated complete frames in the existing provider queue until its 250 ms residence horizon correctly expired one. The repair keeps the 5 ms instantaneous ceiling, carries only the remaining debt up to that existing horizon, and shortens later nominal intervals by at most one twenty-first. That 20/21 interval is exactly 5% above nominal rate; once the debt reaches zero, the flow returns to its unchanged nominal interval. Per-flow debt and the shared nominal fair-share calculation preserve the existing eight-flow fairness and rolling aggregate cap through the existing 25% and 50% capacity changes.
Private Linux run 127/job 481 at exact source `122080ab342d20585d9a45db0017337b9ece570a` is retained as failed evidence. `TestQualificationLossAndSteppedThroughputBounds` reported the 25% step's 11-second convergence sentinel and a 5,207,475-byte five-second maximum. No artifact was uploaded and the run was not retried. The retained log `/private/tmp/versevdi-gitea-run-127-job-481.log` has SHA-256 `2b368413d4b0e954c64ba6f6dcefb1e165166d773b6d8f84ee372bc2c92ff5f1`. The failure exposed a measurement-unit defect: capacity samples were emitted only after complete logical-payload reassembly and were compared with a payload-derived target even though the pacer reserves encoded public datagram bytes. It did not establish a production pacer defect, and `122080ab` is superseded as a final source candidate.
Qualification v9 observes every raw public QUIC datagram immediately after the independent client's `ReceiveDatagram` returns and before the existing decoder/reassembler. Capacity convergence and rolling five-second maxima use those monotonic receive times and encoded lengths. Their target bytes per second and five-second cap derive from `qualificationMediaPacerKbps(profile, reduction) * 1000 / 8`. Complete logical-payload observations remain separate and continue to own payload integrity, loss, reorder, latency, throughput, and queue assertions. The first public delivery at or after a step anchors the four consecutive 250 ms windows so an arbitrary control-plane timestamp cannot split the first observed datagram pair. The delivery-after-step boundary, 90%-105% window bounds, ten-second convergence ceiling, and rolling-five-second 105% gate are unchanged.
Private Linux run 128/job 482 was the single push-triggered attempt at exact source `22433e5c45c179e9d487b59e5d80f1dcf3b285ce`. Linux `make verify`, the sustained gate, strict OpenSpec, deterministic artifact generation, upload, and the clean-checkout step passed. Artifact 28's binaries and SPDX matched the frozen hashes and source metadata. The retained log `/private/tmp/versevdi-gitea-run-128-job-482.log` has SHA-256 `6f5852dce815ba87a55d61aae34cb2fce17a1a62c5ee50e18c0034259ad0a029`. The workflow requested 30 retention days, but Gitea 1.27 floored the positive request delay and the API recorded `2026-08-09T23:18:18+07:00` through `2026-09-07T23:18:18+07:00`, exactly 2,505,600 seconds or 29 elapsed days. The run was not retried. It is passing Linux execution and artifact-byte evidence but retention-nonconforming, so it neither satisfies the private Linux artifact gate nor authorizes normative Section 7. The local 31-day request compensates for verified platform rounding without changing the acceptance threshold; a separately authorized future run must prove an API interval of at least 2,592,000 seconds.
At exact source `c0e362c0285d267f8af4087d07943822311f60e1`, the first Section 7 process created mode-0750 `gateway-rc10-c0e362c` and stopped before fixture startup because the sandbox denied its required loopback bind. The directory remains empty as environment-boundary evidence. The separately authorized escalated attempt retained `gateway-rc10-c0e362c-a2`, passed the v9 runtime checks, and wrote 16 files with manifest SHA-256 `61ea55140dfe6b37332de03879ac33206dd5d47763d09fb0fc2f65bb1f8e02b4`. Independent review denied normative acceptance: the constrained logical CSV and manifest aggregates did not persist every raw public QUIC datagram receive offset/encoded length, and `fairness.csv.gz` lacked manifest transition offsets. A2 therefore remains runtime-passing but normative-raw-evidence-incomplete, without mutation or relabeling.
Qualification v10 keeps the logical impairment CSV unchanged and adds only `impairment-constrained-1080p60-h264-wire.csv.gz`. One monotonic epoch is captured before the constrained delivery and transition sequence. The bounded CSV interleaves two explicit transition records with every public datagram observation in monotonic order using `record_type,reduction_percent,transition_after_ns,received_after_ns,encoded_bytes`; transition-only and delivery-only fields remain empty and are validated as such. Its maximum row count is derived from the existing constrained job bound, the frame-fragment count, and the two configured transitions rather than a captured-run row constant. The manifest binds the file name, SHA-256, compressed bytes, total/delivery/transition row counts, timebase, exact transition offsets, and each capacity summary's recomputation source. Each impairment step is recomputed from every delivery at or after its transition through completion, preserving the original v9 classifier semantics even after the next transition. Fairness remains stage-bounded because `runQualificationFleetStage` records each capacity stage as a separate slice; its summaries retain their 25% and 50% offsets relative to the existing fairness CSV epoch.
The independent parser's caller explicitly selects smoke or normative authority. Normative validation requires exactly 10,000 sent logical units and always enforces the ten-second convergence and 105% rolling-cap gates; retained `sent` data cannot weaken them. Both wire and fairness readers reject compressed input before hashing when it exceeds a writer-derived bound, feed gzip output through a bounded standard-library reader before CSV parsing, and bound fields, offsets, flows, and encoded lengths from the canonical row count, schema, time horizon, and writer values. Aggregate-only v9 data and malformed schema/hash/count/order/transition/length or oversized inputs remain rejected.
Private Linux run 129/job 483 and artifact 29 at `c0e362c0285d267f8af4087d07943822311f60e1` passed execution, artifact-byte, and at-least-30-elapsed-day retention checks. Its subsequent v9 a2 qualification remains runtime-passing but normative-raw-evidence-incomplete. Exact v10 source `ce3b3079837c5745f95a51c98eb6204363962ae7` then passed its one private Linux push attempt in run 130/job 484: `make verify`, the sustained gate, strict OpenSpec, deterministic artifact build/upload, and clean checkout all passed. The retained run log is SHA-256 `7e23011db2cb3118ca6cd940cfdc59ba4b536de39276d6b6a74c5862c745b0fa`. Artifact 30's ZIP is SHA-256 `59f938f9187255ad015fa5608b49909e89a065e4f5ba515b7450608b8048b1cc` at 12,957,227 bytes; its amd64 binary is `33ecd42cee0e01bc15aa87aa0610bfb165b2a2b379133110271c98d717584cc1` at 12,367,020 bytes, its arm64 binary is `4b6fc560e23fd8e283219f63ac4a33881863d12644676fb6e5a7d9fc4523182b` at 11,525,817 bytes, and its source-correct SPDX is `a1c2ae2bc4a8c0e0dea67ac53dbf8bb6135399cecb8d6fd153e9b59174ec35b9` at 8,090 bytes. The API recorded exactly 2,592,000 seconds of retention. The binaries remain unchanged because v10 changes retained test evidence only; the SPDX truthfully remains unscanned and unsigned.
Two subsequent ce3 elevated-execution preflights were denied before `CreateProcess`. No `go test` process started, no target was created, and neither event consumed a qualification attempt; they remain environment/approval history rather than runtime failures. The later owner-approved elevated direct command ran exactly once at ce3 with immutable Protocol RC10, exited zero, and passed `TestSection7Qualification` in 1,917.92 seconds (package 1,918.491 seconds). It produced mode-0750 `gateway-rc10-ce3b307` with 17 files totaling 4,179,714 bytes and a passed v10 manifest SHA-256 `53f1e24a3182887d496b745826af20c7b220a3a0cc2e1682baf1eb746f4c240d`.
Independent review recomputed 36,000/72,000/36,000 processing frames and 11.25 GB of exact payload, including payload digests, statistics, isolated resources, and all eight logical impairment files. It also recomputed the constrained wire artifact's 15,060 rows as 15,058 ordered 1,200/25-byte deliveries plus canonical 25%/50% transitions, including full-tail convergence and maxima. The fairness artifact retained 70,916 rows; stage-bounded recomputation reproduced Jain index `0.9999999546920958`, maximum share error `0.0003749275708101844`, and the two-second 25%/50% convergence/maxima. Identity, hashes, bounded grammar, and preservation checks found no material mismatch. These results remain deterministic fixture evidence only and do not prove live Apollo, the native macOS client, a physical firewall or route, a real encoder, multi-host operation, Phase 3C-C, vulnerability scanning, signing, deployment, or promotion.
The ingress repair follows reviewed behavior rather than copying implementation source:
- Apollo `adc5c5a0bd80831ce495434bb16aee2cd4175fb8`, GPL-3.0, `src/stream.cpp:1463-1474,1573-1627`, supplies the 80%-of-1-Gbps raw-block pacing, 64-KiB/64-packet batch cap, and cross-frame send schedule used by the fixture.
- Moonlight common-C pin `2ea47752c3051d72a64bcca190024e8b354fa1ef`, GPL-3.0, `src/VideoStream.c:28-35,331-333` and `src/PlatformSockets.c:364-405`, supplies the reviewed 2,048-video-packet receive-buffer request and dedicated receive-thread behavior. The cited `VideoStream.c` blob is byte-identical at the local standalone `703a06946861ff82cd33e5e13c59c1b017f7ded9` checkout.
The native provider therefore requests `2,048 * 1,072 = 2,195,456` bytes with `SetReadBuffer()` on the connected video socket immediately after dialing it. A setter error aborts setup; an OS-imposed cap is accepted without privilege or getter dependence. A dedicated drain owns a fixed 2,048-slot FIFO pool. Every slot is 1,433 bytes (`apolloMediaMaximumPacket + 1`), so oversized datagrams remain observably invalid rather than being truncated into the accepted range; packet storage is 2,934,784 bytes (about 2.80 MiB) plus fixed index and timestamp metadata. The existing single decrypt/FEC processor consumes those slots. When every slot is occupied, the drain keeps reading into one fixed 1,433-byte scratch buffer and counts each accepted-size discard in both ingress and drop telemetry; oversized datagrams retain the existing rejection semantics. Socket close cancels the blocking read, and media channels close only after the unchanged audio reader, video drain, and video processor exit. Audio and control behavior are unchanged.
## Goals / Non-Goals
**Goals:**
- Deterministically generate complete variable-size frame units at exact profile frame rates and target bitrates.
- Include bounded periodic keyframes while preserving exact aggregate bytes.
- Measure the existing production path and independent reassembly with frame-level accounting.
**Non-Goals:**
- A real encoder, codec parsing, a second simulator, or a normative run before immutable Protocol publication.
## Decisions
- Derive bytes per fixed interval from bitrate and FPS, distribute integer remainder deterministically, and shift bounded bytes into periodic keyframes while keeping the interval total exact.
- Carry a deterministic frame index/pattern only in the generated payload bytes; no codec semantics are claimed.
- Keep the existing path/impairment/resource driver and change its unit from datagram payload to complete frame.
- Keep video decrypt/FEC single-threaded; only the bounded connected-socket drain is separated so crypto stalls cannot become unexplained kernel loss.
- Preserve valid scheduling debt after bounded host stalls instead of converting it into provider-queue residence; repay it within the existing fair pacer without a new queue, interface, or configured headroom.
- Classify constrained-capacity evidence from actual public datagram observations and configured wire capacity; do not infer transport timing from completed logical frames.
- Persist constrained public-wire observations and transition events in one bounded monotonic-offset CSV, recompute impairment summaries from each transition through completion, and bind both impairment and stage-bounded fairness capacity summaries to their retained raw sources.
- Select smoke versus normative evidence validation through trusted caller input and bound compressed bytes, decompressed bytes, fields, offsets, flows, and encoded lengths before independent CSV parsing.
## Risks / Trade-offs
- [Keyframes can exceed queue budget] → use the reviewed 1 MiB frame ceiling and production byte-bound queue.
- [Short smoke windows have rounding effects] → assert exact generated totals and report measured duration separately from normative ten-minute gates.
- [A stalled video processor exhausts the user-space pool] → keep draining into one fixed scratch buffer and attribute accepted-size overflow to existing ingress/drop counters rather than kernel loss or unbounded allocation.
- [Debt repayment creates a burst or aggregate oversubscription] → retain the 5 ms instantaneous ceiling and limit repayment to a 20/21 nominal interval per flow, with every rolling five-second aggregate window bounded to 105%.