Files
sechmachine 22433e5c45
Verify Data Plane / gateway (push) Successful in 4m46s
test(gateway): measure qualification wire capacity
2026-08-09 23:01:42 +07:00

41 lines
7.8 KiB
Markdown

## MODIFIED Requirements
### Requirement: Fixed media processing qualification
The qualification harness SHALL drive pinned-mTLS Apollo management, encrypted RTSP, ENet, and provider UDP through native source validation, `readUDPMedia`, recovery/FEC, byte/count/latency-bounded production queues, the production fair pacer, Protocol complete-frame fragmentation, Verse framing/QUIC, and an independent bounded client reassembler for 1080p60 H.264 at 20 Mbps, 1440p120 HEVC at 50 Mbps, and 4K60 HEVC at 80 Mbps. The source fixture SHALL emit deterministic variable-size complete encoded frame units at the named 60/120 FPS rate, preserve exact target bytes over each fixed interval, and include bounded larger keyframes without codec operation. After a recorded warm-up, the frozen candidate SHALL run each profile for ten wall-clock minutes, preserve every frame's bytes and boundary, retain every monotonic processing sample plus bounded provider-queue observations, and report frame count, frame rate, bitrate, count, min, median, p90, p95, p99, max, mean, standard deviation, measured batched monotonic-clock overhead and method, and observed bitrate. Processing begins at complete provider-frame receipt and ends at QUIC handoff, excluding client transit and pacing. Queue delay SHALL measure provider-queue residence, processing SHALL measure gateway work before pacing, and pacing delay SHALL measure scheduler waiting. CPU, heap, allocations, and goroutines SHALL be measured from the isolated gateway process only; CPU SHALL be actual OS user plus system consumption and MUST NOT include idle wall capacity or unrelated parent fixture/client work. Successive profiles SHALL use independent resource-counter baselines. Any bypass, payload or boundary mutation, frame-rate/count mismatch, wall-duration violation, bitrate outside both lower and upper bounds, unexplained clean-path loss, zero or unbounded clock overhead, or p95 above 5 ms SHALL fail.
Within each complete frame the source fixture SHALL reproduce pinned Apollo's source schedule by deriving packets per millisecond from the raw UDP block size at 80% of 1 Gbps, bounding each source batch to the smaller of 64 KiB or 64 packets, capturing the monotonic batch start immediately after the first successful shard write, scheduling the next batch no earlier than that start plus the current batch's raw-block serialization interval, carrying that schedule across frames, and making pacing waits context-cancellable. A delayed batch SHALL remain late rather than trigger an overdue catch-up burst.
Native Apollo video ingress SHALL request a 2,195,456-byte socket receive buffer before media ping or worker startup and SHALL drain the connected video socket into a fixed FIFO pool of exactly 2,048 slots before the existing single decrypt/FEC processor. Each slot and the saturation scratch buffer SHALL be `apolloMediaMaximumPacket + 1` bytes so oversized datagrams remain rejected. A full pool SHALL NOT stop socket draining: each successfully read accepted-size discard SHALL increment both media-ingress and media-drop telemetry without allocation, while oversized reads SHALL retain the existing rejection accounting. Socket closure SHALL cancel the video read, and video/audio channels SHALL close only after the audio reader, video drain, and video processor exit. Audio and control ingress SHALL remain unchanged.
The production fair pacer SHALL retain its 5 ms instantaneous catch-up ceiling. When a flow resumes later than that ceiling, it SHALL carry the remaining valid schedule debt only within the existing 250 ms provider-queue horizon and SHALL repay that debt using an interval no shorter than 20/21 of its nominal equal-tier fair-share interval. It SHALL return to the nominal interval when the debt is repaid. Simultaneous debt across eight equal-tier flows and the existing 25% and 50% capacity changes SHALL preserve the existing share-error contract and SHALL NOT exceed 105% of configured aggregate capacity in any rolling five-second window.
Capacity-step convergence and rolling-cap evidence SHALL use the monotonic receive time and encoded length of every raw public QUIC datagram observed immediately after the independent client's `ReceiveDatagram` returns and before decode or reassembly. For each reduction, target bytes per second and the five-second cap SHALL derive from the configured public-wire rate, `qualificationMediaPacerKbps(profile, reduction) * 1000 / 8`. Completed logical-payload observations SHALL remain separate and SHALL continue to measure payload integrity, loss, reorder, latency, throughput, and queue behavior. A capacity step SHALL include only delivery observations at or after its recorded transition, anchor measurement windows at the first such public delivery, require four consecutive 250 ms windows between 90% and 105% of its target, converge within ten seconds, and remain at or below 105% in every rolling five-second window.
#### Scenario: Healthy fixed profile
- **WHEN** a frozen candidate runs one fixed profile for the normative duration in the isolated qualification command
- **THEN** the harness emits compressed raw frame/path and gateway-process resource samples plus a summary tied to the exact command, CPU scope, timing-overhead method, topology, source commit, immutable Protocol version, environment, and payload hash
#### Scenario: Processing gate failure
- **WHEN** any production path stage lacks a per-frame observation, stage accounting does not balance, payload or frame boundaries change, duration, frame-rate, frame-count, or bitrate bounds fail, measured p95 exceeds 5 ms, parent work changes gateway CPU, idle capacity is reported as consumed CPU, or timing overhead is absent
- **THEN** the qualification command exits unsuccessfully without recording a passing candidate
#### Scenario: Source-shaped Apollo pacing is preserved
- **WHEN** the fixture emits 1,072-byte encrypted video shards with 1,040-byte raw blocks for consecutive complete frames
- **THEN** it uses 96 packets per millisecond, batches at most 63 shards, records each batch after its first successful shard write, starts each later batch no earlier than the prior batch's raw serialization interval, carries the schedule into the following frame, and emits no shard after a cancelled pacing wait
#### Scenario: Video crypto processing stalls
- **WHEN** the first video AEAD operation is blocked while a 662-shard keyframe arrives
- **THEN** all 662 successful connected-socket reads reach media-ingress accounting before processing resumes, and after release the exact complete frame traverses recovery, the bounded production queue, pacer, QUIC, and independent reassembly
#### Scenario: Video ingress pool saturates
- **WHEN** all 2,048 fixed video slots are occupied
- **THEN** accepted-size datagrams are deliberately discarded through the fixed scratch buffer and counted as ingress plus drops, oversized datagrams remain rejected, and cancellation closes every media worker without a race or leak
#### Scenario: Repeated media-loop host stalls
- **WHEN** three approximately 95 ms scheduling debts are introduced at separated completed-public-frame barriers while source recovery continues
- **THEN** the pacer limits instantaneous catch-up to 5 ms, repays each remaining debt at no more than 5% above nominal fair share, preserves every frame in exact order and bytes without provider or gateway drops, stays within the existing queue bounds, and closes cleanly on cancellation
#### Scenario: Capacity measurement crosses a short transition phase
- **WHEN** a constrained 1080p flow carries nonzero bounded debt through the approximately 1.572-second 25% phase before the 50% transition
- **THEN** convergence and rolling-cap checks use the observed 1,200-byte and 25-byte public datagrams against the configured wire targets, while the separately retained completed-payload observations cannot substitute for transport delivery timing