test(gateway): add Section 7 qualification harness

2026-07-29 22:13:48 +07:00
parent 8826f5c804
commit f37f53eab2
7 changed files with 1053 additions and 0 deletions
@@ -0,0 +1,2 @@
+schema: spec-driven
+created: 2026-07-29
@@ -0,0 +1,54 @@
+## Context
+
+P3C-029 through P3C-033 require raw, reproducible processing, payload-rate,
+fairness, cap, and impairment evidence. Existing focused tests cover the
+framer, bounded queues, native Apollo fake, and fair pacer, but do not emit the
+normative ten-minute or six-profile artifacts.
+
+## Goals / Non-Goals
+
+**Goals:**
+
+- Run the three 20/50/80 Mbps encoded-payload profiles for ten wall-clock
+  minutes each after a recorded warm-up.
+- Measure the existing gateway framing path with a monotonic clock and retain
+  compressed raw latency samples plus full summary statistics.
+- Run the exact six Section 7.2 configurations once using a deterministic,
+  bounded virtual packet discipline and retain configured and observed values.
+- Exercise one real mTLS/QUIC fake-provider traversal for every media profile
+  and reuse the real fair-pacer implementation for fairness and cap evidence.
+
+**Non-Goals:**
+
+- Live Apollo/macOS/firewall qualification, real encoder fidelity, codec
+  processing, host network mutation, or multi-host scale.
+- A production impairment framework, new gateway API, dependency, cgo, or
+  sidecar.
+
+## Decisions
+
+- Implement the harness as an opt-in `go test` in package `gateway`. This
+  keeps qualification access to the actual unexported fair pacer without
+  adding a production API. Normal suites skip the long run unless an explicit
+  absolute evidence directory is supplied.
+- Use wall-clock duration and target-rate pacing for performance profiles.
+  Measure only receive-to-framed-payload processing; pacing wait and raw-file
+  writes stay outside the measured interval and are reported separately.
+- Stream every raw sample into gzip-compressed CSV while retaining one bounded
+  duration slice per profile for exact percentiles.
+- Use a fixed-seed virtual FIFO for impairment. It records no host claim and
+  identifies its queue discipline and deterministic topology explicitly.
+- Treat any payload mutation, p95 above 5 ms, catalog mismatch, fairness error
+  above 10%, cap excess above 5%, or step convergence beyond ten seconds as a
+  hard command failure.
+
+## Risks / Trade-offs
+
+- [Local virtual impairment cannot prove deployed route behavior] → label every
+  artifact deterministic and retain live Apollo/macOS/firewall as
+  deferred-owner-e2e.
+- [Raw samples can be large] → stream gzip output and bound in-memory samples
+  to the exact profile packet budget.
+- [Host load can invalidate latency] → record OS, architecture, Go version,
+  timing overhead, actual duration, packet count, and observed bitrate; fail
+  rather than substitute configured capacity for measured egress.
@@ -0,0 +1,35 @@
+## Why
+
+The Phase 3C gateway candidate has deterministic transport and scheduler tests
+but no executable artifact generator for the normative ten-minute media
+measurements and six bounded impairment profiles. Without that evidence,
+P3C-029 through P3C-033 cannot be frozen truthfully.
+
+## What Changes
+
+- Add one stdlib-only qualification command for the three fixed encoded-media
+  profiles and the exact six Section 7.2 impairment profiles.
+- Emit bounded machine-readable configuration, raw observations, summaries,
+  environment, topology, direction, queue discipline, and tool version.
+- Fail the command when payload integrity, the 5 ms processing p95, impairment
+  bounds, fairness, capacity-step convergence, or aggregate cap gates fail.
+- Keep live Apollo, macOS, physical firewall, real encoder fidelity, and real
+  multi-host scale explicitly deferred-owner-e2e.
+
+## Capabilities
+
+### New Capabilities
+
+- `gateway-qualification`: Deterministic P3C-029 through P3C-033 media,
+  processing, fairness, cap, and impairment evidence generation.
+
+### Modified Capabilities
+
+None.
+
+## Impact
+
+The Data Plane gains a qualification-only Go command, focused tests, and
+documented evidence output. It adds no dependency, production transport
+abstraction, provider route, codec operation, cgo, sidecar, or Connection
+Server code.
@@ -0,0 +1,63 @@
+## ADDED Requirements
+
+### Requirement: Fixed media processing qualification
+The qualification harness SHALL run 1080p60 H.264 at 20 Mbps, 1440p120 HEVC
+at 50 Mbps, and 4K60 HEVC at 80 Mbps for ten wall-clock minutes each after a
+recorded warm-up. It SHALL preserve encoded payload bytes, record every
+monotonic processing sample, report count, min, median, p90, p95, p99, max,
+mean, standard deviation, timing overhead, and observed bitrate, and fail when
+any p95 exceeds 5 ms.
+
+#### Scenario: Healthy fixed profile
+- **WHEN** a frozen candidate runs one fixed profile for the normative duration
+- **THEN** the harness emits compressed raw samples and a summary tied to the
+  exact source commit, Protocol version, environment, and payload hash.
+
+#### Scenario: Processing gate failure
+- **WHEN** payload integrity fails or measured p95 exceeds 5 ms
+- **THEN** the qualification command exits unsuccessfully without recording a
+  passing candidate.
+
+### Requirement: Bounded impairment qualification
+The harness SHALL run exactly the baseline, latency, jitter, loss, reorder,
+and constrained Section 7.2 profiles once. Baseline SHALL cover all three
+media profiles and the other profiles SHALL cover 1080p60. Each artifact SHALL
+record tool version, exact command/configuration, direction, queue discipline,
+topology, fixed seed, and observed RTT, jitter, loss, reorder, throughput,
+drops, and capacity-step statistics.
+
+#### Scenario: Complete six-profile run
+- **WHEN** the frozen candidate runs impairment qualification
+- **THEN** one result exists for each named profile, with no Cartesian
+  expansion and with observed rather than configured statistics.
+
+#### Scenario: Unsupported or unbounded configuration
+- **WHEN** a profile name, packet count, queue bound, loss, reorder, or
+  bandwidth step falls outside the fixed catalog
+- **THEN** the harness rejects it before allocating or running the simulation.
+
+### Requirement: Fairness and cap qualification
+The harness SHALL exercise the production fair pacer with eight equal-tier
+synthetic sessions for the required 60-second virtual interval, report every
+share error and Jain's fairness index, and fail above 10% share error. It SHALL
+apply 25% and 50% capacity steps, fail convergence beyond ten virtual seconds,
+and fail aggregate egress above 105% of the cap over any rolling five-second
+window.
+
+#### Scenario: Equal-tier and capacity-step evidence
+- **WHEN** the frozen candidate runs scheduler qualification
+- **THEN** the artifact contains per-flow bytes, share errors, Jain's index,
+  step convergence, and rolling cap observations derived from the production
+  pacer.
+
+### Requirement: Honest qualification boundary
+Qualification artifacts SHALL contain no provider endpoint, credential,
+clipboard text, input payload, secret, raw media content, or claim of live
+Apollo/macOS/firewall interoperability. The harness SHALL add no codec
+operation, production dependency, cgo, sidecar, or direct provider route.
+
+#### Scenario: Deterministic evidence publication
+- **WHEN** qualification completes
+- **THEN** the manifest labels fake-provider, virtual impairment, and local
+  processing evidence separately and leaves live interoperability
+  deferred-owner-e2e.
@@ -0,0 +1,24 @@
+## 1. Contract and focused regressions
+
+- [x] 1.1 Add fixed catalog tests for the three media profiles, ten-minute
+  duration, exact six impairment profiles, and bounded output configuration.
+- [x] 1.2 Add summary, payload-integrity, fairness, cap, and failure-threshold
+  tests before implementing the harness.
+
+## 2. Qualification harness
+
+- [x] 2.1 Implement opt-in real-duration processing measurement with compressed
+  raw samples, full statistics, timing overhead, and environment metadata.
+- [x] 2.2 Implement deterministic bounded impairment observations and reuse the
+  production fair pacer for fairness and capacity-step evidence.
+- [x] 2.3 Add one mTLS/QUIC fake-provider traversal per fixed encoded profile
+  and prove the artifact boundary contains no provider route or secret.
+
+## 3. Freeze and evidence
+
+- [x] 3.1 Run focused red/green checks, strict OpenSpec validation, `make
+  verify`, race/fuzz/resource checks, and freeze the harness commit.
+- [ ] 3.2 Run the opt-in ten-minute and six-profile command exactly once
+  against the frozen candidate and archive raw artifacts and hashes.
+- [ ] 3.3 Sync the canonical specification, archive the completed change, and
+  revalidate strictly without claiming live Apollo/macOS/firewall evidence.