Release qualification policy

routerd release qualification tests the product, not the test substrate. The test substrate must be certified before the release run starts.

If an incident requires a release rollback before or during qualification, follow Release rollback procedure. A binary rollback is the default and must not silently replace the current canonical configuration.

Gating rule

Release qualification must not start unless all of the following are true:

• A certification manifest exists.
• The manifest validates against docs/releases/manifests/release-environment-certification.schema.json.
• status is pass.
• expiresAt is later than the current UTC time.
• The manifest environment, topology, provider set, and script revisions match the requested qualification run.
• The mandatory offline regression gate and actual saved preparation checks pass using the current reviewed QA source. Certification runs this gate before its driver; qualification preflight repeats it. See the preparation failure record and procedure.

No certification, failed certification, blocked certification, mismatched certification, or expired certification is a hard stop.

Failure classification

Phase	Failure example	Classification
Environment certification	PVE bridge missing, QGA dead, config disk broken, cloud-init broken, cloud quota missing	Infra failure
Environment certification	Repair succeeds and all checks pass	Certified environment
Release preflight	Manifest missing, expired, invalid, or mismatched	Infra failure / no-go
Any phase	Invalid collector, stale observation, incorrect predicate or incomplete baseline	Test tooling defect; no valid product verdict
Release qualification	routerd readiness fails on certified substrate	routerd product failure
Release qualification	connectivity matrix fails on certified substrate	routerd product failure
Release qualification	provider action check fails on certified substrate	routerd product failure
Release qualification	PVE VM, bridge, QGA, config disk, cloud bootstrap, identity, or quota defect is discovered	Infra failure; stop qualification

Only failures established by valid observations and predicates after a passing preflight, within routerd readiness, matrix, or provider action checks, count as routerd product failures. A passing certification does not make a later test-code defect a product defect. Preserve the original result and explain the root cause separately if its historical schema has no test-tooling classification.

No repair during qualification

After release qualification starts, operators and automation must not repair:

• PVE VMs, templates, bridges, VLANs, firewall state, or node services.
• QGA readiness, guest networking, SSH bootstrap, config disks, or NoCloud media.
• Cloud images, quotas, identities, provider networks, route tables, security groups, public IPs, metadata access, or cloud-init bootstrap.
• Cross-substrate reachability that belongs to the lab fabric rather than routerd behavior.

If any of these fail during qualification, stop the run and return to environment certification. The next qualification attempt must reference a new passing certification manifest.

Dirty fabric with a fresh database

Release qualification does not support reusing a dirty lab fabric with a fresh routerd state database. The fabric includes provider-owned secondary IPs, PVE bridges and guests, guest /tmp/routerd-* residue, config media, provider action side effects, and any other substrate state that can outlive a routerd process or database.

This combination produces ambiguous evidence: routerd sees no prior action or ownership rows, while the fabric can still contain old holders, stale local artifacts, or provider-side mutations. Treat the result as an unsupported test setup, not as a routerd product failure or a valid passing release signal.

Operators must choose one of these supported paths before qualification:

• reuse both the certified fabric and its matching routerd state;
• clean the fabric and start with a fresh routerd state database;
• recertify after explicitly repairing and recording every retained substrate artifact that the fresh database will encounter.

Product qualification scope

With a certified environment, release qualification may evaluate:

• routerd process readiness and API/status readiness.
• Config validation, plan, apply, and rendered service state.
• Full connectivity matrix expected for the release topology.
• Provider action execution and fencing behavior.
• Failure handling that is intentionally injected by the routerd qualification scenario, as long as the substrate remains certified.

The qualification result should reference the certification manifest ID and record whether each failure is product_failure, infra_failure, preflight_failure, or observation_inconclusive. Use product_failure only for an observed violation of the product requirement. A missing or ambiguous stimulus, uncertain clock ordering, or incomplete collection is observation_inconclusive; a missing or invalid driver report is an infra_failure. An inconclusive result keeps status: fail (qualification has not passed), without claiming a product defect.

Retain every attempt and its actual sender start/end time, result, correlation identifier and collection error. Use timestamps from the same host when establishing causality; a management RPC timestamp includes transport delay. A DB receipt alone does not prove that a stimulus was qualified. Any additional observation opportunity must fit the original deadline, capture window and unchanged fault state. Reacquire only missing or unsuccessful observations; never discard a proven product violation when a later attempt succeeds.

Reproduce the saved failure and adjacent healthy observations before a local regression check. Select further retests from the changed paths and untested requirements, and record the scope and reason before running them. Keep sealed results unchanged; a diagnostic replay or a partial local pass does not replace the original run or qualify other nodes, providers or conditions.

Operator workflow

1. Run PVE and cloud certification in routerd-labs.
2. Confirm the manifest records the run ID, routerd artifact, routerd-labs revision, provider profiles/regions, OpenTofu state path, PVE management address source, and the selected fresh or retained state mode.
3. Review any repairs made during certification, including removed guest /tmp/routerd-* artifacts, patched topology outputs, refreshed provider inventory, or retained reusable PVE clients.
4. Store or attach the certification manifest and evidence paths before product qualification begins.
5. Run release preflight against that manifest.
6. Run release qualification smoke on the certified environment.
7. If qualification finds substrate damage, stop and recertify; do not repair in place during the product run.

When a rollback was used, review its binary SHA-256, selected scope, candidate provenance, apply result, and resulting canonical config SHA-256 before step 5.

The generic scripts, schemas, preflight checks, and lifecycle watchdog are versioned in this repository. Private routerd-labs drivers perform the actual provider/PVE checks and qualification scenarios. Qualification must not start unless its independent TTL/heartbeat watchdog and run-ID-scoped cleanup and inventory commands are executable.

This separation keeps release evidence interpretable: certification proves the lab was trustworthy, and qualification proves routerd behavior on that trusted lab.