HowTo go from multiple partially_verifies to a fully_verifies

[AlexanderLanin]

Design draft: aggregate requirement verification without teststatus

Problem statement

The test links already contain the execution result, including whether a test
passed. The missing information is the aggregate meaning of several links:

• Several tests can link to one requirement with partially_verifies.
• A requirement can therefore have five passing test links without anyone being
  able to tell whether those five tests, in combination, cover the complete
  requirement.
• teststatus / testcovered=YES stores that aggregate conclusion as a
  manually maintained, one-time-looking status on the requirement. It does not
  show which tests justified it and can become stale when the requirement or a
  test changes.

The desired information should not be stored in another requirement-level
source. It should be encoded on one selected, representative test and
recalculated from the current test links on every build.

This distinction is especially important for Feature Requirements. A Feature
Requirement describes externally visible feature behavior and does not know which
component or implementation will eventually realize it. Completeness cannot be
inferred by collecting all tests below the feature in the implementation
hierarchy.

What the current process already says

The process has the necessary concepts, but they are currently collapsed into a
requirement attribute:

• The verification concept calls for a linkage document containing the
  requirement hash and the UIDs of the tests required to fully cover it
  (process/process_areas/verification/verification_concept.rst:190-201).
• The Set Requirement Test Coverage workflow says that complete coverage means
  the linked tests in sum fully satisfy the requirement
  (process/process_areas/verification/verification_workflows.rst:193-223).
• The guideline says that only executed and passed tests count towards the
  coverage of a requirement, and that skipped or non-executed tests belong in the
  verification report and release notes
  (process/process_areas/verification/guidance/verification_guideline.rst:145-153).
• Feature Integration Tests are based on Feature Requirements. This is the
  appropriate level at which a Feature Requirement can be verified without
  knowing its implementation.

Proposed model: an anchor test

1. Ordinary test links remain unchanged

Test metadata describes the contribution of an individual test:

@add_test_properties(
    partially_verifies=["feat_req__example__valid_configuration"],
    test_type="requirements-based",
    derivation_technique="requirements-analysis",
)
def test_accepts_valid_configuration():
    ...

partially_verifies means that this test covers some part of the requirement.
The existing test link continues to provide the current execution result.

fully_verifies remains available for the simple case where one test alone
covers the complete requirement:

@add_test_properties(
    fully_verifies=["comp_req__example__valid_configuration"],
    test_type="requirements-based",
    derivation_technique="requirements-analysis",
)
def test_complete_configuration_behavior():
    ...

2. Encode an aggregate on the selected test

When several tests are needed, select one representative or “most important”
test as the anchor. Encode the complete set on that test:

@add_test_properties(
    fully_verifies=["tool_req__docs_common_attr_description"],
    required_for_fully_verifying=[
        "test_returns_valid_checks",
        "test_strips_whitespace",
        "test_raises_assertion_for_invalid_check",
    ],
    test_type="requirements-based",
    derivation_technique="requirements-analysis",
)
def test_returns_empty_list_when_filter_is_empty2():
    ...

The semantics are:

The anchor test plus all tests listed in
required_for_fully_verifying form the complete test set for the referenced
requirement. Their combined coverage is sufficient.

The anchor is only the owner of this declaration. It is not necessarily the
test that covers the most behavior, and fully_verifies does not mean that the
anchor passed. The result still comes from the test link status.

This keeps the information in the test sources, avoids a second requirement
database, and makes the complete set visible next to the test links that justify
it. The test IDs in required_for_fully_verifying must be stable test UIDs.

3. Recalculate the verdict on every build

For every fully_verifies declaration, the tool resolves the anchor and its
required tests and derives one result:

fully_verified_for_release
  = the anchor and all required tests exist
  AND all required tests link to the same requirement as declared
  AND the anchor declaration is valid for the current requirement
  AND the anchor and every required test have result == passed

Nobody has to remember to set teststatus=YES after a test is added, removed,
changed, or rerun. A failed, skipped, or non-executed required test immediately
makes the current release verdict not fully verified, while the anchor
declaration still records the intended complete test set.

Why this is better than a one-time requirement attribute

The existing process describes testcovered=YES / complete test coverage=yes
as the result of a committer confirming that linked tests in sum satisfy the
requirement. That is a useful review decision, but a Boolean on the requirement
loses the evidence behind the decision and looks permanent.

The anchor declaration keeps the decision close to its evidence and remains
declarative rather than being a status field:

Information	Location	Changes when
Test covers part of a requirement	test link metadata	test or traceability changes
These tests collectively cover the requirement	anchor test metadata	test set or coverage argument changes
Test passed for this run/release	existing test link result	test is rerun
Requirement fully verified for this release	generated verdict	any of the above changes

A review remains necessary for the semantic question “is this union complete?”,
but the resulting verdict is always recomputed from the anchor's declared set
and current test results. There is no manual requirement-level YES update.

[4og]

How would this work with GTest's typed and parametrized tests? In junit XML, each instantiation is reported as a separate test.
Example in baselibs: https://eclipse-score.github.io/baselibs/main/baselibs/components/containers/requirements/index.html#comp_req__containers__non_relocatable_vector

[a-zw]

We probably don't want to maintained the list of instantiations manually as this would defeat the point of typed tests.

This we have declare it as a pattern? That goes against the idea of "someone trustworthy looked at all the tests" though.

Don't have a good solution, but I want to remark that it is quite a stretch to me that all these unit tests are "requirements-based".