Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions .github/workflows/downstream-compatibility.yml
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,10 @@ jobs:
repository: DataDog/dd-trace-py
runtime: python
submodule: tests/openfeature/ffe-system-test-data
setup: ":"
# ddtest runs as uid 1000 in Docker while the Actions checkout is uid 1001.
setup: >-
mkdir -p .cache/cython .cache/pip .cache/sccache .riot &&
chmod -R a+rwX .
evaluation: >-
scripts/ddtest riot -v run --pass-env -s 13c4b39 -- -vv
tests/openfeature/test_provider_fixtures.py
Expand Down Expand Up @@ -81,7 +84,7 @@ jobs:
repository: DataDog/dd-trace-php
runtime: php
submodule: tests/FeatureFlags/ffe-system-test-data
setup: ":"
setup: sudo apt-get update && sudo apt-get install -y libcurl4-openssl-dev
evaluation: >-
make test_c TESTS=tests/ext/ffe/system_test_data_evaluate.phpt
MAX_TEST_PARALLELISM=1
Expand Down
33 changes: 33 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,39 @@ configuration, and return the caller default with `ERROR` / `PARSE_ERROR` when
those keys are evaluated. Reserve `ERROR` / `FLAG_NOT_FOUND` for keys absent from
both the active and rejected maps. Replace both maps atomically on refresh.

### Updating Targeting Regex Conformance

The standalone targeting regex contract lives in
`regex-conformance/targeting-regex-conformance.json`. Do not put it in
`evaluation-cases/`; consumers parse every JSON file there as a full UFC
evaluation case.

When changing the contract:

1. Bump `schemaVersion` only for an incompatible schema change and bump
`contractVersion` when portable behavior changes.
2. Keep case IDs stable. Add a new ID instead of changing the meaning of an
existing case.
3. Include every required case field: `id`, `description`, `category`,
`contract`, `rawPattern`, `normalizedPattern`, `expectedCompile`, `input`,
and `expectedMatch`.
4. Use `contract` for the portable authoring decision. Use `expectedCompile`
and `expectedMatch` only for common native-engine observations. Set a common
field to `null` and add `engineExpectations` when native engines differ.
5. Verify accepted cases in every shipped SDK evaluator. Rejected cases may be
accepted by a native compiler; that does not make them part of the portable
contract or require an SDK production change.
6. Recompute `targeting-regex-conformance.sha256` from the exact JSON bytes.

Validate JSON syntax, unique case IDs, required field types, and the hash:

```bash
jq empty ufc-config.json evaluation-cases/*.json regex-conformance/targeting-regex-conformance.json
jq -e -f regex-conformance/validate-targeting-regex-conformance.jq regex-conformance/targeting-regex-conformance.json
regex-conformance/test-validate-targeting-regex-conformance.sh
(cd regex-conformance && shasum -a 256 -c targeting-regex-conformance.sha256)
```

### Modifying Flag Configuration

When adding or modifying flags in `ufc-config.json`:
Expand Down
32 changes: 30 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,13 @@ This repository contains the canonical set of flag configurations and evaluation
```
ffe-system-test-data/
├── ufc-config.json # Master flag configuration (UFC format)
└── evaluation-cases/
└── test-*.json # Evaluation test case files
├── evaluation-cases/
│ └── test-*.json # Evaluation test case files
└── regex-conformance/
├── targeting-regex-conformance.json # FFE authoring and matching contract
├── targeting-regex-conformance.sha256 # SHA-256 of the JSON bytes
├── validate-targeting-regex-conformance.jq # Canonical schema validator
└── test-validate-targeting-regex-conformance.sh # Validator regression tests
```

## Usage
Expand Down Expand Up @@ -106,6 +111,29 @@ The shared fixtures intentionally exclude SDK-specific fields such as `variant`
- **variant**: Derive from the flag configuration in `ufc-config.json` by matching the result value
- **flagMetadata**: Extract from the flag's metadata field in `ufc-config.json`

### Targeting Regex Conformance

`regex-conformance/targeting-regex-conformance.json` is a standalone, versioned
contract for authoring targeting regular expressions in FFE. It is intentionally
outside `evaluation-cases/`; consumers of that directory parse every JSON file
as a complete UFC evaluation case.

The fixture records native observations for four implementations: Go `regexp`,
RE2JS, the Rust rules-based evaluator, and the Rust rkyv evaluator. The accepted
authoring subset is narrower: accepted cases must also evaluate consistently in
the shipped Java, JavaScript, and .NET SDK evaluators. Several SDKs share the
Rust evaluator, so agreement across those SDKs is not evidence from independent
regex engines.

Each regex case has a stable ID, an FFE authoring `contract`, raw and normalized
patterns, native compile observations, an input, and an unanchored match
observation when the modeled engines agree. Go and RE2JS consumers compile
`normalizedPattern`; Rust consumers compile `rawPattern`. Cases with differences
between modeled engines include per-engine expectations. Downstream SDK checks
must require consistent behavior for accepted cases. A native engine accepting
rejected syntax does not change the authoring contract. The adjacent SHA-256
file lets downstream tests detect fixture drift.

## Automated Validation

Pull requests run a blocking static validation check over `ufc-config.json`
Expand Down
12 changes: 8 additions & 4 deletions ci/run-downstream-conformance.sh
Original file line number Diff line number Diff line change
Expand Up @@ -119,7 +119,11 @@ if [[ -n ${GITHUB_STEP_SUMMARY:-} ]]; then
} >>"$GITHUB_STEP_SUMMARY"
fi

# Reflect the proposed fixture result in the check conclusion. Branch protection
# decides whether this advisory check blocks merging; the check itself stays red
# until the downstream implementation accepts the proposed fixtures.
exit "$head_status"
# Fail only when the pull request introduces a downstream regression. Existing
# drift remains visible in the job summary and artifacts without blocking an
# unrelated fixture change.
if [[ $classification == new-regression ]]; then
exit 1
fi

exit 0
39 changes: 39 additions & 0 deletions evaluation-cases/test-case-regex-flag.json
Original file line number Diff line number Diff line change
@@ -1,4 +1,43 @@
[
{
"attributes": {
"comma_regex_case": "xxaaay"
},
"defaultValue": "none",
"flag": "regex-flag",
"result": {
"reason": "TARGETING_MATCH",
"value": "comma-quantifier"
},
"targetingKey": "comma-quantifier",
"variationType": "STRING"
},
{
"attributes": {
"whitespace_regex_case": "prefix exact suffix"
},
"defaultValue": "none",
"flag": "regex-flag",
"result": {
"reason": "TARGETING_MATCH",
"value": "literal-whitespace"
},
"targetingKey": "literal-whitespace",
"variationType": "STRING"
},
{
"attributes": {
"whitespace_regex_case": "exact"
},
"defaultValue": "none",
"flag": "regex-flag",
"result": {
"reason": "DEFAULT",
"value": "none"
},
"targetingKey": "trimmed-whitespace-does-not-match",
"variationType": "STRING"
},
{
"attributes": {
"email": "user.name+tag@capture.example"
Expand Down
Loading
Loading