Skip to content

fix: molfile permutation - #310

Merged
JanCBrammer merged 9 commits into
devfrom
fix-permutation
Nov 10, 2026
Merged

JanCBrammer merged 9 commits into
devfrom
fix-permutation

Conversation

@JanCBrammer

@JanCBrammer JanCBrammer commented Nov 6, 2026 •

Copy link
Copy Markdown
Collaborator

permute_molfile_text called .strip() on the permuted_molfile that was returned by PermuteMolfileText. PermuteMolfileText generates molfiles with an empty first header line. That empty line was removed by the call to .strip() resulting in an invalid molfile. Accordingly, all subsequent reads of the molfile failed. Consequently, the invariance test didn't computed anything since we've introduces permute_molfile_text in August (c612989). We didn't notice because the invariance test didn't catch the errors that resulted from the failure to read the molfiles. Both are fixed now, the invariance test catches errors (see 7064a70) and permute_molfile_text no longer invalidates the molfile (see 3e61dbb). With the fix we're now encountering several invariance failures (see 6f9acc6).

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown

Unit Test Coverage Report

Coverage Regression Summary

Metric PR (%) Base (%) Difference (pp)
Lines 34.3 34.3 +0.00
Branches 22.3 22.3 +0.00

Find details on the base coverage at https://iupac-inchi.github.io/InChI/coverage/index.html

Find details on this PR's coverage by downloading coverage-reports-42817415078 and opening html/index.html

@fbaensch-beilstein fbaensch-beilstein left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, the .strip() fix is right and needed. I checked it over the whole CI set (4190 molfiles, 10 permutations each): with the old wrapper none of the permuted molfiles could be read back, with the new one all can. So the invariance test really did check nothing between c612989 and this PR.

A few things before merge:

  1. Consumer errors never fail the run. They are logged as invariance consumer failed, exit_code stays 0, and parse_log.py only picks up test failed lines, so they don't reach the summary either. If permutation or reading breaks again for every molfile, CI stays green. This already hides a real case: PUBCHEM_COMPOUND_CID-102182973 gives CANON_ERR (-30016) on 8 of 10 permutations and a valid InChI=1S/C32H38N4O4/... on the other 2. Since the consumer breaks at the first error, mixed outcomes like this are dropped. Suggestion: keep going on a make error and record it as a variant (e.g. {"inchi": "", "error": log}), so a mix of errors and InChIs counts as more than one variant, and fail the run on consumer errors unless the ID is in expected_failures. The error branch currently also continues before the expected-failures check.

  2. babf4f7 reverts timeout failures (db39a47). A molfile that has a real result in the reference and now times out is logged and the run exits 0, and test_regression_fails_on_a_timeout_unless_it_is_expected is deleted. INCHI-1-TEST/README.md:166-173, the timeout entry in config_pubchem_substance.py and comparisons/molecular_inorganics/run.py (still JSON, still fails) now disagree with drivers.py, and the new timeout line is no longer JSON. Could you drop this commit from this PR? If you want the change, a separate PR with the reason and those files updated would be easier to review.

  3. The expected-failure list depends on the libc. srand(0) gives different rand() sequences on glibc and musl, so the two CI jobs sample different permutations. From this PR's CI logs: glibc fails chg_O_N_05, NSC-608972, NSC-631477, NSC-683677, NSC-85765-Err_1, NSC-Err_1.004, v1_err_NSC-103019; musl fails the same set with NSC-85765 instead of NSC-683677. Fine for now, but worth a comment in the config, since another platform or seed will give a different list. All of them are InChI=1BB/ organometallics.

  4. No test for the fix. test_permutation.py went away in c612989 and nothing replaced it. One small libinchi-backed test (permute one CI molfile, make InChI, assert rc in {0, 1} and a non-empty InChI) would have caught the July breakage.

  5. Smaller:

    • 3442296542 and 1980967343 only fail to permute because the output buffer is len(moltext) + 1 + 1024; their permuted molfiles are ~9.6–10 kB. A bigger buffer puts them back under test. (iso-H-Bridge is genuinely unreadable.)
    • GetINCHIKeyFromINCHI never returns 11 (inchi_api.h, INCHIKEY_*), so the [0, 11] check is misleading (the wrapper had it before).
    • The assert/json.dumps reflow in drivers.py and inchi_api.py is unrelated; a separate commit would keep the diff readable.

Separate from this PR, found while checking it: PermuteMolfileText reads the input with bDoNotAddH=1 (permutation_util.c:203, the 0, 1, 1, 0, 0 arguments), so implicit hydrogens are lost and about 3600 of 4190 permuted molfiles give a different InChI than the original (methanol CH4O becomes CO). With that argument set to 0, only the 7 real failures above remain. That deserves its own issue.

@JanCBrammer

JanCBrammer commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator Author

@fbaensch-beilstein, I addressed 1. and 2. with 12d4701: The invariance tests now distinguish errors and test failures. Errors (incl. timeouts) are logged and result in an exit code 1 but they aren't logged as test failures. Test failures (more than one distinct InChI string or key) also result in an exit code 1 and are explicitly logged as failures. That is, both errors and failures fail the CI, but only failures count towards the success of the invariance test. A dataset's expected_failures["invariance"] apply only to test failures, not to errors.

I added a test for 4. in 744cd59.

I addressed the remaining comments with 4b5bf0e and ba89ea8.

Note that CI fails due to two remaining errors in the invariance log:

INFO:sdf_pipeline:error:{"time": "2026-11-09T14:10:04", "molfile_id": "iso-H-Bridge", "sdf": "inchi.sdf.gz", "info": {"consumer": "invariance", "parameters": "; number_of_runs=10"}, "error": "Failed to permute molfile (exit code -1)'..."}

INFO:sdf_pipeline:error:{"time": "2026-11-09T14:10:06", "molfile_id": "PUBCHEM_COMPOUND_CID-102182973", "sdf": "inchi.sdf.gz", "info": {"consumer": "invariance", "parameters": "; number_of_runs=10"}, "error": "Error -30016 (CANON_ERR)..."}

I'll open an issue to investigate those.

@fbaensch-beilstein

Copy link
Copy Markdown
Collaborator

@JanCBrammer thanks, 1 and 4 look good, and the bigger buffer puts 3442296542 and 1980967343 back under test. What's left from my side:

  1. CI can't go green as it stands. The latest invariance logs (glibc and musl) have 0 test failures and exactly 2 errors: iso-H-Bridge (can't be permuted; its molfile is genuinely unreadable) and PUBCHEM_COMPOUND_CID-102182973 (CANON_ERR on some permutations). Errors always fail the run and expected_failures doesn't cover them, so this stays red until both are fixed. An issue alone won't unblock the merge. Could we have a way to mark known errors as expected (e.g. expected_errors["invariance"], referencing the issue), so new errors still fail?

  2. Point 2 is only half addressed. Invariance fails on timeouts again, but regression still calls _log_timeout, which logs plain text and exits 0. test_timeout.py (incl. test_regression_fails_on_a_timeout_unless_it_is_expected) is still deleted, and the README no longer says regression fails on a timeout. Could you drop the regression part of babf4f7 from this PR?

  3. The expected-failure list is now stale. With fix: handle empty V3000 bond block #314 on dev, none of the 8 IDs in expected_failures["invariance"] (chg_O_N_05, NSC-608972, …) fail any more, so the list only hides future regressions on them. I'd drop it, which also makes the libc comment I asked for unnecessary.

  4. Smaller:

    • The assert/json.dumps reflow in drivers.py is still in the diff.
    • The timeout line is logged as error: (with a space) and has no time/info, unlike the other error: lines.

On merge order: I'd merge #312 first, so the next CI run here exercises the permutation with implicit hydrogens kept.

@JanCBrammer

JanCBrammer commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator Author

Regarding 1., I'd rather take the two structures that produce the errors out of CI testset than introduce "expected failures". The structures don't meaningfully contribute to either the invariance or regression tests at the moment. See #317.

Regarding 2., I agree that the error handling has to be extended to the regression tests, but I'd rather implement that in a separate PR. See #319.

I implemented 3. with b5d3588, closing #318.

I'd homogenize the error logging as part of 2..

@JanCBrammer
JanCBrammer merged commit fa2e4ac into dev Nov 10, 2026
1 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Remove stale expected CI failures

2 participants