Repository navigation
fix: molfile permutation - #310
Conversation
Unit Test Coverage ReportCoverage Regression Summary
Find details on the base coverage at https://iupac-inchi.github.io/InChI/coverage/index.html Find details on this PR's coverage by downloading coverage-reports-42817415078 and opening html/index.html |
6644239 to
6f9acc6
Compare
fbaensch-beilstein
left a comment
There was a problem hiding this comment.
Thanks, the .strip() fix is right and needed. I checked it over the whole CI set (4190 molfiles, 10 permutations each): with the old wrapper none of the permuted molfiles could be read back, with the new one all can. So the invariance test really did check nothing between c612989 and this PR.
A few things before merge:
-
Consumer errors never fail the run. They are logged as
invariance consumer failed,exit_codestays 0, andparse_log.pyonly picks uptest failedlines, so they don't reach the summary either. If permutation or reading breaks again for every molfile, CI stays green. This already hides a real case:PUBCHEM_COMPOUND_CID-102182973gives CANON_ERR (-30016) on 8 of 10 permutations and a validInChI=1S/C32H38N4O4/...on the other 2. Since the consumerbreaks at the first error, mixed outcomes like this are dropped. Suggestion: keep going on a make error and record it as a variant (e.g.{"inchi": "", "error": log}), so a mix of errors and InChIs counts as more than one variant, and fail the run on consumer errors unless the ID is inexpected_failures. Theerrorbranch currently alsocontinues before the expected-failures check. -
babf4f7 reverts timeout failures (db39a47). A molfile that has a real result in the reference and now times out is logged and the run exits 0, and
test_regression_fails_on_a_timeout_unless_it_is_expectedis deleted.INCHI-1-TEST/README.md:166-173, the timeout entry inconfig_pubchem_substance.pyandcomparisons/molecular_inorganics/run.py(still JSON, still fails) now disagree withdrivers.py, and the new timeout line is no longer JSON. Could you drop this commit from this PR? If you want the change, a separate PR with the reason and those files updated would be easier to review. -
The expected-failure list depends on the libc.
srand(0)gives differentrand()sequences on glibc and musl, so the two CI jobs sample different permutations. From this PR's CI logs: glibc failschg_O_N_05, NSC-608972, NSC-631477, NSC-683677, NSC-85765-Err_1, NSC-Err_1.004, v1_err_NSC-103019; musl fails the same set withNSC-85765instead ofNSC-683677. Fine for now, but worth a comment in the config, since another platform or seed will give a different list. All of them areInChI=1BB/organometallics. -
No test for the fix.
test_permutation.pywent away in c612989 and nothing replaced it. One small libinchi-backed test (permute one CI molfile, make InChI, assert rc in {0, 1} and a non-empty InChI) would have caught the July breakage. -
Smaller:
3442296542and1980967343only fail to permute because the output buffer islen(moltext) + 1 + 1024; their permuted molfiles are ~9.6–10 kB. A bigger buffer puts them back under test. (iso-H-Bridgeis genuinely unreadable.)GetINCHIKeyFromINCHInever returns 11 (inchi_api.h,INCHIKEY_*), so the[0, 11]check is misleading (the wrapper had it before).- The assert/
json.dumpsreflow indrivers.pyandinchi_api.pyis unrelated; a separate commit would keep the diff readable.
Separate from this PR, found while checking it: PermuteMolfileText reads the input with bDoNotAddH=1 (permutation_util.c:203, the 0, 1, 1, 0, 0 arguments), so implicit hydrogens are lost and about 3600 of 4190 permuted molfiles give a different InChI than the original (methanol CH4O becomes CO). With that argument set to 0, only the 7 real failures above remain. That deserves its own issue.
|
@fbaensch-beilstein, I addressed 1. and 2. with 12d4701: The invariance tests now distinguish errors and test failures. Errors (incl. timeouts) are logged and result in an exit code 1 but they aren't logged as test failures. Test failures (more than one distinct InChI string or key) also result in an exit code 1 and are explicitly logged as failures. That is, both errors and failures fail the CI, but only failures count towards the success of the invariance test. A dataset's I added a test for 4. in 744cd59. I addressed the remaining comments with 4b5bf0e and ba89ea8. Note that CI fails due to two remaining errors in the invariance log: I'll open an issue to investigate those. |
|
@JanCBrammer thanks, 1 and 4 look good, and the bigger buffer puts
On merge order: I'd merge #312 first, so the next CI run here exercises the permutation with implicit hydrogens kept. |
|
Regarding 1., I'd rather take the two structures that produce the errors out of CI testset than introduce "expected failures". The structures don't meaningfully contribute to either the invariance or regression tests at the moment. See #317. Regarding 2., I agree that the error handling has to be extended to the regression tests, but I'd rather implement that in a separate PR. See #319. I implemented 3. with b5d3588, closing #318. I'd homogenize the error logging as part of 2.. |
b5d3588 to
93c0bb4
Compare
permute_molfile_textcalled.strip()on thepermuted_molfilethat was returned byPermuteMolfileText.PermuteMolfileTextgenerates molfiles with an empty first header line. That empty line was removed by the call to.strip()resulting in an invalid molfile. Accordingly, all subsequent reads of the molfile failed. Consequently, the invariance test didn't computed anything since we've introducespermute_molfile_textin August (c612989). We didn't notice because the invariance test didn't catch the errors that resulted from the failure to read the molfiles. Both are fixed now, the invariance test catches errors (see 7064a70) andpermute_molfile_textno longer invalidates the molfile (see 3e61dbb). With the fix we're now encountering several invariance failures (see 6f9acc6).