Skip to content

faircode benchmark leaks a raw traceback for a missing manifest path #616

Closed

Description

@yakew7

Where: faircode/cli.py's benchmark handler only catches ImportError/ValueError around run_benchmark(); run_benchmark() itself (faircode/benchmark.py) wraps each manifest's processing in except (yaml.YAMLError, KeyError, ValueError) specifically "to avoid a raw, manifest-path-less traceback all the way to the CLI" per its own comment - but load_manifest() (faircode/manifest.py) does a bare open(path), raising FileNotFoundError, which isn't in that tuple.

Repro:

$ python3 -m faircode.cli benchmark nonexistent/audit.yaml
Traceback (most recent call last):
  ...
  File ".../faircode/manifest.py", line 211, in load_manifest
    with open(path) as f:
FileNotFoundError: [Errno 2] No such file or directory: 'nonexistent/audit.yaml'

Every other file-reading path in this CLI (_read_or_exit) catches FileNotFoundError cleanly with a error: file not found: ... message - this is the one path that doesn't, breaking the exact promise run_benchmark()'s own comment makes.

Fix direction: add FileNotFoundError to run_benchmark()'s per-manifest except tuple (or to the CLI handler's), converting it to the same clean error: ... style as everywhere else.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workinggood first issueGood for newcomers

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions