English | 简体中文
MedDefer is a research codebase for uncertainty-aware medical agents and selective deferral. The dissertation's main study samples plans before the first tool call, then tests whether disagreement ranks failures and improves retained task success at a fixed review budget.
The repository also retains earlier medical QA/RAG pipelines and independent pre-action Agent Failure Detection (AFD) experiments. These extensions are separate from the dissertation's offline referral evaluation. This is research software, not a clinical decision-making system.
The main implementation is
experiments/medagentbench_v2_AFD/.
It uses official MedAgentBench v2 tasks and the new_refsol evaluator with an
adapted OpenAI-compatible runner. Job scripts pin the upstream harness to
46c93f39bd8679af35790eaaa5dd41bc067024a0.
| Backbone | Official task success | Canonical plan-only analysis |
|---|---|---|
| Qwen3.5-9B | 134/300 (44.7%) | 290 tasks |
| Gemma4-12B | 121/300 (40.3%) | 299 tasks |
| Llama-3.1-8B | 108/300 (36.0%) | 295 tasks |
Each retained task has eight canonical candidate strings: recorded prose where
available, otherwise deterministic tool-action representations. Canonical
candidate counts are distinct from raw prose availability. See
build_canonical_plans.py.
The comparison covers NLI, BGE, METEOR, discrete semantic entropy, graph/kernel entropy and direct pairwise disagreement. Metrics include pooled failure AUROC, failure AUPRC, within-family AUROC, patient-grouped out-of-fold threshold selection, patient-cluster bootstrap intervals and retained task success.
Comparator rankings depend on the backbone, and task-family composition can inflate pooled discrimination. Good failure ranking does not establish clinical safety. The out-of-fold family prior is a control; its within-family fold-noise outputs must not be treated as valid within-family AUROC.
Use Python 3.12 and uv.
git clone https://github.com/Stwinklein/MedDefer.git
cd MedDefer
uv sync --extra dev
uv run pytest -qTests cover the core packages and AFD unit/smoke paths without starting model
servers, FHIR containers or Slurm jobs. Generating new NLI/BGE/METEOR scores
needs the additional packages in
requirements_technical.txt.
Use a separate experiment environment:
uv venv --python 3.12 .venv-afd
uv pip install --python .venv-afd -r experiments/medagentbench_v2_AFD/requirements_technical.txt
# Linux/macOS
source .venv-afd/bin/activate
# Windows PowerShell: .venv-afd\Scripts\Activate.ps1Compact numeric artifacts for the three canonical analyses are included in
results/.
They retain synthetic benchmark task/patient identifiers but contain no raw EHR
context, complete agent transcripts or model weights.
Run from the repository root:
uv run python Dissertation/build_dissertation_assets.py \
--skip-run-dependent --output-dir artifacts/dissertationThis rebuilds the main/deferral LaTeX tables, four-panel rejection curves, primary-versus-pooled AUROC figure and numeric JSON. The best-backbone panel uses distinct colors, markers and line styles.
Additional family-difficulty, protocol and worked-example figures:
# Bash; PowerShell: $env:DISSERTATION_FIGURE_DIR = "artifacts/dissertation/Images"
export DISSERTATION_FIGURE_DIR=artifacts/dissertation/Images
uv run python Dissertation/make_quant_figs.py
uv run python Dissertation/make_pipeline_fig.py
uv run python Dissertation/make_qualitative_fig.pyThe dataset-composition figure needs the upstream new_patient_tasks.json.
Full prose-stability plots and numeric macros need original run artifacts.
See Dissertation input requirements.
With a completed benchmark run and the experiment environment activated:
python -m experiments.medagentbench_v2_AFD.build_canonical_plans \
--run-dir experiments/medagentbench_v2_AFD/runs/<completed_run> \
--output-dir experiments/medagentbench_v2_AFD/runs/<completed_run>_canonical
python -m experiments.medagentbench_v2_AFD.evaluate_pairwise_protocol \
--run-dir experiments/medagentbench_v2_AFD/runs/<completed_run>_canonical \
--output-dir experiments/medagentbench_v2_AFD/artifacts/<analysis> \
--input-condition plan-only --methods nli,bge,meteor \
--model-registry /absolute/path/to/locked_model_revisions.json \
--bootstrap 2000 --random-state 20260730The registry records immutable comparator revisions and local snapshot paths.
Use prepare_technical_models.py --help for setup; retain the original locked
revisions when reproducing an existing study.
Benchmark generation additionally needs model serving and the upstream FHIR
environment. Cluster entry points include submit_b1_plancompact.sh,
submit_gemma4_12b_full.sh and submit_llama31_8b_l40s.sh; inspect their
configuration before submitting jobs. Sequence-aware comparisons,
incremental-value audits, memory and online Pre-Act intervention are independent
extensions. Historical commands remain in the
AFD experiment notes.
- V1: MedRAG answer-or-defer code; start with
uv run meddefer-v1 doctorand the V1 configuration. - V2: multi-signal evaluator and LangGraph agent.
- V3: reusable AFD fusion/policy code;
uv run meddefer-v3-benchmark --help. The default benchmark consumes previously generated MedAgentBench/PathVQA records and skips missing inputs. It is not the dissertation's three-backbone evaluation. - Earlier AgentUQ medical experiments and uncertainty extraction.
The code audit documents missing GitHub code and this release's additions. It publishes project-authored code, prompts, configuration, job scripts, tests, figure builders and selected numeric results. Raw runs, inference caches, credentials, downloaded third-party repositories, meeting recordings and personal submission documents remain local.
Set API credentials through environment variables. For each run, record task/evaluator revision, model revision, decoding settings, prompt hash, sample count, split, threshold-selection rule and code revision. Archived manifests retain their original experiment identities; the publication commit is a separate release revision.