[Bug] Official inference/evaluation pipeline does not consume generated outputs

[botwu]

Summary

On current main (49b73a94775fb489063f60ca1865e3a650079a79), the
official end-to-end pipeline cannot reliably evaluate generated workbooks:

1. evaluation/evaluation.py compares each answer workbook against the
   original *_input.xlsx; the intended model *_output.xlsx path is
   commented out.
2. inference/inference_single.py writes conversation records under
   inference/outputs/, but run_solution() reads them from
   data/<dataset>/outputs/.
3. The evaluator defaults to all_data_912, while the currently published
   archive extracts to all_data_912_v0.1.

These are source-level path/selection issues and reproduce without calling a
model API or using LibreOffice.

The local checkout was clean at the upstream commit before reproduction:

HEAD        49b73a94775fb489063f60ca1865e3a650079a79
origin/main 49b73a94775fb489063f60ca1865e3a650079a79

Reproduction: evaluator reads inputs instead of model outputs

After extracting data/spreadsheetbench_912_v0.1.tar.gz:

mkdir -p outputs
cd evaluation
python3 evaluation.py --dataset all_data_912_v0.1 --setting single --model repro

This command evaluates the original input files even when
data/all_data_912_v0.1/outputs/single_repro/ does not exist.

The current source constructs:

gt_path = f"{dataset_path}/spreadsheet/{data['id']}/{test_case_idx + 1}_{data['id']}_answer.xlsx"
proc_path = f"{dataset_path}/spreadsheet/{data['id']}/{test_case_idx + 1}_{data['id']}_input.xlsx"
# proc_path = f"{dataset_path}/outputs/{opt.setting}_{opt.model}/{test_case_idx + 1}_{data['id']}_output.xlsx"

I also instrumented the unmodified HEAD evaluator to record every path
passed to compare_workbooks(). It made 2,736 comparisons for the 912-task
dataset. The first processed workbook was:

.../data/all_data_912_v0.1/spreadsheet/13-1/1_13-1_input.xlsx

and none of the processed paths pointed to the model output directory.

This also appears to be a source regression rather than a local setup issue.
git log -p shows that commit
03242e9
changed proc_path from the model-output workbook to the input workbook and
commented out the former output path. The same input path is still present in
the current
evaluation.py.

Expected behavior

The evaluator should compare each *_answer.xlsx against:

data/<dataset>/outputs/<setting>_<model>/<case>_<id>_output.xlsx

If a model output is missing, that test case should score zero rather than
silently evaluating the original input.

Reproduction: inference conversation path mismatch

gen_solution() writes:

with open(f'outputs/conv_single_{opt.model}.jsonl', 'a+') as fp:

but run_solution() immediately reads:

with open(f'{dataset_path}/outputs/conv_single_{opt.model}.jsonl', 'r') as fp:

Using a one-task dataset with the model/executor calls mocked, the unmodified
source produced:

generation_record_written=True  inference/outputs/conv_single_repro.jsonl
run_solution_read_target_exists=False  data/mini/outputs/conv_single_repro.jsonl
run_solution_error=FileNotFoundError

The same mismatch exists in inference_multiple.py.
It is also present in the current
inference_single.py.

Suggested fix

• Restore the model-output proc_path in evaluation.py.
• Use one consistent directory for conversation JSONL records in both
  generation and replay.
• Change the full-dataset default to all_data_912_v0.1, or accept an explicit
  dataset path instead of relying on a stale directory name.
• Consider enumerating the test cases present for each task instead of assuming
  exactly three.

Environment

• Repository: RUCKBReasoning/SpreadsheetBench
• Commit: 49b73a94775fb489063f60ca1865e3a650079a79
• Dataset archive: data/spreadsheetbench_912_v0.1.tar.gz
• Reproduced on macOS arm64 with Python 3.9; the path-selection bugs are
  platform-independent.

Related issues checked

• Issues in requirements.txt #6 covers the incompatible pandas/numpy requirements pins.
• Evaluation issues: 41 input files already satisfy evaluation! #7 covers dataset tasks whose original input already passes at the annotated
  answer range, not the evaluator selecting inputs for every task.
• more incorrect filenames in 912 dataset #16 and missing xlsx files in 912 dataset #18 cover incorrect/missing filenames in the 912-task dataset.
• Evaluation bug: formulas written by openpyxl/pandas aren’t calculated, causing false negatives #31 covers formula recalculation.

I could not find an existing issue covering the evaluator's proc_path or the
conversation JSONL write/read mismatch.