Description of issue
For several tasks in this dataset, comparing the original input to the provided ground truth spreadsheet at the specified answer_position shows that the spreadsheet is already passing the test without any changes being made.
For example, task 7-5 calls for the editing of row 4 in Sheet1: the issue is that the specified answer position is at the empty row 5 and so the test case ALWAYS passes, even when no changes of any kind have been made!
{
"id": "7-5",
"instruction": "How can I count the number of green and red cells within a specific row (D4:AO4) of an Excel 2010 worksheet, where there are additional conditions that need to be met for the counting? The details of these conditions are illustrated in my provided example workbook. Furthermore, in subsequent data provided, it is indicated that I am seeking to count by color break within two rows of data, and I'm looking for a solution that works with the real dataset.Count Green cells：Condition-1 (before 2 rows are red like L1:L2)，Condition-2 If both row are (0-0) Like G1:G2 & I1:I2 then place 0.If Both Are Red Start Count From 1",
"spreadsheet_path": "spreadsheet/7-5",
"instruction_type": "Sheet-Level Manipulation",
"answer_position": "'Sheet1'!D5:AO5",
"answer_sheet": "Sheet1",
"data_position": "D1:AO4"
},
Furthermore, there is an additional issue in the evaluation code that means that multi-sheet answer positions are being ignored.
For example, task 430-43 calls for three different ranges to be checked.
{
"id": "430-43",
"instruction": "How can I create a macro that allows me to input a key variable in a 'Filter Criteria' tab to filter the data across all other tabs in an Excel worksheet without the need to copy the data?",
"spreadsheet_path": "spreadsheet/430-43",
"instruction_type": "Sheet-Level Manipulation",
"answer_position": "data1'!A1:B5,'data2'!A1:B5,'data3'!A1:B5",
"answer_sheet": "'data1','data2','data3'",
"data_position": "Filter Criteria!A1:B5','data1!A1:B5','data2!A1:B5','data3!A1:B5',"
},
However, because in lines 187-195 of evaluation.py, the cell_level_compare is outside the for-loop, only the third range is actually checked. As a result,
sheet_cell_ranges = answer_position.split(',')
for sheet_cell_range in sheet_cell_ranges:
if '!' in sheet_cell_range:
sheet_name, cell_range = sheet_cell_range.split('!')
sheet_name = sheet_name.lstrip("'").rstrip("'")
else:
sheet_name = wb_gt.sheetnames[0]
cell_range = sheet_cell_range
result, msg = cell_level_compare(wb_gt, wb_proc, sheet_name, cell_range , is_CF)
By automatically comparing each input to the ground truth spreadsheets, I was able to identify several more tasks where tests passes without any changes being made. The full list of 41 problematic tasks (~5% of the dataset) is here:
{'CF_11177', 58114, 'CF_13024', 34435, '298-22', '395-48', 'CF_24632', '131-38', '53-12', 'CF_513', 32023, 52379, '172-10', '7-5', '189-9', '139-16', '73-45', 59643, '260-16', '13-1', '122-39', '575-15', 'CF_357', '248-48', 'CF_3575', 41801, 'CF_13984', '570-43', 'CF_9945', 'CF_21040', 'CF_29431', 48357, 'CF_26100', '164-22', '482-22', 57590, 48378, 10747, 45181, '430-43', '203-33'}
This would clearly inflate benchmarks on this dataset.
Thanks very much.
How to verify
with open(f"{dataset_path}/dataset.json", "r") as fp:
dataset = json.load(fp)
for data in dataset:
test_case_results = []
for test_case_idx in range(1, 4):
result, message = compare_workbooks(
f"{dataset_path}/spreadsheet/{data['id']}/{test_case_idx + 1}_{data['id']}_answer.xlsx",
f"{dataset_path}/spreadsheet/{data['id']}/{test_case_idx}_{data['id']}_input.xlsx",
data["instruction_type"],
data["answer_position"],
)
if result:
print(data)
Description of issue
For several tasks in this dataset, comparing the original input to the provided ground truth spreadsheet at the specified
answer_positionshows that the spreadsheet is already passing the test without any changes being made.For example, task 7-5 calls for the editing of row 4 in Sheet1: the issue is that the specified answer position is at the empty row 5 and so the test case ALWAYS passes, even when no changes of any kind have been made!
Furthermore, there is an additional issue in the evaluation code that means that multi-sheet answer positions are being ignored.
For example, task 430-43 calls for three different ranges to be checked.
However, because in lines 187-195 of
evaluation.py, the cell_level_compare is outside the for-loop, only the third range is actually checked. As a result,By automatically comparing each input to the ground truth spreadsheets, I was able to identify several more tasks where tests passes without any changes being made. The full list of 41 problematic tasks (~5% of the dataset) is here:
This would clearly inflate benchmarks on this dataset.
Thanks very much.
How to verify