VQA 2021 Test Dev Submission Kept Failing

Whenever I submitted my answers, it kept saying:

Traceback (most recent call last):
  File "/code/scripts/workers/submission_worker.py", line 538, in run_submission
    submission_metadata=submission_serializer.data,
  File "/tmp/tmpcuzph_cj/compute/challenge_data/challenge_830/main.py", line 202, in evaluate
    prepare_objects(annFile, resFile, phase_codename)
  File "/tmp/tmpcuzph_cj/compute/challenge_data/challenge_830/main.py", line 109, in prepare_objects
    vqaRes = vqa.loadRes(res, resFile)
  File "/tmp/tmpcuzph_cj/compute/challenge_data/challenge_830/vqa.py", line 170, in loadRes
    'Results do not correspond to current VQA set. Either the results do not have predictions for all question ids in annotation file or there is atleast one question id that does not belong to the question ids in the annotation file. Please note that this year, you need to upload predictions on ALL test questions for test-dev evaluation unlike previous years when you needed to upload predictions on test-dev questions only.'
AssertionError: Results do not correspond to current VQA set. Either the results do not have predictions for all question ids in annotation file or there is atleast one question id that does not belong to the question ids in the annotation file. Please note that this year, you need to upload predictions on ALL test questions for test-dev evaluation unlike previous years when you needed to upload predictions on test-dev questions only.

I’ve double checked all my answers, and it matches the required format and have the exact number of questions. and I’m still getting that problem.

I even just directly load all V2 test dev questions into json and created a mock answer for each question, e.g. each questions will have the same “foobar” answer, so I’ve guaranteed that at least the question id all match. But I still get the same error after submitting that mock output file.

is the test dev completely broken? Can someone help unblock me?

Thanks

Thanks,

OK. I figured it out. The error message was incorrect.

It said

Please note that this year, you need to upload predictions on **ALL test** questions
for test-dev evaluation unlike previous years when you needed to upload predictions on test-dev questions only.

It should actually be:

Please note that this year, you need to upload predictions on **ALL test-std** questions
for test-dev evaluation unlike previous years when you needed to upload predictions on test-dev questions only.

Once I include the all ** 447793 ** questions. it finished without issues.