Currently, should we still evaluate on submission_all_questions.json of GQA version 1.3?
Sorry to bother, did you submitted the submission_all_questions.json v1.3 and get nonzero results?
I haven’t tried yet. I am still dealing with validation set and wait for their official test set.
It confuses me whether questionset v1.3 includes their official test set or not, where did you get this version-1.4-dataset infomation? I did’t see this on their website. Sorry to bother again.
I found the information from their readme file in questions1.3, they said “1.4 (coming soon): Introducing new Test and Challenge splits for the GQA challenge 2020.” And from their challenge website, they also describe a new version of test will be released in April.
Dear Participants!
Thank you very much for the interest in the challenge, and I would like to begin by the most honest and sincere apology for replying late. Due to unexpected personal difficulty I couldn’t run the competition during the spring months and wasn’t available online. I sincerely apologize for the inconvenience it may have created.
I’m happy and excited to let you know that the GQA competition has been accordingly rescheduled and will take place over the summer. I’m working currently to release the updated test set that will be used for evaluation (training and dev sets remain the same, as well as the data distribution). See http://visualreasoning.org/challenge.html for further details and timeline, hope to see you there!