How to submit test result to EvalAI

Cooperative Vision-and-Dialog Navigation
1

Hi,

Firstly, thank you for your project!

When I run python task/NDH/eval.py , an error occurs:

800×159 20.6 KB

I’ve already read https://evalai.cloudcv.org/web/challenges/challenge-page/463/submission. There is no test split provided by https://github.com/mmurray/cvdn/blob/master/tasks/NDH/data/download.sh but test_cleaned split. Could you tell me the details about how to evaluate and submit the result of test set?

Thank you

2

Hi Jack_Sun,

We don’t release the test set and instead do test set evaluation on the EvalAI server. Locally, you should evaluate using the val_seen and val_unseen splits. I’ve updated the eval.py script to reflect that policy, thanks for pointing the error out!

The test_cleaned.json file has everything you need to generate a test submission but it should be renamed to test.json (I just pushed a change for that too). Then you can look at this function for an example of how to train and generate a test submission: https://github.com/mmurray/cvdn/blob/master/tasks/NDH/train.py#L119-L144

The result is a JSON file which you can submit to the EvalAI server to evaluate against the test split.

Hope that helps. Let me know if you have any other questions.

Thanks,
Michael

3

Thank you for your reply! @mmurray
However, when I use test_submission(), a similar error is reported.

image

Do you have any suggestions?

Thank you!

4

Hi Jack_Sun,

I think I see the issue and I submitted a patch here: https://github.com/mmurray/cvdn/commit/d9701b814d10f56f2f5f5df9afb58049fd0b66f9

I believe this resolves the issue so please pull the latest code and try again. Thanks again for pointing the error out!

Thanks,
Michael