How to submit test result to EvalAI


Firstly, thank you for your project!

When I run python task/NDH/ , an error occurs:

I’ve already read There is no test split provided by but test_cleaned split. Could you tell me the details about how to evaluate and submit the result of test set?

Thank you

Hi Jack_Sun,

We don’t release the test set and instead do test set evaluation on the EvalAI server. Locally, you should evaluate using the val_seen and val_unseen splits. I’ve updated the script to reflect that policy, thanks for pointing the error out!

The test_cleaned.json file has everything you need to generate a test submission but it should be renamed to test.json (I just pushed a change for that too). Then you can look at this function for an example of how to train and generate a test submission:

The result is a JSON file which you can submit to the EvalAI server to evaluate against the test split.

Hope that helps. Let me know if you have any other questions.


Thank you for your reply! @mmurray
However, when I use test_submission(), a similar error is reported.


Do you have any suggestions?

Thank you!

Hi Jack_Sun,

I think I see the issue and I submitted a patch here:

I believe this resolves the issue so please pull the latest code and try again. Thanks again for pointing the error out!