Firstly, thank you for your project!
When I run
python task/NDH/eval.py , an error occurs:
I’ve already read https://evalai.cloudcv.org/web/challenges/challenge-page/463/submission. There is no
test split provided by https://github.com/mmurray/cvdn/blob/master/tasks/NDH/data/download.sh but
test_cleaned split. Could you tell me the details about how to evaluate and submit the result of
We don’t release the test set and instead do test set evaluation on the EvalAI server. Locally, you should evaluate using the val_seen and val_unseen splits. I’ve updated the eval.py script to reflect that policy, thanks for pointing the error out!
The test_cleaned.json file has everything you need to generate a test submission but it should be renamed to test.json (I just pushed a change for that too). Then you can look at this function for an example of how to train and generate a test submission: https://github.com/mmurray/cvdn/blob/master/tasks/NDH/train.py#L119-L144
The result is a JSON file which you can submit to the EvalAI server to evaluate against the test split.
Hope that helps. Let me know if you have any other questions.
Thank you for your reply! @mmurray
However, when I use test_submission(), a similar error is reported.
Do you have any suggestions?
I think I see the issue and I submitted a patch here: https://github.com/mmurray/cvdn/commit/d9701b814d10f56f2f5f5df9afb58049fd0b66f9
I believe this resolves the issue so please pull the latest code and try again. Thanks again for pointing the error out!