Submission issues with NuScenes prediction challenge

1

Hi, I’m trying to submit to the NuScenes prediction leaderboard, and one of two things happens:

(1) The submission remains stuck in the ‘submitted’ status

(2) The evaluation script fails with this error:

Traceback (most recent call last):
File “/code/scripts/workers/submission_worker.py”, line 462, in run_submission
submission_output = EVALUATION_SCRIPTS[challenge_id].evaluate(
AttributeError: module ‘challenge_data.challenge_591’ has no attribute ‘evaluate’

The submissions also seem to duplicate in case (2) by themselves and they all eventually fail.

I tried using the file upload and URL options. I haven’t yet been able to see any results for my entry.

Can someone help?

2

Hi @ndeo, this error was happening because of a bug in the evaluation script. We have fixed the evaluation script. For your second question, multiple failed submissions are because they were re-ran. As there was an issue with the evaluation script we re-ran the pending submissions.

3

Thanks for the response! Currently I have a submission stuck in the submitted status. Would you be able to run the evaluation script for it? Or should I resubmit?

4

The evaluation script runs now! Thanks for the fix.

However, now I’m getting this error:

FileNotFoundError: [Errno 2] No such file or directory: ‘/tmp/tmpmwtnom8f/dataset/nuscenes/maps/expansion/singapore-onenorth.json’