To test the evaluation pipeline, I have submitted a solution but the result file is None (see attached file).
Is there a way to find if the submission was successful? I don’t see any other failure in the information provided.
Is the evaluation for a given solution going to take time? If yes, how long?
WIll the scores of each scenario out of three? Are there going to be more information about the evaluation of performance part from scores like the navigation sequences?
Hi vishsangale,
Thanks for your submission. Your submission has been evaluated and you can check stderr and stdout. In the future we will also be providing video sequences of robot navigating for debugging purposes.
The evaluation should be automatic and minival takes about 15min, other phases takes a few hours. We take the total score to rank our participants.
We haven’t started evaluating for other phases apart from minival yet (this serves as a warm-up and sanity check for our participants) and will be starting doing that this week. Are you currently using our baseline docker? That was trained with a slightly different robot configuration so the performance is not great, we will be providing a better one soon! But you can take advantage of the code and data provided in the starter package to train your own awesome policy!
Thank you for the information.
Yes, I did use the baseline docker with SAC algorithm to test the evaluation pipeline on evalai. Interestingly, scores in all the scenarios are zero. Looking at stdout and stderr, there are no obvious failures in the whole pipeline at least, but weird that all scores are zero for the different robot configuration.
Video sequences of robot navigating for debugging purposes will be awesome.
Thanks
Hi,
I’ve submitted 3 solutions (2 yesterday, 1 ~30 mins ago), but it seems like none of my submissions have been evaluated. Is there something wrong on my end?
Thanks
They are evaluating now, sorry for the inconvenience. (evaluation server died earlier yesterday due to running out of disk space)
Looks like your submissions are fine and passed evaluation without problems.
When will the Challenge in Simulation phase be evaluated? I submitted a solution but it seems like my solution still hasn’t been evaluated.
hi, when will the evaluation phase start?
Thank you for your patience, the results are available now. And from now till the end of the challenge, the new submissions will be evaluated daily.
Hi @fxia22, are all submissions supposed to be evaluated by now? None of my submissions show up (submitted May 7 and May 9).
Also could you clarify how much computation time is given in the simulation phase? My current solution is computation heavy, so im concerned whether it will complete evaluation in time.
Hi @karkus Your policy finished evaluation, but looks like the docker didn’t exit after finishes running, I had to manually upload your results. As for computation time, I give some details in this post.
Great thanks for your help.
I’m not sure why docker didn’t exit. I have submitted the exact same container to the minival phase and it worked fine there. I will let you know if this happens again
