Submitted models is not tested

RoboTHOR Challenge 2020
1

Hi, I have a quick question about the testing.

My last submission is on May 18, 2020 11:54:59 PM and the status is still ‘Submitted’ and no error message returned. Therefore I can not submit new models. I was wondering if there was something wrong with my submission.

The submitted file link is

{"submitted_image_uri": "937891341272.dkr.ecr.us-east-1.amazonaws.com/robothor-challenge-2020-558-participant-team-4425:ad338774-f701-4899-9046-4125b20272e7"}

Any suggestion will be helpful. Thanks very much.

2

Sorry for the delay. The submission has been processed. One issue I ran into with your submission was the version of werkzeug was not high enough (this was previously not defined). I have added the min version required to the robothor-challenge requirements.txt in the repo. Before your next submission please pull from master.

Eric

3

Thanks for your help.

4

Sorry to reopen this topic. I have submitted a model for the “Test Challenge Sim RGB” phase on Jun 11, 2020 9:38:22 PM. The status is still Submitted. Could you please help me to confirm that everything is going well with my submission? Thanks very much.

The submitted file link is

{"submitted_image_uri": "937891341272.dkr.ecr.us-east-1.amazonaws.com/robothor-challenge-2020-558-participant-team-4425:7d62557f-6762-486d-9a86-f0b132789467"}
5

Your best performing public submission has been submitted on your behalf to the challenge phase. Evaluation is still running (it was started ~2 days ago) - it has finished about 60% of the episodes. Do you know if your model uses the GPU it appears to be idle on our evaluation server?

thank you,
eric

6

Thanks for your help. Our model has two parts – frame feature extraction and action prediction. The first part is running on Tensorflow using CPU which I think leads to the low inference speed. (The second part is running on PyTorch using GPU.) I think I underestimated the consequences of slow reasoning. Sorry for the trouble.

7

Hi，would you please help me to check if the mode testing process finished? Thanks very much.

8

Your model is 96% complete. It should complete within the next 12 hours.