Challenge in Simulation Evaluation

Gibson Challenge 2020
1

Hi,

When will Phase: Challenge in Simulation, Split: Dev Split be evaluated? Will we be able to see our ranking on that leaderboard?

Thank you.

2

Hi @jtruong,

We will evaluate your submissions before the end of this week. Stay tuned.

Yes, you will be able to see your ranking on that leaderboard once your submissions are evlauted.

Thanks!

3

Hello team,

Thank you for guiding me in making the submission in correct format. However, the submission is not scoring as per expected. In my local system, the robot is reaching either to the goal or near the goal most of the time. And getting a minimum score of 0.2 when I use the file test_locally.sh (the default file provided for local evaluation) on the docker container that I have submitted ans_26may. Yesterday I have also addressed the path issue and brought the code out in the root folder, and also the pytorch version issue, ugraded to torch 1.5. Resubmitted the corrections with the name ans_28may. I have not adressed the code running speed yet. However I thought that was a problem with running in a real environment.

Could you please guide me in understanding why I am getting such a low score? Maybe if you could give me the state-action pairs of a few runs, it will be very helpful. Thank you.

4

Hi Ruddra @Ruddra_dev , sorry to hear to about the discrepancy of test_locally and our evaluation. Can you share with me your yaml file used for test_locally (probably through email), there might be some discrepancy in the parameters. Which scene are you evaluating on when doing test_locally? Our evaluation will be on Castro scene.

1 Like
5

Hi Fei, I think I just needed to update the assets folder. I have corrected it and resubmitted. Thank you.

6

Hello @fxia22:

I see my latest submission in Sim phase submitted on Jun 1, 2020 12:39:48 AM is still under execution. Could you please see if it is stuck for some reason? Submission Id - 82048.

Regards,
Ruddra

7

I have started the job and will get back to you when it finishes. Will take ~15h I guess.

-Fei

8

Hi Fei,

It still shows as “Submitted” I hope it isn’t stuck due to some issue? It did pass the minival phase.

9

Hello,

When submitting in challenge-sim it shows the phase is not available anymore. Is it possible to reopen the phase?

10

Hi @Ruddra_dev,

our challenge in simulation phase has ended, and winner will be announced at CVPR. You can still submit to the real phase which is due on June 8. The winner for the real phase will be announced at CVPR too.

1 Like