Announcement: Challenge in Real is now open

Gibson Challenge 2020
1

Dear Participants:

“Challenge in Real” phase is now open for submission. Are you excited? We know that we are. You can submit your policy, and we will try our best to evaluate it and get back to you within 1 business day with statistics and footage. During this week, we will only be evaluating the “static” scenario.

Your policy used for Challenge in Simulation phase would be a good first policy to submit!

Cheers,
Organizers

1 Like
2

File format explanation:

For the Challenge in Real phase, we will be sharing with the participants more data than the log itself, we will be sharing a google drive folder, with the contents explained as below.

  • stdout: participant’s docker image stdout log
  • stderr: participant’s docker image stderr log
  • state_machine_log: This is the output of our evaluation logic, it provides information like success rate and distance to goal. The participants only need to look at the log printed in the state “Evaluating”.
  • surveillance.mp4: The camera footage of third-person view of robot navigation. We include 4 cameras. You can observe the robot’s behavior from these cameras. When the light on the robot is showing green, it is doing automatic resetting using ros navigation stack, and when the light on the robot is showing red, it is evaluating your policy.
  • *.pickle: a pickle file containing (state,action) pairs of each episode.

We are happy to hear from participants to include additional important information in these data packages.

1 Like
3

Hi Fei,

Regarding the 10Hz control frequency in Real Environment, I wanted to check if it is a requirement that we need to give a response at more than 10Hz or is there just a boundation on the total time of execution? Like if 500 iterations at 10Hz are allowed, can we take 250 steps at 5 Hz for example to reach our goal?

Warm Regards,
Ruddra

4

Hi @Ruddra_dev,

We highly recommend participants to output actions at more than 10Hz because each action will only be executed for exactly 0.1 second. For example, if your policy output actions at 5Hz, the robot will execute your last action for 0.1 second and then “sleep” for 0.1 second, waiting for your next action to arrive.

Let me know if this answers your question. Thanks!

1 Like
5

@chengshu

Yes it does, thank you!

6

If your policy can run at fastest 5Hz, one hack is that you can probably send each command twice (if it is safe to do so). This way you can reduce wasted time caused by sleeping.

7

Yes, thank you for the suggestion. I was thinking on similar lines as that would also prevent jerks to some extent if I send a lower velocity actuation twice.

8

Hi Team,

Could you please let me know if the submission I posted yesterday is executing in the Real Phase? It shows “Submitted”. Also if I could have some idea of how the interactive objects look in the real phase if that is permitted, that would be great. Thank you.

9

Hi,
I submitted 1 job yesterday, and am trying to submit 1 more today before the challenge ends. However, my job yesterday has not been evaluated yet (status is “submitted”). When I try to push my docker, I get the error Error: You have 1 submissions that are being processed. Please wait for them to finish and then try again.. From the instructions, I thought we had 1 submission per day. Is it possible for me to submit my docker?

Thank you.

1 Like
10

Hi @jtruong, sorry we didn’t have time to evaluate your previous submission yet. But I canceled it for now and you can submit a new one and let us know which one you want to count as final submission.

Best,
Fei

11

Hi @fxia22, I’ve submitted a new docker, but will only one submission be evaluated? Since I submitted one yesterday, and am submitting one today, shouldn’t both be evaluated?
Thank you.