Slow evaluation servers unfair to participants

1

This is regarding the LVIS challenge 2021. But could apply to others as well.

The evaluation servers are meant to help us get scores for technical reports and also get some confidence that we are submitting the right format of submissions.

We can see that multiple people are facing the issue of submissions not evaluating/showing “None”.

Is it not possible to have a size limit on the submission file? Or any other guideline we can follow which can assure us that our submission will get evaluated in x time.

This is a critical issue I feel. If you have ever faced this issue as a participant can you comment here? Would help get more support for this issue.

1 Like
2

@xcdh @dasdx @waizei I believe you guys are facing similar issues? @ram81 you do help out everytime when someone creates this issue. But can’t we have a permanent solution?

3

Hi @kamathsutra,

I have fixed the issue for now. Your submissions will be done evaluating soon.

you do help out everytime when someone creates this issue. But can’t we have a permanent solution?

Yes, we are working on it. We know why this issue is happening, we will make a fix for it as soon as possible and update you. The issue is partly due to large input files and partly due to a bug in our cleanup setup. Once we deploy the fix we will let you know.

4

I am not sure if restricting the input size is possible, you’ll have to ask this to challenge hosts. We can try to fix the issue at our end but due to large submission the evaluation time for LVIS challenge is slightly higher even at max compute resources we support at the moment.

5

@ram81 Could you please evaluate our result as well? We are waiting for 1 day until now.

6

Hi @ram81, we are facing the same issue. We are waiting for more than 5 day.

7

Hi @ram81 the issue was not fixed, rather it shows “Cancelled” now :confused:

@xcdh @dasdx are your problems solved?

8

@kamathsutra Nope, looks like he’s offline.

1 Like
9

Hi @kamathsutra @dasdx, we had to re-run all pending submissions after the fix. The temporary fix I made didn’t work. I am looking into the issue, I’ll update on the thread once I have more details.

10

@ram81 we have a deadline in 9 hours…

11

Hi @kamathsutra, I have deployed the fix. Can you share your team name? I can prioritize your submissions. Sorry about the delay from our end

12

CopyPaste++ is our team name

13

@kamathsutra I have prioritized submissions from your team. You’ll see bunch of cancelled submissions, please ignore them. I am closely monitoring our evaluation setup to make sure all submissions get done evaluated soon. If I find any other issues that could cause delays I’ll update on this thread. So far the setup looks good and you should receive your results in couple of hours (LVIS evaluation script takes ~40-60mins per submissions based on our data).

1 Like
14

Hi @ram81, our first result has been submitted for 7 hours, and it is running now. As we will submit the second one, can you help us check whether the results is running normally, and when will it finish running?

15

Hi @flytocc, can you share your team name?

16

@ram81, BUPT-PRIV_ICA is our team name.

17

Hi @flytocc, there was a issue with previous run of your submission with old setup. I have requeued it now. It should be done evaluating in couple of hours

18

Hi @ram81, we has submmitted two results, one is running for about 1 hour, the other one is subbmitted just now, please help us speed up the evalutation, thanks!

19

@ram81
is the deadline extended by a day? It shows Sept 29th for me now.

20

Hi @kamathsutra, only challenge hosts can update the challenge end date. We manage the EvalAI infrastructure. If the challenge end date has changed then the challenge hosts would’ve modified the challenge end date.