How to Solve Submission Status Failed

1

Hi, EvalAI Team.

We created a new challenge " NumerSense Challenge." We have some problems that 1)the results are not shown on the leaderboard and 2) the submission status is always “Failed.”

The stderr file says
ORGINIAL EXCEPTION: No such relation between Challenge Phase and DatasetSplit specified by Challenge Host
Traceback (most recent call last):
File “/code/scripts/workers/submission_worker.py”, line 513, in run_submission
dataset_split__codename=split_code_name,
File “/usr/local/lib/python3.7/site-packages/django/db/models/manager.py”, line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File “/usr/local/lib/python3.7/site-packages/django/db/models/query.py”, line 408, in get
self.model._meta.object_name
challenges.models.ChallengePhaseSplit.DoesNotExist: ChallengePhaseSplit matching query does not exist.

The result files seem like what we expected. We are not sure how to solve this issue.
Please guide us to figure it out.

Thank you!

2

Hi @seyeonlee,

The error suggests that you are not passing the results from evaluation script to EvalAI server correctly. You can refer to an example here. We expect result from the evaluation script to be a dictionary with “result” key having array of jsons that have results for each challenge phase split. I went through the challenge evaluation script. And I saw you are returning

output["result"][0][phase_key] instead of ouput["result"][0]. Another issue is that the key you are using in the result dict for phase codename is incorrect. We expect the key used in result dict to match with phase code name. In the evaluation script we pass correct phase_codename as a parameter, you can use that instead of hard coding the key. Let me know if you face any issues after changing result keys to phase codename.