How to Solve Submission Status Failed and result now showing in the leaderboard

Hi, EvalAI Team and everyone in the community.

I am sorry if I am writing a long one. But I feel like I’ve been stuck for days, so I just thought that maybe you should see all the configuration that I made.
So I cloned evalai and deployed it on my ssh server. I made some “testing” challenges through the django admin panel 8000 to learn about the challenges. And I met some problems that have been making me stuck for days. 1) the results are not shown on the leaderboard although it is scored in the submission and 2) the submission status is always “Failed.”

The stderr filed shows:
ORGINIAL EXCEPTION: No such relation between Challenge Phase and DatasetSplit specified by Challenge Host
Traceback (most recent call last):
File “/code/scripts/workers/submission_worker.py”, line 542, in run_submission
dataset_split__codename=split_code_name,
File “/usr/local/lib/python3.7/site-packages/django/db/models/manager.py”, line 82, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File “/usr/local/lib/python3.7/site-packages/django/db/models/query.py”, line 408, in get
self.model._meta.object_name
challenges.models.ChallengePhaseSplit.DoesNotExist: ChallengePhaseSplit matching query does not exist.

Here is the function evaluate in the “testing” evaluation scripts that I made.

def evaluate(test_annotation_file ,user_annotation_file, phase_name, **kwargs):
score = 0
finalScore = 0
test = prepareArray(test_annotation_file)
user = prepareArray(user_annotation_file)
total = len(user)
for element in user:
if (findAinB(element, test)):
score+=1
finalScore = (score/total * 100)

output = {}
if phase_name == "random":
    try:
        output['result'] = [
            {
                'split1': {
                    'score': finalScore,
                }
            }
        ]
        output['submission_metadata'] = "This submission metadata will only be shown to the Challenge Host"            output['submission_result'] = output["result"][0]["split1"]
        return output
    except Exception as e:
        sys.stderr.write(traceback.format_exc())
        return e

if phase_name == "final":
    try:
        output['result'] = [
            {
                'split1': {
                    'score': finalScore,
                }
            },
            {
                'split2': {
                    'score': finalScore,
                }
            }
        ]
        output['submission_metadata'] = "This submission metadata will only be shown to the Challenge Host"
        output['submission_result'] = output["result"][0]
        return output
    except Exception as e:
        sys.stderr.write(traceback.format_exc())
        return e

Heres the challenge phase config in the django admin panel:


And here’s the leaderboard schema
{“labels”: [“score”], “default_order_by”: “score”}

I cannot put all the configuration here since it only allows me to put up to 1 photos. So in case you wanna see all the screenshots that I made in the django admin panel: you can open the docs that I made :

I hope you can guide and tell me the steps to solve the issue.
Thank you so much!

1 Like