Hi, EvalAI Team.
We attempted to create a new challenge using github and encountered some issues: If we create a challenge branch and then modify the leaderboard or phase in challenge_config.yaml , will report an error:
ERROR: Leaderboard 2 doesn’t exist. Addition of new leaderboard after challenge creation is not allowed.
ERROR: Challenge phase split (leaderboard_id: 1, challenge_phase_id: 1, dataset_split_id: 2) doesn’t exist. Addition of challenge phase split after challenge creation is not allowed.
And the error will still be reported, Even if we make modifications in the main branch before creating a challenge branch, Which means we cannot create leaderboard and phase other than templates.
Please guide us to figure it out.