I found that each team can only show one submission which have the highest NDCG score on the test-std leaderboard, even if we select “Show on LeaderBoard” for multiple submission results.
It seems more reasonable to show all the selected results on the leaderboard because each submission has its own configuration and its own purpose usually (discriminative or generative, single model or ensemble, etc.). And, according to the information on the visualdialog.org website, the challenge winners will be picked based on not only NDCG but also MRR evaluation metrics.
So, can we show multiple submission results on the leaderboard?
1 Like
Clarification on picking winners — we’ll be using a single submission’s rank across both NDCG and MRR metrics to pick winners.
The idea is for approaches to simultaneously do well on both metrics, instead of each entry doing well on one metric but poorly on the other.
Given the above, being able to make one submission public seems reasonable.
Your point on not being able to easily distinguish between generative and discriminative approaches still stands, and we’re in discussion to figure out a solution for that (e.g. collecting meta-data or having two separate leaderboards).