Please find the evaluation metrics for semantic segmentation.
The winning entry will be evaluated using the performance metric 0<M<=100, defined as follows: where 0<P<=1, measures model-accuracy as defined by the unweighted mean intersection-over union score over all classes for the test-set, and S>0, measures model-complexity, as defined by the number of model parameters, measured in unit of model-size in MB. More precisely, model-size in MB will be computed as follows: S = number of learned model parameters x 4 (floating point) / (1024*1024)
Please find common submission errors and corresponding solutions below.
assert pred.shape == gt.shape
This happens for cases where #masks / segmentation of images is different from #masks from ground truth. This usually means that the #masks you generated doesnât equal to ground truth. Please go back and check your results.
raise ValueError(âInvalid submission zip!â) ValueError: Invalid submission zip!
This has been fixed now and please re-submit your results for evaluation.
Challenge participation deadline: September 15, 2019
Clarifications on metrics below-
GA: Global Pixel Accuracy
CA: Mean Class Accuracy for different classes
Back: Background (non-eye part of peri-ocular region)
Sclera: Sclera
Iris: Iris
Pupil: Pupil
Precision: Computed using sklearn.metrics.precision_score(pred, gt, âweightedâ)
Recall: Computed using sklearn.metrics.recall_score(pred, gt, âweightedâ)
F1: Computed using sklearn.metrics.f1_score(pred, gt, âweightedâ)
IoU: Computed using the function below
def compute_mean_iou(flat_pred, flat_label):
'''
compute mean intersection over union (IOU) over all classes
:param flat_pred: flattened prediction matrix
:param flat_label: flattened label matrix
:return: mean IOU
'''
unique_labels = np.unique(flat_label)
num_unique_labels = len(unique_labels)
Intersect = np.zeros(num_unique_labels)
Union = np.zeros(num_unique_labels)
for index, val in enumerate(unique_labels):
pred_i = flat_pred == val
label_i = flat_label == val
Intersect[index] = float(np.sum(np.logical_and(label_i, pred_i)))
Union[index] = float(np.sum(np.logical_or(label_i, pred_i)))
mean_iou = np.mean(Intersect / Union)
return mean_iou
Regarding the evaluation metrics, please find our comments below:
Our interest in designing the measure is to meet some practical considerations, ie. any model less than or equal to 1MB in model size, is good enough for our purposes.
Amongst all the models that satisfy 1MB or less model-size constraint, mIOU becomes the key deciding factor.
======== Notification of Paper Submission ========
This is a notification that the deadline of paper submission to ICCV workshop regarding OpenEDS challenges has changed.
Please note that there is a two tier submission, and the major difference with early deadline is that the accepted paper will appear in ICCV workshop proceedings. See table below for the two tiers.
======== Common questions regarding paper submission ========
The challenge deadline remains the same, , which will close on Sept 15th.
Deadline to submit paper is stated in above table.
Specifically, (1) the first deadline is for folks who want to submit manuscript and are interested in seeing their paper (if accepted) appear in ICCV workshop proceedings; (2) the second deadline is for folks who are not ready to submit paper or do not necessarily care about the paper appearing in ICCV workshop proceedings.
For challenge participants they are welcome to submit papers to meet either of the two deadlines.
For winners, we expect the authors to submit paper, which are considered accepted, i.e the challenge winning papers will not undergo formal review if submitted after the challenge ends.
All submitted papers will be considered for publication. Accepted papers will also have an opportunity to present.
With regards to challenge specific submissions, only the top spot winner will be asked to present their work.
Papers submitted after tier 1 deadline, which is Aug 19th will all be considered for tier 2 submissions as long as they meet the Aug 31st submission deadline.
Only the top place winner will have the opportunity to submit their camera ready version of paper after the challenge deadline.
Might there be any opportunity to extend the Aug 19th deadline by 1 day?
We have been having trouble submitting our latest model over the last few daysâŚ
Iâm currently in the first place and will hopefully keep it till the end, so I wonder about this point: I canât submit a paper via link provided, since it wonât allow to submit past paper registration deadline, which was March 15, 2019.
Anything I can/should do?
Thanks for reaching out to us. If you are indeed the winner, we will officially announce the winners and reach out to them individually requesting them to submit their camera ready version of the manuscript.
Also winners are expected to submit their code base for evaluation.
Just to confirm since it doesnât say so explicitly anywhere in the rules or discussions, but common tactics for prediction improvement like ensembling and test-time augmentation arenât allowed since they defeat the purpose of producing faster predictions, right?
where 0<P<=1, measures model-accuracy as defined by the unweighted mean intersection-over union score over all classes for the test-set, and S>0, measures model-complexity, as defined by the number of model parameters, measured in unit of model-size in MB. More precisely, model-size in MB will be computed as follows: S = number of learned model parameters x 4 (floating point) / (1024*1024)