Possible evaluation script error

There appears to be an error in the evaluation script in the devkit. When two nodes are close to each other and predictions are present for both, the first truth node will correctly be associated with the first predict, but the second truth node will also be associated with the first predict and will be scored as a false positive.

A suggested fix is to add the following at line 32:

            (p_object['matched'] == False) &

Regards,
Jeff

Hi Jeff,

Good catch! We just updated it :slight_smile:

Thank you so much!
Víctor

Hi Victor,

I don’t understand why this code is at line 52 - p_object.loc[matching_participant_events.index[0], 'matched'] = True, its not nested under the FP if-else construct and therefore adds all counts of TPs and FPs to the additional_fp variable instead of just FPs. Is this correct?

Regards,
Avhijit

Avhijit,

I hope you don’t mind if I jump in here.

The line you mention is line 54 in the current devkit version, so you may want to update to follow the below remarks.

Line 20 adds a column called ‘matched’ to the participant dataframe and sets all values to ‘False’ to track which participant nodes have already been matched. The loop starting at line 30 iterates on each ground truth node and line 31 identifies ALL participant nodes within tolerance of the current ground truth node, ignoring any that have been previously matched. Line 38 tests for the existence of any matching participant nodes, in which case the first one at index 0 is assumed to be the match. After going through and comparing the ground truth and participant nodes, line 54 that you highlighted above sets the ‘matched’ value on that participant node to ‘True’ so that it isn’t used to match to any subsequent ground truth nodes.

I hope that clarifies it for you.

Regards,
Jeff

1 Like

Thank you @beckja ! We will send an email soon to notify about some of the latest changes and remind everybody to pull the latest version of the devkit.

Best!
Víctor

Thanks a lot for the explanation, Jeff!
That clarified my doubt.

Regards,
Avhijit