Precision Recall Curves

Worked Example

Consider a dataset named "Four" with model scores [0.1, 0.4, 0.35, 0.8] and true labels [0, 0, 1, 1]. Let's calculate the metrics at a threshold of 0.5.

1. First, we apply the threshold to the model scores. Scores ≥ 0.5 are predicted as positive (1), else negative (0). Our predictions are [0, 0, 0, 1].
2. We compare our predictions with the true labels:
   • Score 0.1, True 0: True Negative
   • Score 0.4, True 0: True Negative
   • Score 0.35, True 1: False Negative
   • Score 0.8, True 1: True Positive
3. Calculate the number of True Positives (TP), False Positives (FP), and False Negatives (FN). Here, TP = 1, FP = 0, FN = 1.
4. Calculate Precision: TP / (TP + FP) = 1 / (1 + 0) = 1.0.
5. Calculate Recall: TP / (TP + FN) = 1 / (1 + 1) = 0.5.

The Average Precision (AP) is computed by summarizing the PR curve as the weighted mean of precisions achieved at each threshold, with the increase in recall from the previous threshold used as the weight. For this dataset, the stepwise Average Precision is exactly 0.8333333333333333.

Interactive Experiment

You can interactively explore the precision and recall curve for different test datasets using the Precision Recall Curves module. Follow these steps:

1. Set the Dataset control to "Four".
2. Adjust the Threshold control to 0.35.
3. Observe that at this threshold, the model correctly identifies both positive examples (TP = 2), reaching a Recall of 1.0. However, it also incorrectly classifies one negative example as positive (FP = 1), which lowers the Precision to approximately 0.6667.
4. Change the Dataset control to "Ties" and note how ties in predicted scores are handled simultaneously, creating a single joint step in the PR curve.

Limitations

The PR curve and average precision have important limitations. Precision becomes undefined when the model makes zero positive predictions across the entire dataset. In our visualization, we denote this algebraically undefined state as "Undefined". Similarly, recall and average precision become undefined if a dataset has zero actual positive examples (e.g., the "No positives" dataset). Furthermore, different libraries might implement average precision interpolation differently; this module specifically uses the stepwise sum over recall increments without substituting a trapezoidal PR area, ensuring accurate attribution of precision to the recall change.

References

The following references define the metrics and explain how they are calculated step-by-step for evaluating binary classifiers.

• scikit-learn: precision_recall_curve - Defines the method for computing precision-recall pairs for different probability thresholds.
• scikit-learn: average_precision_score - Explains the stepwise calculation of average precision from prediction scores.