Target Encoding and Leakage

Target encoding (or mean encoding) is a powerful technique to handle categorical features with high cardinality. Instead of creating many one-hot columns, each category is replaced with the mean of the target variable for that category. However, calculating this mean on the same dataset used for evaluation can lead to severe data leakage.

Worked Example

Consider a dataset where each row has a categorical feature and a binary target. Suppose a category "B" appears in rows 2, 4, and 5 with targets 1, 0, and 1 respectively. For an in-sample mean encoding, the encoded value for "B" is (1 + 0 + 1) / 3 = 0.6667.

But what if a category appears only once (a singleton)? If category "A" is only in row 1 with target 1, its in-sample encoded value is 1/1 = 1.0. The model will perfectly memorize this target, leading to severe overfitting. To prevent this, we can use a Leave-one-out (LOO) protocol: when encoding row i, we compute the statistics using all rows except row i.

Let's trace this for a singleton row where category is "A" and target is 0, across 6 total rows where the remaining 5 rows have 3 positive targets.

Row i (Category A, Target 0) LOO Training Indices: The other 5 rows. Target Sum for Category A in Training: 0 Count for Category A in Training: 0 Global Prior in Training: 3/5 = 0.6 Encoded Value (with 0 smoothing) = Prior = 0.6 Prediction Error = (0.6 - 0)^2 = 0.36

Experiment

Explore the impact of the protocol in the interactive Target Encoding and Leakage module.

Set the Dataset to "Singletons" and Row index to 0.
Choose "In sample" as the Protocol. Note the Encoded value is 0.0, which perfectly matches the Observed target, resulting in 0 error. This is leakage.
Change the Protocol to "Leave one out". The Encoded value changes to 0.6, utilizing the Training prior, as the group count becomes 0. The error is now 0.36.
Adjust the Smoothing parameter to see how alpha values pull the encoded value closer to the global prior when group counts are small.

Limitations

The Leave-one-out method demonstrated here is a pedagogical simplification. Excluding only the evaluated row from the category statistics, while retaining its presence implicitly in other ways, can still leak information. In practice, production models use K-fold cross-fitting where statistics are computed on out-of-fold data. Also, scikit-learn's `TargetEncoder` uses continuous cross-validation, not just LOO, to further prevent overfitting.

References

For further reading on practical target encoding and preventing leakage, consult the following resources:

Target Encoding and Leakage