In this article, I compare model explainability techniques for feature interactions. In a surprising twist, two commonly used tools, SHAP and ALE, produce opposite results.
I probably shouldn’t have been surprised. After all, explainability tools measure specific responses in different ways. Interpretation requires understanding testing methodologies, data characteristics, and problem context. Just because something is called explainer does not mean that it generates a explanation, if you define an explanation as a human understanding of how a model works.
This post focuses on explainability techniques for feature interactions. I use a common project dataset derived from real loans (1) and a typical mode type (a boosted tree model). Even in this everyday situation, explanations require thoughtful interpretation.
If the details of the methodology are overlooked, explainability tools can impede understanding or even undermine efforts to ensure model fairness.
Below, I show disparate SHAP and ALE curves and demonstrate that the disagreement between the techniques arises from differences in the measured responses and feature perturbations performed by the tests. But first, I will introduce some concepts.
Trait interactions occur when two variables act in concert, resulting in an effect that is different from the sum of their individual contributions. For example, the impact of a bad night’s sleep on a test score would be greater the next day than a week later. In this case, a feature representing time would interact with or modify a sleep quality feature.
In a linear model, an interaction is expressed as the product of two characteristics. Nonlinear machine learning models typically contain numerous interactions. In fact, Interactions are fundamental to the logic of advanced machine learning models. However, many common explainability techniques focus on isolated feature contributions. Methods for examining interactions include 2-way ALE plots, Friedman’s H, partial dependence plots, and SHAP(2) interaction values. This blog explores…
