LIME, SHAP, or InterpretML? Matching Your Explainability Goal to the Right Open-Source Tool

Choosing the right explainable AI (XAI) tool depends on your primary goal: explaining an existing blackbox model or building an inherently interpretable glassbox model. LIME and SHAP offer model-agnostic explanations for blackbox systems, while InterpretML provides a unified toolkit for both glassbox model training and blackbox explanation.

Armen Bedrosian

November 8, 2026 · 5 min read

The fastest way to pick an explainable AI library is to answer one question first: are you trying to explain a model you already have, or build a model that explains itself? LIME and SHAP are model-agnostic explainers that probe an existing blackbox system from the outside. InterpretML, by contrast, is a toolkit that lets you train inherently interpretable "glassbox" models and also explain blackbox systems, including via LIME and SHAP techniques. Your answer to that question, plus your need for local versus global explanations, largely determines the right choice.

Understanding XAI: Blackbox vs. Glassbox Strategies

Explainable AI splits into two fundamental strategies. The first treats the model as a blackbox: you keep your existing classifier or regressor untouched and run a separate explanation method alongside it. The second builds interpretability into the model itself, training a glassbox model whose behavior a person can inspect directly.

The InterpretML project, hosted on GitHub, frames this dual capability explicitly: with the package, "you can train interpretable glassbox models and explain blackbox systems," and it supports understanding either a model's global behavior or "the reasons behind individual predictions." LIME and SHAP, the two most widely used model-agnostic tools according to the XAI frameworks directory maintained by Peter India, sit firmly in the first camp. They answer the question "which features drove this prediction?" without ever looking inside the model's internals.

LIME: Local Explanations for Any Model

LIME (Local Interpretable Model-agnostic Explanations) works, as its name states, by "explaining predictions of any machine learning classifier by learning a locally faithful interpretable model around each prediction," in the words of the same directory. In practice, it perturbs the input around a single instance, watches how the model's predictions change, and fits a sparse linear model to that local behavior.

The payoff is speed and scope. Because LIME only approximates the model in the vicinity of one prediction rather than evaluating every feature combination, it runs quickly even against models it was never designed for. Writing for Domino in 2018, Josh Poduska made the trade-off plain: "Simply put, LIME is fast, while Shapley values take a long time to compute."

The limitations mirror the strengths. LIME is local by construction—it tells you why one prediction came out the way it did, not what the model believes in general. Its linear approximations can also miss behavior the full model exhibits elsewhere. And there is a maintenance caveat: the Peter India directory, which checked GitHub default-branch activity in October 2026, records LIME's last commit as July 2021 and advises checking a project's activity "before relying on it."

SHAP: Game Theory for Feature Contributions

SHAP (SHapley Additive exPlanations) is, per its own official documentation, "a game theoretic approach to explain the output of any machine learning model" that "connects optimal credit allocation with local explanations using the classic Shapley values from game theory and their related extensions." Rather than perturbing inputs ad hoc, SHAP computes each feature's average marginal contribution over all possible feature coalitions.

That exhaustive accounting buys theoretical guarantees. As Poduska explains, "Shapley values consider all possible predictions for an instance using all possible combinations of inputs," and because of this "SHAP can guarantee properties like consistency and local accuracy." He also notes that the original SHAP paper's authors showed Shapley values are the only approach with these guarantees and that LIME "is actually a subset of SHAP but lacks the same properties."

Although SHAP's explanations are local—per-prediction feature attributions—aggregating many local explanations yields global insight into overall feature importance, which is where its flexibility shows. The computational cost is the constraint: Poduska found that SHAP's tree explainer "runs fast on trees, such as gradient boosted trees from XGBoost and scikit-learn and random forests," but "for a model like k-nearest neighbor, even on a very small dataset, it is prohibitively slow."

XAI Tool Selection Decision Framework

The table below maps the three tools across the dimensions that actually drive the choice: your primary goal, the explanation scope you need, the underlying methodology, relative speed, and maintenance status. Read it by matching your project's first row to a column, then verifying the constraints in the lower rows before committing.

Decision framework for choosing among LIME, SHAP, and InterpretML
Dimension	LIME	SHAP	InterpretML
Primary goal	Explain predictions of any pre-existing blackbox model	Explain the output of any pre-existing blackbox model with guaranteed credit allocation	Train inherently interpretable glassbox models and explain blackbox systems
Explanation scope	Local only, around a specific instance	Local explanations, aggregatable for global insight	Both global behavior and individual predictions
Core methodology	Perturbation of inputs, fitted with a locally faithful interpretable model	Game-theoretic Shapley values across all feature coalitions	Toolkit: glassbox models such as Explainable Boosting Machines plus blackbox explainers including LIME and SHAP
Relative speed	Fast, because it approximates locally rather than exhaustively	Fast for tree-based models; reportedly prohibitively slow for k-nearest neighbors even on small datasets	Varies by chosen component; glassbox models are interpretable by design
Maintenance status	Last default-branch commit July 2021, per GitHub activity checked in October 2026; verify before relying on it	Actively used, model-agnostic, with optimized explainers for common model families	Actively maintained toolkit offering both model types and explainers

Decision framework for choosing among LIME, SHAP, and InterpretML

Dimension

LIME

SHAP

InterpretML

Primary goal

Explain predictions of any pre-existing blackbox model

Explain the output of any pre-existing blackbox model with guaranteed credit allocation

Train inherently interpretable glassbox models and explain blackbox systems

Explanation scope

Local only, around a specific instance

Local explanations, aggregatable for global insight

Both global behavior and individual predictions

Core methodology

Perturbation of inputs, fitted with a locally faithful interpretable model

Game-theoretic Shapley values across all feature coalitions

Toolkit: glassbox models such as Explainable Boosting Machines plus blackbox explainers including LIME and SHAP

Relative speed

Fast, because it approximates locally rather than exhaustively

Fast for tree-based models; reportedly prohibitively slow for k-nearest neighbors even on small datasets

Varies by chosen component; glassbox models are interpretable by design

Maintenance status

Last default-branch commit July 2021, per GitHub activity checked in October 2026; verify before relying on it

Actively used, model-agnostic, with optimized explainers for common model families

Actively maintained toolkit offering both model types and explainers

Three decision paths fall out of this mapping. If you need a quick answer about one prediction from an existing model, LIME is the lowest-friction route—provided its maintenance status doesn't matter for your deployment. If you need robust, theoretically grounded attributions and your model is tree-based, SHAP gives both local precision and aggregated global views at acceptable cost. If you are starting a new project where interpretability is a requirement rather than an afterthought—or you want one toolkit covering both worlds—InterpretML is the only one of the three that offers glassbox training.

LIME, SHAP, or InterpretML? Matching Your Explainability Goal to the Right Open-Source Tool

Understanding XAI: Blackbox vs. Glassbox Strategies

LIME: Local Explanations for Any Model

Tags

SHAP: Game Theory for Feature Contributions

InterpretML: A Unified Toolkit for Glassbox and Blackbox

Practical Applications of XAI Tools in the Development Lifecycle

XAI Tool Selection Decision Framework

Making Your XAI Tool Choice

Sources

Related articles

Monitoring vs. Observability in MLOps: A Three-Pillar Framework for Keeping Production Models Healthy

Choosing an Enterprise AI Code Assistant: How Copilot, Tabnine, Codeium, and Amazon Q Developer Compare on Security and Deployment

Asana, ClickUp, Jira, or monday.com: Which AI Project Management Tool Fits Your Agile Team?

Milvus, Weaviate, or Qdrant? Matching Your RAG Project to the Right Open-Source Vector Database