Explainable AI (XAI): Why AI Needs to Show Its Work

In 2016, Pro Publica published an investigation into COMPAS — a proprietary AI system used by US courts in 45 states to assess recidivism risk and inform sentencing decisions. The investigation, titled ‘Machine Bias,’ found that COMPAS incorrectly flagged Black defendants as future criminals at nearly twice the rate of white defendants.

The company that made COMPAS, Northpointe (now Equivant), disputed the findings. But the deeper problem was structural: COMPAS was a black box. Neither defendants, their attorneys, nor even judges could see how the score was calculated. The algorithm’s inner workings were treated as a trade secret.

This case became a touchstone for the field of Explainable AI (XAI) — the effort to make AI systems transparent, interpretable, and accountable.

Why Explainability Matters

Explainability matters for several interconnected reasons:

• Accountability: When AI makes high-stakes decisions (lending, hiring, healthcare, criminal justice), affected individuals have a legitimate interest in knowing the basis for those decisions
• Debugging: ML models can learn spurious correlations from training data that produce correct predictions in testing but fail in deployment. Explainability tools help engineers identify these issues
• Trust: Doctors, lawyers, and other professionals are more likely to use AI recommendations if they can evaluate the reasoning
• Legal compliance: The EU’s GDPR (Article 22) and the EU AI Act both require explanations for automated decisions; the US CFPB requires adverse action notices for AI credit decisions
• Safety: In high-stakes domains like autonomous vehicles and medical AI, understanding why a system made a decision is critical for safety validation

The Interpretability Spectrum

AI systems fall along a spectrum from inherently interpretable to inherently opaque:

Inherently Interpretable Models

• Linear regression: Coefficients directly show the contribution of each feature
• Decision trees (small): Can be visualized and followed step by step
• Logistic regression: Coefficients interpretable as log-odds contributions
• Rule-based systems: Decisions follow explicit if-then rules

These models sacrifice some predictive power for transparency. In regulated industries, this tradeoff is often mandated — US credit scoring regulations, for example, effectively require that lending decisions be explainable using traditional statistical methods.

Post-Hoc Explainability Methods

Most high-performing AI (deep neural networks, gradient boosting models, LLMs) requires post-hoc explanation methods — tools applied after training to explain what the model is doing:

Key XAI Techniques

LIME (Local Interpretable Model-Agnostic Explanations)

Developed by Ribeiro, Singh, and Guestrin (2016), LIME explains individual predictions by approximating the complex model locally with a simpler interpretable model. For an image classifier, LIME highlights which parts of the image most influenced the prediction. For text, it shows which words drove the classification.

Practical use: If your fraud detection model flags a transaction, LIME can show which transaction features (merchant category, unusual time, foreign location) drove that specific flag.

SHAP (SHapley Additive exPlanations)

Developed by Lundberg and Lee (2017), SHAP is based on Shapley values from cooperative game theory — it calculates each feature’s contribution to a prediction by averaging its marginal contribution across all possible feature orderings. SHAP is theoretically grounded and consistent across models.

SHAP has become the dominant XAI tool in industry. It’s integrated into scikit-learn, XGBoost, LightGBM, and the popular Python SHAP library (2.5M+ monthly downloads). Many financial institutions use SHAP to generate adverse action notices for loan denials.

Attention Visualization

In transformer-based models (BERT, GPT, etc.), attention weights can be visualized to show which parts of the input the model ‘attended to’ when generating an output. However, research by Jain and Wallace (2019) and Wiegreffe and Pinter (2019) showed that attention weights are not always faithful explanations — high attention doesn’t necessarily mean causal importance for the output.

Gradient-Based Methods

• Saliency maps: Show which input pixels or tokens have the highest gradient (most influence on the output)
• Integrated Gradients (Sundararajan et al., 2017): A theoretically grounded method for attributing model outputs to input features
• GradCAM: Gradient-weighted Class Activation Mapping — widely used in medical imaging AI to highlight which regions of an X-ray or scan drove a diagnosis

Counterfactual Explanations

Rather than explaining what happened, counterfactuals explain what would need to change for a different outcome: ‘Your loan was denied. If your annual income were $5,000 higher and your credit utilization 10% lower, it would have been approved.’ This approach is particularly valuable for actionable explanations in regulated decisions.

The Wachter et al. (2017) GDPR paper specifically argued that the GDPR’s ‘right to explanation’ should be operationalized through counterfactual explanations, as they are both understandable and actionable.

XAI in LLMs: The Hard Problem

Explaining why a large language model generates specific text is significantly harder than explaining tabular ML predictions. LLMs have billions of parameters and generate text through complex, interconnected computations that don’t reduce to simple feature attributions.

Current approaches include:

• Chain-of-thought prompting: Asking the model to ‘think step by step’ generates visible reasoning, but this isn’t necessarily faithful to the underlying computation
• Mechanistic interpretability: Research identifying specific circuits and features within transformer models — active area at Anthropic, DeepMind, and academic labs
• Probing classifiers: Training small classifiers on model activations to identify what information is encoded at each layer
• Retrieval attribution: In RAG systems, explicitly tracing which retrieved documents contributed to which parts of the output

A key finding from Anthropic’s 2024 interpretability research: ‘features’ in LLMs (directions in activation space representing specific concepts) behave in ways that are interpretable — the model has distinct internal representations for concepts like ‘banana,’ ‘deception,’ or ‘constitutional amendment.’ But translating this into explanations of specific outputs remains an open research challenge.

Legal Requirements for AI Explainability

• GDPR Article 22: Right not to be subject to solely automated decisions with significant effects; right to obtain explanation
• EU AI Act (High-Risk): High-risk AI systems must be designed to allow human oversight and provide output logs; providers must ensure users can understand and interpret outputs
• US Equal Credit Opportunity Act / CFPB: Lenders must provide specific reasons for adverse credit decisions, applying to AI-based credit scoring
• NYC Local Law 144: Requires bias audits for automated employment decision tools — auditors must understand what factors drive outcomes
• Colorado AI Act: Requires disclosure and opportunity to appeal algorithmic decisions in insurance, lending, and similar domains

XAI in Practice: Industry Applications

• Healthcare: FDA increasingly expects XAI documentation for AI/ML-based medical devices (SaMD); radiology AI must explain why it flags suspicious regions
• Finance: Major US banks (JPMorgan, Bank of America) use SHAP-based explanations for loan underwriting models to comply with fair lending laws
• HR/Recruitment: Companies like HireVue have faced FTC scrutiny over AI hiring tools; XAI methods are used to demonstrate non-discriminatory feature use
• Criminal Justice: Post-COMPAS, jurisdictions have moved toward requiring explainability in risk assessment tools; some have banned proprietary black-box tools entirely

Frequently Asked Questions

What is the difference between interpretable AI and explainable AI?

Interpretable AI uses models that are inherently transparent — their logic can be directly read (like a small decision tree). Explainable AI (XAI) refers to techniques for explaining any model’s behavior after the fact, including black-box models. All interpretable models are also explainable, but not vice versa.

What is SHAP and why is it popular?

SHAP (SHapley Additive exPlanations) calculates each feature’s contribution to a specific prediction using Shapley values from game theory. It’s popular because it’s model-agnostic, mathematically rigorous, produces consistent explanations, and has efficient implementations for popular ML frameworks. It’s widely used in regulated industries for generating audit-ready explanations.

Is explainability required by law in the EU?

Yes, in certain contexts. GDPR Article 22 gives people the right to an explanation for significant automated decisions. The EU AI Act requires high-risk AI systems to provide interpretable outputs that enable human oversight. Specific sector regulations (credit, insurance, employment) also impose explanation requirements.

Can we explain why ChatGPT or Claude said something specific?

Only partially. Attention visualization and gradient-based methods provide partial insight, but no technique fully explains why specific tokens were generated. Mechanistic interpretability research is making progress on understanding LLM internals, but faithful per-token output explanations remain an open research problem.

What is the tradeoff between model accuracy and explainability?

Historically, simpler interpretable models (linear regression, shallow decision trees) are less accurate than complex black-box models on many tasks. However, this tradeoff is decreasing — SHAP and similar methods allow explaining complex models without sacrificing accuracy. Modern regulated industries increasingly achieve both through post-hoc explanation methods rather than deliberately limiting model complexity.

Sources

• Grokipedia
• Stanford HAI AI Index Report

Related reading: AI Glossary | AI Ethics for Beginners | What is an LLM? | AI Safety and Alignment | AI for Doctors

Sources: ProPublica ‘Machine Bias’ (2016), Ribeiro et al. LIME (2016), Lundberg & Lee SHAP (2017), Wachter et al. GDPR counterfactuals (2017), Anthropic interpretability research (2024), EU AI Act (Regulation 2024/1689).

You May Also Like

• What Is Artificial Intelligence
• Best AI Tools for Beginners
• How to Use AI
• AI Tools Directory
• Best Free AI Courses