What's Wrong with Post-hoc XAI Methods for Loan Approval Systems?
Part one of a three-part series on AI, credit decision-making, and the regulatory shift from associational explanations to causal transparency under the EU AI Act.
The Impact of AI and Credit on Society
Access to credit is more than just a financial transaction as it is a gateway to societal participation. For an individual, it represents a home, an education, or a lifeline for a small business. For society, it is the engine of economic mobility. When we hand the decision on loan approval or rejection to Artificial Intelligence (AI), we aren’t just automating data—we are automating life-altering decisions.
For the last decade, the mandate in AI development and application has been simple: performance at all costs. We traded the transparency of simpler models for the predictive performance of gradient-boosted trees and deep neural networks. When asked how these models made a decision, we rely on “post-hoc” explainability methods like SHAP and LIME. These are explanation methods that are applied after a machine-learning model has already been trained.
Our research suggests that the financial industry is currently standing in front of an “explainability gap” when applying more complex AI systems. With the arrival of the European Union (EU) AI Act, the tools we currently use to explain AI may also soon be legally, and logically, obsolete.
Transparency means that AI systems are developed and used in a way that allows appropriate traceability and explainability, while making humans aware that they communicate or interact with an AI system, as well as duly informing deployers of the capabilities and limitations of that AI system and affected persons about their rights.
A “Path to Approval”
Imagine a young entrepreneur applying for a loan to scale her business and gets rejected. Under the new EU AI Act, this is considered a high-risk AI application. The legal requirements for such systems are shifting from passive disclosure to active comprehension. Article 13 of the Act stipulates that high-risk AI must be transparent enough for deployers to “understand how the system works” and “comprehend its strengths and limitations”1. Crucially, the Act emphasizes that information should cover “deployer action that may influence system behavior,” effectively suggesting that a customer has a right to know not just why they were rejected, but what they can change to be approved.
In today’s fintech landscape, we attempt to satisfy this with tools like SHAP2. A bank might tell our entrepreneur: “Your rejection was 30% due to your debt-to-income (DTI) ratio and 20% due to your recent credit inquiries.” This looks like an explanation, but it is functionally hollow. If she pays off a loan to lower her DTI ratio, will the model actually flip to “Approved”?
Current industry standards fail this mandate because they target feature attribution—which correlations most influenced a the predicted value. They are associational by nature. Associational explanations answer whether two variables are related or correlated, without claiming that one causes the other. Consequently, they remain silent when faced with the two critical questions for both regulators and customers:
How do inputs actually influence the approval decision?
What specific changes would lead to a different approval outcome?
These are not associational questions; these are causal and interventional questions.
Standard post-hoc feature attribution methods such as SHAP or LIME3 operate at the associational level. They decompose a prediction into contributions based on statistical dependencies learned during training. An associational feature attribution method cannot answer this. To move from rejection to approval, we must move beyond associational predictors and embrace models capable of causal and interventional reasoning.
Pearl’s Ladder: The Ceiling of Current XAI Methods
To understand why we are stuck, we can take a look at the work of researcher Judea Pearl, and his Ladder of Causality4. His framework will serve as the starting point of our exploration into this topic.
Rung 1: Association (The “Seeing” Level)
Almost every XAI tool used in financial applications today lives on the bottom rung5. They address questions of association. They see that A and B often happen together. If a model sees that people with “Premium Bank Accounts” rarely default, it assigns a high importance to that feature. But having a premium account doesn’t prevent default; it’s likely a proxy for wealth. Standard XAI methods like SHAP and LIME operate entirely at this level.
Rung 2: Intervention (The “Doing” Level)
This is where the EU AI Act is pushing us. Intervention asks: “If I take this action, what is the effect on the outcome?” To answer this, a model cannot just be a associational predictor; it must model cause and effect in the world. If the entrepreneur takes a specific action today—say, increasing her cash reserves by 10%—an interventional model can predict the direct effect on her score. It answers: ‘What happens if I change the world?’
Rung 3: Counterfactuals (The “Imagining” Level)
The highest level of causal reasoning. “What would have happened if I had 10% more collateral?” This requires the model to simulate an alternative reality. Counterfactuals look into an alternative world, to ‘what could have been’. If the entrepreneur had applied for the same loan but had five more years of credit history, would the result have changed? Counterfactual reasoning allows us to test if a specific variable was the ‘but-for’ cause of a rejection.
Our investigation starts with the realization that post-hoc XAI methods address the wrong level of questions. It attempts to explain Rung 1 associations to satisfy Rung 2 and 3 requirements.
An Explanatory Shift: From Post-hoc to By-Design
In this three part series, we are exploring a different path. Instead of training a black-box and trying to “explain” it later, what if we built the model to be explanatory by design?
In Part 2, we stop trying to fix the symptoms of black-box models and look at an alternative approach. We’ll show how we built a model that doesn’t just predict credit risk, but understands the ‘why’ by design.
About the Project
Our work within Work Package 3 of the MSCA Digital Finance Doctoral Network is an exploration into the next generation of financial AI. We are investigating how to move beyond black-box solutions toward models that are inherently more transparent and trust-oriented. By developing and testing methodologies, such as non-perturbation-based XAI, we aim to describe how well current and novel tools meet the diverse needs of the financial value chain.
As we continue to validate these approaches against industry baselines and explore their impact on algorithmic fairness, we invite a dialogue with the following stakeholders:
Industry Practitioners: Are you interested in the practical trade-offs between predictive performance and explainability, or looking to test how audience-dependent explanations function in real-world use cases?
Regulators: Would you like to discuss technical benchmarks for “meaningful transparency” and how to evaluate AI systems against modern European requirements for trust and unbiasedness?
Researchers: Are you working on novel XAI methods, time-series dependencies, or causal inference, and looking to collaborate on advancing the state of the art in high-risk financial applications?
We believe that building human-centric AI requires an interdisciplinary effort. We invite you to reach out to discuss our findings, share your perspectives, or explore potential collaboration in this evolving field.
Contact Information
Connect with the author:
Follow my research updates: Applicable Reinforcement Learning
Direct Inquiry: mathis.jander@utwente.nl | LinkedIn
Connect with our project:
Follow our research updates: MSCA Digital Finance Doctoral Network | LinkedIn
Direct Inquiry: branka.hadjimisheva@bfh.ch
Acknowledgments
Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence and amending Regulations (EC) No 300/2008, (EU) No 167/2013, (EU) No 168/2013, (EU) 2018/858, (EU) 2018/1139 and (EU) 2019/2144 and Directives 2014/90/EU, (EU) 2016/797 and (EU) 2020/1828 (Artificial Intelligence Act) [2024] OJ L2024/1689
Scott M. Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions. In Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS’17). Curran Associates Inc., Red Hook, NY, USA, 4768–4777.
Ribeiro, Marco & Singh, Sameer & Guestrin, Carlos. (2016). “Why Should I Trust You?”: Explaining the Predictions of Any Classifier. 1135-1144. 10.1145/2939672.2939778.
Pearl J. An introduction to causal inference. Int J Biostat. 2010 Feb 26;6(2):Article 7. doi: 10.2202/1557-4679.1203. PMID: 20305706; PMCID: PMC2836213.
Černevičienė, J., Kabašinskas, A. Explainable artificial intelligence (XAI) in finance: a systematic literature review. Artif Intell Rev 57, 216 (2024). https://doi.org/10.1007/s10462-024-10854-8
Nicely and accessibly explained! Well done, Mathis.