Understanding AI models with integrated gradients

However, the progress and the strong media presence of ML methods is putting one of the
cornerstones of banking - the trust of customers in their bank and, indirectly, in its business
processes - to the test. However, the increasing complexity makes it difficult to understand this topic
in detail.

Traceability of AI

National and international supervisory authorities and laws, such as CRR/CRD (Capital Requirements Regulation / Capital Requirements Directive) and GDPR (General Data Protection Regulation) already regulate the framework conditions for classic data processing. However, these framework conditions require revision or expansion due to the changes in the way modern methods work. BaFin (Federal Financial Supervisory Authority Germany), Deutsche Bundesbank (German Federal Bank) and also EBA (European Banking Authority) (with its discussion and position papers), as important supervisory bodies, are expressing their views on the use and development of such systems for data processing. One decisive factor is the explainability or traceability of artificial intelligence.

To illustrate this, we present below (in a highly simplified way) a bank's decision to grant a loan based on client scores, driven by a machine learning model. The issue can be summarized quite easily: John Doe wants to take out a loan, the clerk enters the client data into the model and gets a high score as a result. The clerk then gives the green light for the loan to be granted. But how exactly does the model calculate this score? Does it adhere to existing rules or laws? And does this decision stand up to close scrutiny by the auditors? To answer these questions, the bank needs to be able to trace which features the model used to reach the decision for the calculated score.

Models such as logistic regression or decision trees are the easiest to explain because they have inherent explainability due to their straightforward functioning. In the case of logistic regression, each client feature has a weight and the sum of these weights yields the result. In the above example, John Doe owns collateral worth many times the amount of the loan. The model weighs this value (after optimization in accordance with past granted loans) with 90 percent of the total decision, yielding a high score. These weights can be read and interpreted directly, providing a high degree of transparency and traceability.

However, a key drawback with such simple methods is their limited accuracy. While many areas of the banking industry use these methods with significant success in daily business, more and more institutions are turning to  complex machine learning approaches to gain a competitive advantage or to catch up with other institutions.

Modern types of model, such as neural networks, do not have this inherent explainability. At first glance, they appear like big black boxes that do not permit insights into their inner workings. Nevertheless, techniques exist that attempt to specify the role of client features. A distinction is made between local and global methods: Local methods slightly change the input to the model and measure the impact on the result (for example, ten percent less collateral reduces the score by five base points). Whereas global methods attempt to determine comprehensive decision-making criteria of the model. As a rule, it is not possible to make precise statements about an individual client and detailed statements about the interaction of client features for the entire client base at the same time. Compromises have to be found in order to strike a happy medium in terms of application.

A promising approach – Integrated Gradients

A promising approach, which combines local and global explainability as best as possible, is integrated gradients (IG). This method, which is specifically tailored to banking applications, uses both information about the global decision-making behavior of the model and influences from the individual client's data. This way, the best possible balance of global and local decision explanation can be found.

The IG method is based on representative baselines. They describe samples, which represent the various statistical features of existing data and subsets of these in as balanced a manner as possible. Taking the example of the bank that wants to grant a loan to John Doe, a balanced baseline would contain a set of client data with a wide variety of financial and personal features. This means that both lower-income and high-income clients, as well as clients with different payment histories and family statuses, must be used to examine the model decision and the associated comparisons with the overall client base. This ensures that all possible influences on the model decision are mapped by the choice of baseline. Accordingly, such data analyses must comply with the rules of data protection. This article will mainly focus on technical aspects, but it should be said that such privacy aspects play a major role in process and model development as well as validation with IG.

In the next step, the individual clients of the sample are evenly transformed to the intended client to be analyzed, John Doe. During this transformation, it is measured how the model changes it statement about the scoring and, similar to LIME (Local Interpretable Model-​Agnostic-Explanations), this change is assigned to the individual features. This is done by determining the arithmetical gradient of the decision function. Comparable to a bicycle tour through the mountains, an elevation profile is created for each feature. With this profile, the overall influence can be determined.

All gradients of the complete transformation are summed up and displayed in an overview. This overview contains the statistical information about the baseline and the client features of the intended client, John Doe. This makes it possible to quickly classify how much John Doe differs from the average client. In addition, the contributions to the client score value are shown for all features; these (contributions) can have either a positive or a negative effect on the result.

Using these influencing factors, conclusions can be drawn quickly as to whether the decision made is logical and as traceable as possible. Integrated gradients can thus generate a comprehensive decision profile of a machine learning model by appropriate application to individual clients and client groups. For a detailed analysis and model test, several intended clients must also be validated using the same baseline. Moreover, different baselines must be created with the same base set and the analyses must be repeated.

Advanced analyses still include baselines that consist only of sub-segments of the client base, for example, only of business clients or clients with very low deposits. Such sub-segment analyses provide insight into the dynamics that exist in the decision making of the model when statistically very different domains are handled. This is sometimes very important for stability analyses or studies to prevent discrimination that are required by supervisory authorities and auditors.

Conclusion

Understanding and traceability of complex models is a cornerstone for future data processing in the banking industry. To ensure the robustness of systems and compliance with all current and future regulations, a deep understanding of the delivered results using local and global explanatory methods - for example with integrated gradients - is irreplaceable. Comprehensive analyses and detailed reports on the model behavior provide the basis for trust in these modern systems for data processing – both from the client's perspective as well as from the perspective of the supervisory authorities and auditor.