January 20, 2023 | by Tobias Schaefer, Bogdan Loukanov, Sam Fisher, and Dmitry Lesnik
Many algorithms are capable of underpinning decision systems. They vary in efficacy regarding properties such as accuracy, speed, and interpretability. In order to fulfill business requirements and objectives, companies are often torn about which algorithms to use. One of the most common yet thorniest issues is the tradeoff between accuracy and interpretability, especially when business goals require the algorithm to be both, but available methods outperform in one area while underperforming in the other.
Logistic regression models, for one, are highly interpretable, but not necessarily accurate. They are heavily used, for example, for credit scoring, where interpretability is a key feature to satisfy regulatory requirements and to ensure fairness. The lack of accuracy, however, is a problem — inaccurate decisions can be costly, both from a financial and social impact perspective. On the other hand, a different class of algorithms like neural networks or extreme gradient boosting (XGBoost) are often highly accurate, but much less interpretable, and are therefore described as ‘black-box’ in nature. While they excel in applications like image recognition, their lack of interpretability makes it difficult to apply them in other fields such as loan underwriting, where transparency is required.
Evaluating Accuracy and Interpretability in ML Approaches
So how do we approach this tradeoff? Over the last two decades, there has been tremendous and exciting development in Markov random fields and Markov logic networks, which are the foundation of probabilistic rules models¹. These are capable of offering both interpretability and accuracy at the same time. Let us first have a look at the performance of such models and then delve into the basic ideas of such models to understand why they work so well.
In the following study, we compare the performance of probabilistic rules to the performance of two traditional machine learning algorithms, logistic regression (LogReg) and XGBoost, which have opposing strengths in terms of accuracy and interpretability. The probabilistic rule models were developed using the PRE (“probabilistic rules engine”), a commercial tool developed by Stratyfy. Logistic regression and XGBoost were implemented using Python and the scikit-learn library.
For our initial set of benchmarks, we chose two publicly available Kaggle datasets (“Fraud ecommerce” and a credit card fraud dataset generated using Sparkov) and one synthetic dataset generated with the scikit-learn library. The graph below shows the performance of the different models (Logistic regression, XGBoost, PRE) in terms of the AUC².
While all models perform at a similar level for the Ecommerce dataset, logistic regression lags behind XGBoost and probabilistic rules for the Sparkov dataset and the synthetic Sklearn dataset. This is to be expected, as both XGBoost and the probabilistic rules are able to capture more complex (e.g. nonlinear) dependencies. The accuracy of XGBoost, however, comes at a price: XGBoost models lack transparency and are much less interpretable than models created using Logistic Regression or the probabilistic rules. The sacrifice of interpretability in favor of accuracy is a typical feature of such boosting methods.
Comparing the Performance of ML Approaches on Mortgage Data
In addition to these data sets, we also compared the performance of the three methods for Lending Club data and Fannie Mae mortgage loan data. The first is an example for models describing consumer loan credit risk and the second serves as an example for mortgage loan risk. Here, in terms of predictive power, all three methods performed at the same level. In this case, the client can choose the modeling technique that fits best into their business from other perspectives (e.g. transparency, fairness, bias control, etc). The table below summarizes the performances of the different models for all data sets we studied.
If we present our results in terms of accuracy and interpretability, we see that only the probabilistic rules can deliver both across all data sets studied.
How Probabilistic Rules Models Work
Now let us have a closer look at probabilistic rules models. How do they work? How are they built and calibrated? Let us use an example from credit risk. Given a set of input features x₁, x₂, … xₙ, the decision system’s task is to predict the likelihood of default. Let us imagine that x₁ is annual income and x₂ is a debt-to-income (“dti”) ratio. A probabilistic rule is then, for example, of the form
IF (income < $25,000 AND dti > 40%) THEN (default with likelihood 0.15)
This rule indicates that a customer whose data satisfies the conditions in the if-statement (someone with an income below $25,000 and dti above 40%), will default with a likelihood of about 15%. Unlike traditional knock-out policy rules used in underwriting however, this rule doesn’t automatically decline customers with income below $25,000 and dti above 40%. Instead, it only reduces the customer’s score, but there is still a chance that the loan will be granted if other circumstances help to assert the customer’s creditworthiness.
A typical model consists of a set of such probabilistic rules which interact in a complex way, governed by the mathematical rules of Markov random fields to produce a comprehensive score. Rules and likelihoods are found from data, but can also be controlled by subject matter experts to incorporate domain knowledge that is not reflected in the data set at hand. For example, in fraud detection, there are often fraud patterns that are known to experts, but happen to occur too rarely in the training data to be picked up by a statistical machine learning algorithm. When using probabilistic rules, it is still possible to incorporate this knowledge in terms of probabilistic rules for developing a more efficient fraud model. In addition, models based on probabilistic rules often show advantages when working with small data sets.
Once the model has been created, it can be used to predict the outcomes of new, previously unseen data, just as with other decision systems. The model’s predictive performance can then be analyzed using any commonly available evaluation metric, including confusion matrices, ROC curves, AUC, etc. In addition, the quality of the system can be assessed using metrics that might be business specific, like if the cost of errors of the system (false positives vs. false negatives) differ.
Why Probabilistic Rules Outperform
If you are interested in decision support systems that are accurate and interpretable at the same time, probabilistic rules are the solution. Many existing “transparent” systems utilize a combination of a ‘black-box’ model and a so-called explainer, but probabilistic rules provide global explanations and inherent interpretability. Probabilistic rules are setting the new standard for decision systems of today and tomorrow.
Want to learn more about Stratyfy’s technology? Connect with us.
Appendix — Description of Datasets
Notes
[1] In this context, a Markov random field is a graphical model of random variables representing features and output variables of the decisioning system. More details about the theory and implementation of Markov graphical networks can be found, for example, in the article by Richardson and Domingos (Richardson, M., Domingos, P., 2006: Markov logic networks. Mach. Learn. 62, 107–136) or the book by Koller and Friedman (Koller, D. and Friedman, N., 2009: Probabilistic graphical models: principles and techniques. Cambridge, MA, MIT Press).
[2] Recall that the ROC (receiver operating characteristic) curve shows the relationship between the true positive rate and the false positive rate for different decision thresholds for a binary classifier. The AUC (area under the ROC curve) is a commonly used measure for the quality of the predictions of the decisioning system. A random classifier with no predictive power has an AUC of 0.5 whereas a perfect classifier has an AUC of 1.