Tech x Media x Impact | Analytics Manager & Data Scientist |Past: Legendary Entertainment, VEVO, AOL
Introducing PeerVest: A free ML app to help you pick the best loan pool on a risk-reward basis
Peer-to-peer lending marketplaces like LendingClub and Prosper Marketplace are driven by what is essentially a brokers fee for connecting investors and borrowers. They are incentivized to increase the total number of transactions taking place on their platforms.
Driven by ease-of-use, their off-the-shelf credit risk assessments are scored in grouped buckets. On a loan-by-loan basis, this is inefficient given each loan’s uniqueness and the sheer amount of data collected from borrowers. Scoring risk on a more granular, continuous basis is not only possible but preferable over discrete, grouped buckets.
Credit risk is something all peer-to-peer lending investors and bond investors must carefully consider when making informed decisions. Institutions, including large banks, have been employing researchers and quantitative analysts to wrangle and analyze this data hoping to become more confident in their risk-reward assessments.
What is the underlying risk that a borrower fails to make required payments, leading to a loss of principal and interest? Given the terms of the loan, what return can an investor expect? There should be a more simple and accessible tool for the everyday investor that can hope to outperform the marketplace’s own off-the-shelf advisor.
PeerVest helps individual investors augment their portfolio by intelligently allocating funds to Peer-to-Peer Lending Marketplaces using machine learning trained on LendingClub.com historical loan data to assess risk and predict return.
PeerVest recommends the best loans to invest in given a user’s available funds, maximum risk tolerance, and minimum desired annualized return. There are plenty of institutions that are thought leaders in utilizing modern, alternative datasets to assess credit risk and predict investment potential, but I’ve simplified the problem into 2 key models: (1) scoring risk by predicting the probability that a loan defaults, and (2) predicting annualized returns.
Source: Business Insider
To assign a quantity to each loan that measures its risk level, or the probability that the borrower will default on the loan, I approached this like a standard classification problem where each historical loan is assigned a 1 if it was Fully Paid and a 0 if it defaulted. To provide a more granular risk assessment than LendingClub and sites like it provide off the shelf, I was looking not for the class (1 or 0), but for the probability that each loan is predicted to fall in each class. My initial approach was to use a Logistic Regression, which despite classifying each loan quite confidently, and despite probability calibration, didn’t seem to interact well with sklearn’s .predict_proba() function. This function returns a probability between 0 and 1 that a loan belongs to class 1. Taking 1 minus this very number yields an interpretable probability of default metric (the larger it is, the higher likelihood that it will default).
Given 1,107 unique features, there seemed to exist many non-linear relationships and the effort of manually reducing dimensions would be cumbersome. There was no true benefit to using Principal Component Analysis since it was not improving the situation — it reduced dimensionality and reduced explained variance in equal magnitude. I decided to keep all the features, despite the increased likelihood of feature redundancy. Knowing neural networks would be able to handle this complexity over many epochs of training, I began architecting a model using Keras, a neural network library built on TensorFlow. I iterated through many different Keras architectures by employing different combinations of hidden layer sizes, dropout regularization, L2 kernel regularization, early stopping, epochs, class weight balancing, and ReLU/Sigmoid activation functions. Despite some stagnation in validation loss, the model eventually broke through and began improving more significantly within a few epochs, for both train and test sets, while calibrating prediction probabilities very well.
Underperformers: 3 Scikit-Learn Logistic Regression models, 7 Keras Neural Networks
Best Performer: Keras Neural Network v8
Best Neural Network Model Architecture:
Input Layer (1107, Sigmoid) ->
Dense (1000, Sigmoid, L2) ->
Dense (300, Sigmoid) ->
Dense (50, Sigmoid, L2) ->
Dense Output Layer (1, Sigmoid, L2)
Best Model Parameters:
Best Model Evaluation Metrics (Test Error):
I used a very simple Annualized Return calculation based on the available data from LendingClub. I decided against using LendingClub’s complicated Adjusted Net Annualized Return computation due to interpretability, though as I continue iterating my model I am open to refined calculations that utilize more traditional financial loss estimates. The element of time as it relates to loan terms and payments sits within this calculation. Macro information like the federal interest rate, inflation and the time value of money is inherent in the interest rate quoted to the borrower, though there is certainly room to dive into time series and survival analysis with increased rigor to productionalize this model for financial institutions.
Linear Regression worked decently on the training set, but was overfit and did not generalize well to the future set or the test set within the same period. Ridge Regression, which is similar but applied L2 regularization fixed the overfitting issue in my 2nd iteration. That said, I wanted to know if my evaluation metrics would improve using something that could make regression decisions differently, and if there was a model that wouldn’t be as reliant on total payment so that my prediction could be used for a brand new listing on LendingClub without payment history. Random Forest was the perfect candidate given the overfitting problem and its proven ability to bootstrap subsamples of features and aggregate many decision trees. In simple terms, a Random Forest would be able to construct and combine many not-so-good models that are not correlated with each other to create a single, decent model — all while being equipped to handle my 1,107 features without much additional preparation besides finding the best parameters. Given the power of a Google Cloud GPU, I let GridSearchCV handle this by searching for the best combination.
Underperformers: 1 Linear Regression model, 3 Ridge Regression Models, 1 Keras Neural Network, 4 Random Forest Regression models (using Scikit-Learn)
Best Performer: Random Forest Regression v5 with GridSearch Cross-Validation
Best Model Parameters:
Best Model Evaluation Metrics (Test Error — not including payment history):
The best Random Forest regression model achieves a root-MSE of 0.16 on the test set, which implies that the predicted annualized return is estimated to differ from the true annualized return by 0.16. While this may appear very large at first, the model can be very useful in formulating a loan selection strategy. Loan defaults usually happen soon after loan funding, and the chance of default decreases as more payment is made. As a result, I recommend loans with annualized return predictions higher than a reasonable threshold set by the user. Intuitively, the threshold can serve as a parameter investors can tune according to their investment account size: the bigger the minimum annualized return (higher return) and the smaller the maximum probability of default is (lower risk), the more stringent the overall loan selection is, so less total dollars can be invested. Hopefully, the annualized return will be higher due to investing in loans more selectively.
That said, we can become much more confident in our prediction if we were to know something about the borrowers’ payment history, though in practice we won’t be able to incorporate initial borrow payment behavior as an investor. That said, with a bit of payment history to work with, the model improves significantly:
Evaluation Metrics (Test Error — if payment history is included):
This app is free to use. There are no fees or benefits to me should you decide to apply my models to your own investment portfolio. I am not a certified financial advisor of any kind — invest at your own risk.
The model has been deployed on the web as a Flask app hosted on an Amazon Web Services EC2 instance, utilizing HTML, CSS, and Brython. It can collect a user’s maximum risk tolerance and their corresponding minimum desired portfolio return, and output a comprehensive list of loans currently live on LendingClub.com based on my model’s recommendation.
Note About Optimal Diversification:
Thank you for reading,
Let’s connect! I encourage you to like, comment, share, or message me directly with your thoughts on the ideas presented here, or suggestions on interesting topics I should look into going forward.
Create your free account to unlock your custom reading experience.