Introducing PeerVest: A free ML app to help you pick the best loan pool on a risk-reward basis
Problem
Peer-to-peer lending marketplaces like LendingClub and Prosper Marketplace are driven by what is essentially a brokers fee for connecting investors and borrowers. They are incentivized to increase the total number of transactions taking place on their platforms.
Driven by ease-of-use, their off-the-shelf credit risk assessments are scored in grouped buckets. On a loan-by-loan basis, this is inefficient given each loan’s uniqueness and the sheer amount of data collected from borrowers. Scoring risk on a more granular, continuous basis is not only possible but preferable over discrete, grouped buckets.
Credit risk is something all peer-to-peer lending investors and bond investors must carefully consider when making informed decisions. Institutions, including large banks, have been employing researchers and quantitative analysts to wrangle and analyze this data hoping to become more confident in their risk-reward assessments.
What is the underlying risk that a borrower fails to make required payments, leading to a loss of principal and interest? Given the terms of the loan, what return can an investor expect? There should be a more simple and accessible tool for the everyday investor that can hope to outperform the marketplace’s own off-the-shelf advisor.
Solution: PeerVest
PeerVest helps individual investors augment their portfolio by intelligently allocating funds to Peer-to-Peer Lending Marketplaces using machine learning trained on LendingClub.com historical loan data to assess risk and predict return.
PeerVest recommends the best loans to invest in given a user’s available funds, maximum risk tolerance, and minimum desired annualized return. There are plenty of institutions that are thought leaders in utilizing modern, alternative datasets to assess credit risk and predict investment potential, but I’ve simplified the problem into 2 key models: (1) scoring risk by predicting the probability that a loan defaults, and (2) predicting annualized returns.
Important Terms & Concepts
- Peer-to-Peer Lending (aka P2P, Crowdlending): the practice of lending money to individuals or businesses through online services that match lenders with borrowers. Because of their online-only nature and low overhead, generally, lenders can earn higher returns compared to savings and investment products offered by banks, while borrowers can borrow money at lower interest rates.
Source: Business Insider
- Artificial Neural Network: a computational learning system that uses a network of functions to understand and translate a data input of one form into a desired output, usually in another form. Inspired by the way neurons of the human brain function together to understand inputs from human senses. Learn More Here
- Random Forest Model: an ensemble learning method for classification, regression and other tasks that operates by constructing a multitude of decision trees at training time and outputting the class that is the mode of the classes or mean prediction of the individual trees. Learn More Here
- Probability of Default: an estimate of the likelihood that a borrower will be unable to meet its debt obligations. Learn More Here
- Annualized Return: returns over a period scaled down to a 12-month period. The formula I used: AR = (xTP / xLA) ^ (365/D) — 1, where xLA is the loan amount, xTP is the total payment made by the borrower, and D is the number of days between loan funding and date of last payment. Learn More Here
Predicting the Probability of Default
To assign a quantity to each loan that measures its risk level, or the probability that the borrower will default on the loan, I approached this like a standard classification problem where each historical loan is assigned a 1 if it was Fully Paid and a 0 if it defaulted. To provide a more granular risk assessment than LendingClub and sites like it provide off the shelf, I was looking not for the class (1 or 0), but for the probability that each loan is predicted to fall in each class. My initial approach was to use a Logistic Regression, which despite classifying each loan quite confidently, and despite probability calibration, didn’t seem to interact well with sklearn’s .predict_proba() function. This function returns a probability between 0 and 1 that a loan belongs to class 1. Taking 1 minus this very number yields an interpretable probability of default metric (the larger it is, the higher likelihood that it will default).
Given 1,107 unique features, there seemed to exist many non-linear relationships and the effort of manually reducing dimensions would be cumbersome. There was no true benefit to using Principal Component Analysis since it was not improving the situation — it reduced dimensionality and reduced explained variance in equal magnitude. I decided to keep all the features, despite the increased likelihood of feature redundancy. Knowing neural networks would be able to handle this complexity over many epochs of training, I began architecting a model using Keras, a neural network library built on TensorFlow. I iterated through many different Keras architectures by employing different combinations of hidden layer sizes, dropout regularization, L2 kernel regularization, early stopping, epochs, class weight balancing, and ReLU/Sigmoid activation functions. Despite some stagnation in validation loss, the model eventually broke through and began improving more significantly within a few epochs, for both train and test sets, while calibrating prediction probabilities very well.
Underperformers: 3 Scikit-Learn Logistic Regression models, 7 Keras Neural Networks
Best Performer: Keras Neural Network v8
Best Neural Network Model Architecture:
Input Layer (1107, Sigmoid) ->
Dense (1000, Sigmoid, L2) ->
Dense (300, Sigmoid) ->
Dense (50, Sigmoid, L2) ->
Dense Output Layer (1, Sigmoid, L2)
Best Model Parameters:
- Loss: Binary Cross-Entropy
- Optimizer: Adam
- Early Stopping
- Class Weight Balancing
- Batch Size: 5000
Best Model Evaluation Metrics (Test Error):
- Precision Fully Paid: 0.943
- Recall Fully Paid: 0.979
- F-1 Score Fully Paid: 0.961
- ROC-AUC Score: 0.86
- Prediction Probabilities calibrated
Predicting Returns
I used a very simple Annualized Return calculation based on the available data from LendingClub. I decided against using LendingClub’s complicated Adjusted Net Annualized Return computation due to interpretability, though as I continue iterating my model I am open to refined calculations that utilize more traditional financial loss estimates. The element of time as it relates to loan terms and payments sits within this calculation. Macro information like the federal interest rate, inflation and the time value of money is inherent in the interest rate quoted to the borrower, though there is certainly room to dive into time series and survival analysis with increased rigor to productionalize this model for financial institutions.
Linear Regression worked decently on the training set, but was overfit and did not generalize well to the future set or the test set within the same period. Ridge Regression, which is similar but applied L2 regularization fixed the overfitting issue in my 2nd iteration. That said, I wanted to know if my evaluation metrics would improve using something that could make regression decisions differently, and if there was a model that wouldn’t be as reliant on total payment so that my prediction could be used for a brand new listing on LendingClub without payment history. Random Forest was the perfect candidate given the overfitting problem and its proven ability to bootstrap subsamples of features and aggregate many decision trees. In simple terms, a Random Forest would be able to construct and combine many not-so-good models that are not correlated with each other to create a single, decent model — all while being equipped to handle my 1,107 features without much additional preparation besides finding the best parameters. Given the power of a Google Cloud GPU, I let GridSearchCV handle this by searching for the best combination.
Underperformers: 1 Linear Regression model, 3 Ridge Regression Models, 1 Keras Neural Network, 4 Random Forest Regression models (using Scikit-Learn)
Best Performer: Random Forest Regression v5 with GridSearch Cross-Validation
Best Model Parameters:
- n_estimators = 100
- max_depth: 15
- min_samples_leaf: 4
- min_samples_split: 2
Best Model Evaluation Metrics (Test Error — not including payment history):
- R-Squared: 0.56
- Mean Squared Error: 0.02
- Root Mean Squared Error: 0.16
The best Random Forest regression model achieves a root-MSE of 0.16 on the test set, which implies that the predicted annualized return is estimated to differ from the true annualized return by 0.16. While this may appear very large at first, the model can be very useful in formulating a loan selection strategy. Loan defaults usually happen soon after loan funding, and the chance of default decreases as more payment is made. As a result, I recommend loans with annualized return predictions higher than a reasonable threshold set by the user. Intuitively, the threshold can serve as a parameter investors can tune according to their investment account size: the bigger the minimum annualized return (higher return) and the smaller the maximum probability of default is (lower risk), the more stringent the overall loan selection is, so less total dollars can be invested. Hopefully, the annualized return will be higher due to investing in loans more selectively.
That said, we can become much more confident in our prediction if we were to know something about the borrowers’ payment history, though in practice we won’t be able to incorporate initial borrow payment behavior as an investor. That said, with a bit of payment history to work with, the model improves significantly:
Evaluation Metrics (Test Error — if payment history is included):
- R-Squared: 0.97
- Mean Squared Error: 0.001
- Root Mean Squared Error: 0.04
How To Use the PeerVest App with LendingClub.com
This app is free to use. There are no fees or benefits to me should you decide to apply my models to your own investment portfolio. I am not a certified financial advisor of any kind — invest at your own risk.
The model has been deployed on the web as a Flask app hosted on an Amazon Web Services EC2 instance, utilizing HTML, CSS, and Brython. It can collect a user’s maximum risk tolerance and their corresponding minimum desired portfolio return, and output a comprehensive list of loans currently live on LendingClub.com based on my model’s recommendation.
- Visit PeerVest: www.peervest.online
- Download the Recommendation CSV
- Use Loan IDs in the downloaded CSV to guide your investments on LendingClub.com! Click Path: Click Here > Login (or Create Account) > Invest > Manual Investing > More Filters > Filter by Loan ID > Add $
Note About Optimal Diversification:
- There are often less than 100 available investments at any given time on LendingClub.com, though an optimized loan portfolio should contain at least 100 different loans to adequately reduce lack of diversification risk. In practice, I would continue using the app to guide new investments over a few weeks. Get to that 100-loan portfolio as quickly as possible to properly diversify.
- From LendingClub: Accounts with fewer than 100 total Notes have been much more likely to see negative returns
Future Work — Sample User Stories
- I want to know if the app recommendations will definitely make me more money than the off-the-shelf LendingClub/Prosper recommendations
- I want all new listings to be featured in the recommendation set so that my investment decision is based on recent updates
- I want to be able to filter by loan purpose so that I can curate a mission-driven loan set
- I want to see how the P2P lending platforms set interest rates over time
- I want to see a Python code output that I can use to execute my own orders based off of the recommendation set to reduce manual allocation time
Thank you for reading,
Alex
Let’s connect! I encourage you to like, comment, share, or message me directly with your thoughts on the ideas presented here, or suggestions on interesting topics I should look into going forward.
LinkedIn | GitHub Portfolio | PeerVest GitHub Repo | Medium Blog
