Data, documentation, monitoring: the best practices of AI lending
Credit unions considering switching to AI-based lending always want to know about three big things: what kinds of data should I be using, how do we document an AI/ML credit model, and how do I make sure my results hold up over time?
We thought it would be helpful to answer those questions here:
Data
Machine learning algorithms have the ability to consume virtually limitless data types from unlimited sources in limitless quantities to make predictions. However, the Fair Credit Reporting Act (FCRA) limits what data can be used for credit underwriting, so credit unions need to ensure that they’re using fully FCRA-compliant data sources in their underwriting. That means those “virtually limitless” data types and sources that machine learning can use is cut down significantly — but that doesn’t make machine learning algorithms obsolete for your organization.
One type of data — alternative data — has become all the rage these days. Alternative data provides plenty of new cash-flow evidence like rent, utilities, cellphone payments, and other data that can boost thin-file scores. While there are great new platforms for sourcing open data, for now, you still need to make sure your data is FCRA compliant.
As an aside, we find that a richer use of standardized bureau data remains an untapped resource for data collection. We get a lot of signals out of the bureau data that help our partners lend further down the spectrum and to more thin-file folks.
Documentation
The AI/ML approach will look similar to what you already do… with some twists. For example, ML models usually require more transparency in feature analysis and validation because their variables don’t always point in the same direction. Plus, there are many more of them.
That said, the documentation doesn’t have to be a chore. Zest AI’s technology comes with an Autodoc application that, at the push of a button, produces a model risk management report compliant with the SR 11-7 model risk management standard set in 2011 by the Fed and OCC, the FDIC Financial Institution Letter 22-2017, and the NCUA’s Corporate Credit Union Guidance Letter 2013-02. This report is dozens of pages long and explains model development, implementation, and use — along with the soundness of its validation, governance, policies, and controls. Honestly, you may only need the executive summary for your purposes though.
Either way, this technology is a chance to up your compliance game. And now is a good time to do so, as federal and certain state regulators discuss imposing new or additional documentation requirements on risk models. Sure, those requirements could be two years or ten years out. Still, to the extent that you’re not already prudentially regulated, you should be thinking about model documentation because it is a regulatory regime that’s coming down the pike.
One last note about documentation: Take special care to show your validation steps. All prudential regulators require that models be validated to ensure that they’re accurate and performing as expected. But, because machine learning is new to most regulators, they will scrutinize your validation work more than they would if you used a traditional underwriting method. So you should be prepared to show at least three essential requirements for a solid validation process, which we share in the next section.
Monitoring
1. Evaluation of conceptual soundness: You’ll need to assess the quality of your model design and construction quality and how careful you were to keep within common industry practice. This section is where you justify the modeling method used and why it’s appropriate for the purpose. For ML, it could be as straightforward as saying this model is “intended to improve the risk assessment of loan applications to better support underwriting decisions to increase approvals and/or reduce losses within the loan portfolio.”
2. Ongoing monitoring: Many lenders mistakenly believe that underwriting algorithms are living, breathing organisms that learn and adapt over time. Far, far from it. AI lending models are trained and then locked down to pass validation. But they can drift a little if you’re not watching. Monitoring is essential to ensure the model isn’t losing predictive accuracy, especially as products, clients, and market conditions fluctuate.
3. Outcomes analysis: This step involves comparing model outputs to corresponding outcomes. One way that we recommend to do this is to back-test the trained model using an “out-of-time” period not used in model development but at a roughly similar frequency as the model’s performance window. You can also run stress and scenario tests on a model, seeing what would happen if every one was missing bankruptcy data or every applicant had a thin file.
People say that getting an ML model through a compliance review is impossible. That’s not the case.
While your first pass will take longer because regulators can be unfamiliar with it, the truth is that many regulators, in our experience, look favorably on this kind of move because these models tend to be more accurate and fair. So, it is not an impossible task. In fact, it is a task that will only get easier with time as more industry participants and examiners get used to it.
AI-driven lending can transform your organization’s growth and bottom line. Putting in the time to understand the results and prepare your organization early will reap outsized rewards.
