If you aren’t approaching model risk management correctly, you’re putting your organization in a compromised position, because your models are being underutilized or used blindly in your decision making. An effective approach, on the other hand, gives your organization strategic advantages.
This summary statement was provided by Jeff Prelle, Managing Director of Analytics and Head of Risk Modeling at MountainView Financial Solutions, a Situs company, in a recent interview about why model risk management isn’t simply a periodic exercise involving a list of functionality checks.
While deriving strategic advantages from model risk management involves many principles, Prelle said the most basic principle is to continually evaluate a model based on its originally intended use. Over time, a business line may try to use a model for multiple functions related to its intended use. You ultimately need to look at modeling technique, regulatory scrutiny, statistical complexity and statistical validity related to the intended use; that way, your strategic advantage will come from ensuring that you’re relying on a model or modeling technique that’s well suited for the function.
Your approach to model risk management should also acknowledge that all models degrade over time, and this degradation can affect performance, according to Prelle. For example, a model implemented 10 years ago may be producing results that don’t reflect current conditions, even though your business knowledge and theory have changed significantly over 10 years. By tempering current business theory with an understanding of your model’s factors and limitations, you will know when you should re-estimate the model specification and when you should re-calibrate the model.
“Models are never going to be perfectly accurate, but you want them to be directionally correct in utilizing business knowledge,” Prelle said. “Ongoing re-calibration helps to keep models directionally correct.”
According to Atul Nepal, Assistant Vice President of Analytics at MountainView, your ongoing model risk assessments should also consider all of the data a model is built to capture and generate, and you should analyze how this data fits into the modeling framework – how it fits into the business theory behind the decisions that need to be made. The model was at one time built off of the perfect data set, and you’ve probably been cleansing all of the data as part of your model governance. But what if your data set has degraded over time? Nepal said ongoing reviews of the efficacy of your data will give you renewed confidence about your modeling.
Nepal also stated that ongoing model risk assessments should continually confirm whether your hypotheses about the outcome drivers are correct. At some point, you may realize that you’ve been mistaken about a main driver – you may realize that a factor was highly correlated but not an actual driver, and that you therefore were receiving the wrong results and making misinformed decisions. By continually confirming your hypotheses about the drivers of outcomes, you gain a strategic advantage in figuring out whether the primary connection can be tied back to your business theory.
On this point, Prelle added that the customers of a financial business are more informed every day, and this means the drivers are constantly changing, so the most important principle is that model risk should continually be re-assessed.
“You’re always trying to model behavior, and that behavior is unpredictable in a lot of cases,” said Prelle. “That’s not saying you shouldn’t try to model this behavior, but you need to be cognizant that it’s changing and unpredictable, and you need to be able to change and adapt with it. Ongoing model risk assessment helps you to constantly identify what’s changing.”