Regulators require the model validation process to ensure that financial institutions are properly modeling for risk. Beyond regulatory compliance, model validation also provides business leaders with confidence in their models and helps to reinforce or reassess business and balance sheet decisions shaped by model outcomes. Considering the importance of model accuracy and effectiveness, it is critical to understand hidden risks in model validation practices. Whether a financial institution develops its financial models internally or works with a third-party model software vendor, it is critical to ensure your model validation partner understands and considers hidden risks in model validation. Some such risks include:
1. Inadequate or fragmented data.
The accuracy of your model relies on data. If the data used in your model is limited, fragmented or low in quality, your model will suffer. Ensure your model validation partner is asking the right questions about how your model utilizes data, how it sources, supplements and transforms data, and how it accounts for missing data values. Including a full-data validation will provide confidence in the data quality and the extract-transform-load (ETL) process and make the model more accurate.
2. Lack of third-party model details.
Models that are developed by third-party software vendors provide financial institutions with many benefits, including ease-of-use and peer comparison. However, one drawback in utilizing these models is the lack of transparency about what data is supplied within the model and any assumptions, macroeconomic trends and unique model features that contribute to the outcomes. When validating its third-party model, a financial institution should supply its model validation partner with supporting documentation and insights that may help “get under the hood” of the model without jeopardizing its proprietary nature. The more information provided, the more thorough your model validation.
3. Technical model challenges
Many smaller institutions using model software must rely completely on the technical integrity of the model vendor. In most cases, the model vendor will ensure that its models are performing optimally from a technical standpoint, as it is the crux of its business. While a financial institution is responsible for implementing and customizing the model to incorporate institution-specific scenarios and parameters, this can potentially create technical errors in the process. The model validation needs to focus on the set-up attributes that are applied to define how the data is utilized.
As financial institutions move forward in obtaining a financial model validation, it is important to ask your validation provider if it is aware of some of the hidden validation challenges and risks and how it assesses a model when such risks are present. By proactively and openly discussing potential validation challenges, the model validation provider will be able to focus on areas of the validation that pose a potential problem and may be able to help the financial institution use the model validation as a stepping stone to improving the model and model process.