Getting model risk management right

By Michael Guglielmo, managing director, Darling Consulting Group 

A year ago author Mike Guglielmo set out the basics of model risk management and the three lines of defense as regulators first began to focus on this, in “Model risk management meets three lines of defense.” Now he digs in deeper.

Risk management is part of the daily routine of a banker—after all, banks are in the business of taking risk. As part of banks’ financial and operational risk assessment and decision-making processes, they need timely and relevant information to inform evaluation of potential risks and their relative impact.

To facilitate this process, we use a combination of data, models, and business judgment. Combined, these help paint the picture and inform our everyday decisions.

Bad data or bad models lead to poor decision making. But how do we know the data or models that we rely upon are accurate and reliable? And who bears this responsibility?

The answer: We all do!

Model risk management and the three lines of defense

From the 2011 regulatory guidance on model risk management has emerged the practice of the “three lines of defense”:

1. Model developers, owners, and managers.

2. The model risk management function.

3. Audit, along with the board of directors and examiners.

Each area plays an important, collaborative role in ensuring the data and models used are appropriate, accurate, and well managed. Regardless of institution size and complexity, “effective challenge” must come from all three stakeholders, a concept that larger banks have addressed with a significant increase in resources, education, and effort over the past few years.

Community banks have been slower to adopt more substantive model risk management processes. This is due in part to a lack of perceived risk, limited regulatory emphasis, or financial constraints.

However, as the need for—and dependence on—risk modeling and analysis grows, the potential impact of being wrong grows more profound. The responsibility of effective challenge needs to be addressed and shared across the organization.

Policy development and education

Community bankers are developing more formalized model risk management policies with greater frequency. 

Model risk management policies lay out an institution’s MRM framework and specify the roles and responsibilities from the board to the model developers, owners, and operators. Such policies:

• Identify all the models being used by the organization. This is often referred to as the “model inventory.”

• Assign a risk-rating to each model (i.e. low, medium, and high).

• Set the established review frequency for each model by risk level and the validation methods or standards to be applied.

• Establish the institution’s overall model lifecycle management process. This includes development or acquisition; implementation and testing; documentation; and ongoing monitoring and retirement.

Developing the policy itself may not seem like a significant undertaking. However, what is challenging is increasing both officers’ and directors’ understanding and awareness of model risk and their respective roles, along with implementing the various elements with model developers, owners, and managers.

Getting this done requires leadership, education, and time and financial commitment.

Good model stewardship proves critical

The foundation of successful model risk management ultimately lies with model developers, owners, and operators.

As the term “first line of defense” implies, they are on the front line of this process. They create a model, or select a vendor model, based upon a specified need; implement and test it; document it; and manage its ongoing use.

These steps are common and have been standard practice well before the rise of modern model risk management practices. However, organizations—particularly community banks—are struggling with:

• The degree to which these activities now need to be conducted.

• The level of governance and oversight associated with these actions, including data management, change control, and formal review and sign-off procedures.

• The additional expectations for ongoing performance monitoring (i.e. back testing, outcomes analysis, and assumption sensitivity testing).

• The level and depth of documentation.

All that said, the first line of defense holds the key to model risk management success. A well-documented, well-managed model with performance demonstrated by meeting its intended purpose is going to result in a favorable model risk management assessment. This, in turn, will generate confidence with auditors, the board and examiners.

By establishing a model risk management policy, organizations can inform model owners and their sponsors (senior and executive management) with the desired level of care and documentation expected. Well-written model risk management policies can also serve as a means to educate all of the stakeholders with regard to their contribution to model risk management success.

Documentation: heart of good MRM

Documentation associated with a model and the related modeling process is key to successful model risk management.

The ability to sufficiently document a model is one of the fundamental challenges we see with existing models in relation to newly developed corporate model risk management policies. Having substantive documentation—and keeping it current—has been considered important.

However, documentation often takes a back seat to model managers’ other priorities. But this is quickly being reprioritized. That’s because such documentation represents a key component relied on by model risk management experts and validators. This documentation not only signifies the care and effort taken to properly manage a model, but also provides the benchmark for comparison and validation/effective challenge.

Traditional model documentation has most often simply represented the procedures associated with the model management/update processes. The documentation has been written from a model operator’s perspective. While this is still an important aspect of good documentation, it is only one facet the documentation now needs.

Elements of good model documentation

Let’s consider how to put together sound model documentation.

First and most importantly, model documentation should convey the purpose of the model to stakeholders, along with its intended use. This aspect proves especially important in regard to those stakeholders, such as outside directors, who may not be experts with regard to the particular discipline.

Setting forth the model’s purpose is often addressed within an executive summary. The summary should also include a high-level description of the model; highlight key aspects, drivers, or assumptions related to the model; explain how the model has performed; and list any known limitations or risks associated with the model.

This narrative should be written in layman’s terms. It ought to provide a clear and succinct explanation of the model. This gives the second and third lines of defense a defined purpose against which the model’s performance can be judged.

Second, the model’s theory and design should be thoroughly described.

Providing insight into the philosophy behind the design, development, or selection of the model, this aspect of the documentation is the place to describe the mathematical construct, the data, variables, assumptions, selection processes, and the architectural considerations.

Essentially, you want to describe why the model was built; what the general design is; what the model does; and how the model was developed (or selected, if obtained outside the bank).

These are basics. Yet organizations often struggle with this aspect of documentation because many legacy models were never documented.        

Third, considerable effort should also be invested in data, assumptions, and output documentation—the nitty gritty of the model.

A listing of all the data and how it is obtained, prepared, transformed, reconciled, and applied should be assembled.

For assumptions, documentation should include a listing of all of the assumptions, and how they are developed and supported; reviewed; tested; and approved. In addition, activities related to assumption sensitivity testing should be expressed, along with a narrative that highlights the key assumptions and their relative impact on model results.

This allows the stakeholders to appreciate that results are not absolute. This also makes it clear that results depend heavily on assumptions made. The degree to which each key assumption affects results is also important to set out.

Models often include assumption overlays or overrides—factors that are applied to assumption logic or outright replacements of quantitative assumptions, when warranted. In these instances, these overlays or overrides need to be highlighted along with the rationale/support for the adjustments.

A critical component of good model management practices (and documentation) relates to ongoing performance monitoring. Stakeholders need to understand the efforts being employed to confirm the regularity. They must also be shown that there are protocols to address exception handling.

Lastly, everything related to model governance needs to be thoroughly documented. Everyone’s respective roles and responsibilities should be represented—including the various levels of review and oversight, change control, change logs, etc.

Other documentation considerations

Don’t forget the varying technical expertise of your stakeholders—organize your documentation with an emphasis on readability. You might even consider a glossary of terms—don’t assume everyone knows all the technical language.

Additionally, write your documentation with a thought towards manageable change—because change will come. This could include use of an appendix section for details that may be adjusted with regularity.

Lastly, take advantage of electronic mapping. Publish your documentation with an interactive table of contents, hyperlinks to appendices, etc. Easy navigation through your documentation will go a long way with helping your stakeholders use what you’ve built.

Don’t forget to practice what you preach

It is one thing to document a “desired” process. It is another thing to actually follow through and do what you say you are going to do.

Make sure whatever you document represents what you are actually doing in practice and there are demonstrated/verifiable steps. The worst thing you could do is to document a process and not follow it.

Doing so is an instant recipe for increased regulatory scrutiny.

Moving forward

Regardless of size, examiners are expecting institutions to adopt more substantive model risk management practices. At the heart of this heightened activity is good documentation. Good documentation leads to better, high-performing models with improved development and testing disciplines; increased transparency and understanding; and an ability to leverage strengths of other good modeling practices.

In addition, good documentation has a direct benefit to the bottom line as development, validation, and remediation costs decrease, potential operational risks are reduced, and more confident stakeholders make more proactive strategic decisions.

Oct. 13 Darling Consulting and Banking Exchange will present a web seminar, "Coping With Rate Uncertainty." The event will feature DCG President Matt Pieniazak and bankers Tracy Bacon and Steve Ward. Learn more/enroll

About the author

With nearly 30 years in strategic risk management, Michael Guglielmo, managing director, at Darling Consulting Group, provides both technical and strategic consulting to a diverse group of financial institutions in the U.S. and abroad. He is also a frequent author and top-rated speaker on a variety of balance sheet management topics. During his tenure at DCG, he has served in various capacities including Director of Financial Analytics. Prior to joining DCG, he managed the asset-liability management and strategic planning process for a regional bank in the northeast.

Guglielmo currently serves as Finance Chair on the Financial Managers Society Board of Directors and as a faculty member of the Association for Management Information in Financial Services Institute.