Good risk models or “weapons of math destruction”?

“The [2008] crash made it all too clear that mathematics, once my refuge, was not only deeply entangled in the world’s problems but also fueling many of them.” —author Cathy O’Neil, then a hedge fund data scientist

There are situations where, if all actors operate to maximize their own self-interest, a detrimental impact will be had on the whole. This concept, called “The Tragedy of the Commons” was first identified in Victorian England by economist William Forster Lloyd.

Lloyd framed this concept at a time when shared pasturage, owned “in common,” was still utilized. Lloyd offered this idea: Let’s say that every herder with rights to a common grazes as many animals as possible, acting in self-interest for the greatest short-term personal gain. Eventually, they will use up all the grass in the pasture. They will destroy the common.

You’d suspect—at least, hope—that we’ve learned and grown since Victorian times. While in most areas we have, this scenario still exists. Contemporary examples include overfishing, traffic congestion, and the rise of drug-resistant diseases.

Add to this list the irresponsible use of data models, as targeted by data scientist Cathy O’Neil in her excellent book, Weapons of Math Destruction.

Power of data and risk of destruction

As with herders in Victorian England, banks today face a quandary:

Continue the irresponsible (as defined by author O’Neil) use of data models as the fuel and engine of client engagement and the resulting sale—or lose competitive edge.

Reliance upon data models, it seems, is great for individual companies, but ungoverned the collective use of them can be bad for society as a whole.

This is the dilemma posed by O’Neil. She doesn’t place it in those terms, but the concerns she raises on the proliferation of analytical tools is real, as would be the consequences for a financial institution, if it were to not leverage data models, while others took advantage.

Power of models

Big Data, AI, machine learning. Pick up any industry publication or visit many banking websites and there will be at least one article evangelizing the need for financial institutions to invest in, leverage, and master these capabilities.

Why?

Every institution owns a wealth of data on their clients. All of that can be easily augmented by appending data from third-parties. More can be done with that data than most consumers realize. The challenge has become how to leverage that data to improve client engagement in today’s digital world, where most consumers don’t regularly speak with a banker.

This drives the need to make the most of every interaction, regardless of channel. Ensuring we are communicating on what is most relevant to that client requires harnessing the available data and creating scalable systems to present an offer, initiate a discussion, or address a potential service issue.

As O’Neil defines it, a model is an abstract representation of some process. It takes what we know and tries to predict responses.

Models then, are being used to directly determine when, where, and for what purpose we engage with our clients. Big Data, AI, and Machine Learning are all methods to improve our ability to consume available data and deliver more accurate results from our models—and more accurate models themselves.

This can be incredibly powerful and beneficial to all parties:

• For the institution, it allows for the matching of the message, offer, or action with those clients and prospects identified as the best recipients.

• For consumers, it enables the receipt of offers or messages best suited to their individual needs.

This should be—and in many cases is—a win-win.

The challenge comes when the models are poorly designed or misused, creating what O’Neil calls “weapons of math destruction”.

Where are the WMDs?

O’Neil calls “weapons of math destruction” the “dark side” of big data. This is where flaws in how models are designed and managed can lead to inadvertent bias against certain groups, or worse, intentional harm.

An example of inadvertent harm shared by O’Neil in her book occurred at American Express. In 2009, as a response to the emerging Great Recession, Amex sought to reduce the risk on its balance sheets by reducing credit lines for certain high-risk cardholders.

What made them high-risk? Amex used a model that showed a strong correlation between consumers who shopped at certain merchants and the likelihood that they would fall behind on payments.

These cardholders were notified that their lines were being cut based on where they shopped. As a consequence, the lower limit was reflected on their credit reports, driving up borrowing costs for these cardholders.

O’Neil notes one ticklish point: “It was up to the unhappy Amex customers to guess which establishment had poisoned their credit. Was it the weekly shop at Walmart or perhaps the brake job at Grease Monkey that placed them in the bucket of potential deadbeats? Whatever the cause, it left them careening into a nasty recession with less credit.”

The resulting client anger became a story in The New York Times and Amex was forced to concede it would no longer correlate stores to risk. (In a response to the Times, it claimed the letters were poorly worded.) By this time, the damage was done. 

It is more than the existence of inadvertent harm which causes O’Neil to designate a model as a WMD. Other considerations include:

• Opacity—Lack of transparency into the algorithms upon which the model is based; in addition, lack of clarity regarding the actions consumers can take to improve their scoring.

• Scale—Large swaths of consumers have the potential to be impacted by the model.

• Damage—Results of the model can have significant negative impact on consumers; either through the presence of “false positives” and “false negatives.”

With great power comes great responsibility

Not every data model is a WMD. The use of models is prevalent in every industry and done right, they make an organization more productive, efficient, and effective. They can enable a richer client experience.

The challenge is not to avoid use of models, but to ensure they don’t devolve into a WMD. Beyond the Amex story, O’Neil provides several other examples from which banks can learn.

As I reflected on how to approach the use of models, the sage guidance of Peter Parker’s Uncle Ben resonated with me “with great power comes great responsibility.”

In practical terms for financial institutions this means:

• Establish clear objectives. It is easy to create a WMD if there is misalignment or a lack of clarity on goals and objectives.

• Create feedback loops. Work to measure and monitor the accuracy of the model.

• Never be satisfied. Always challenge your assumptions. And remember, correlation is not causation.

• Continue to test, measure, learn.

• Design for the benefit of the client. Don’t devolve into product pushing/fee generation.

Implied in the list above is this: Be ready to commit the time and resources in validating and managing your model. This can be the difference between the damage that can be done by handing a chainsaw to a four-year-old and realizing the benefits from your investments in data management.

Back to the pasture

So where does the Tragedy of the Commons thing come in?

Consumer behavioral changes, economic pressures, and the growing availability of solutions designed to help financial institutions make the most use of their client data all point to continued growth in the use of data models.

If the result for individual companies is the creation of WMDs, even with the best of intentions, as an industry we are inviting greater regulatory oversight/guidance and potentially most damaging of all, the loss of consumer trust—our most valuable asset.

O’Neil describes the tension between fairness and efficacy. Being able to leverage our competitive advantages in access to data and engagement-points with consumers will require financial institutions to assign fairness far greater weight than efficacy.

We are looking for additional banker-reviewers. If you are interested in volunteering to write reviews for BankingExchange.com, send an email with a writing sample to Steve Cocheo, executive editor and digital content manager.