Covering Disruptive Technology Powering Business in The Digital Age

Big data is the new next big thing that everybody, from governments to banks and universities, are embracing. But what if mathematical models are not only entangled in the world’s problems, but fuelling many of them too? Cathy O’Neil’s book sounds a discordant note in the choir of big data.

WEAPONS of Math Destruction is the Big Data story Silicon Valley proponents won’t tell. The author, Cathy O’Neil, is a former academic mathematician and ex-hedge fund quant at D.E. Shaw, once part-owned by Lehman Brothers.

Her book pithily exposes flaws in how information is used to assess everything from creditworthiness to policing tactics, with results that cause damage both financially and to the fabric of society.

O’Neil had an epiphany in the aftermath of the 2008 financial crisis: mathematical models were “not only deeply entangled in the world’s problems but also fueling many of them.” A convert’s passion runs through the book as the author explores how math can flip over to the dark side — flawed algorithms, applied widely, become the WMDs of the title.

Take, for example, recidivism models used by many US states in sentencing convicts. The idea is to eliminate the biases and inconsistencies of individual judges, and that’s positive. Drawing on all kinds of data, the models try to gauge how likely a criminal is to offend again. Sentencing can be tailored accordingly.

Data directly concerning the individual, such as prior convictions, can shed meaningful light on the potential for recidivism.

But O’Neil points to questionnaires used to gather inputs for these models in which prisoners are asked about things like prior involvement with the police, whether their friends have been arrested and so on.

These are environmental factors, which at best are proxies — a concept referred to frequently in the book — for how an average individual might behave. At worst they are proxies for simply growing up in a poor neighborhood or, sometimes, being black.

“The question … is whether we’ve eliminated human bias or simply camouflaged it with technology,” she said.

Such models are often proprietary black boxes, produced by for-profit companies. It is impossible, therefore, to tell exactly how the inputs influence the output. That makes it equally impossible for a convict to appeal against his score, which assumes something like the force of law.

Some police forces also use models to decide where to concentrate patrols. Again, the goal is benign: to identify where serious crime is most likely to occur and head it off. Proxies, however, can cause trouble in these big data algorithms, too.

There’s the additional concern that the police have to do something, so if they spend time in a particular neighborhood they will crack down on minor crime, perhaps condemning residents to a life of interaction with cops for small infringements that would never be acted upon in other, luckier areas.

The first ingredient in the author’s list of “bomb parts” for WMDs is opacity. The others are scale and damage. The potential for real-life harm to particular groups of people from bad recidivism or policing models is obvious. The scale may be limited for now but such approaches, if perceived as useful, tend to spread.

Another characteristic of O’Neil’s weapons is that they are rarely tested and improved by comparing real life with what they predict. The Big Data algorithms of online retailer Amazon, for example, are constantly refined with the simple goal of selling more goods, more profitably.

It’s unlikely, though, that enough research goes on as to whether American felons’ actual recidivism bears any relation to the scores the models spit out.

O’Neil ticks through examples like this involving teacher evaluation, university applicant selection and other processes where imperfect and sometimes completely spurious results can lead to outcomes like top-notch educators being fired. Yet if black boxes seem to work well enough, organisations that use them are unlikely to stop.

The alternative is applying much more human judgment, which is costly — and carries its own risks.

The author recognises that many WMDs start with the admirable intention of eliminating human biases. America’s ubiquitous credit scores, in their basic FICO form, largely escape O’Neil’s ire because the inputs relate directly to the individual — they are not dodgy proxies. And their effect on the output is transparent.

When entities like insurers and financial startups combine them with other data to create so-called e-scores, however, they risk morphing into opaque WMDs.

The book is a thought-provoking read for anyone inclined to believe that data doesn’t lie. Among others, there’s a timely example about the potential for Facebook’s newsfeed algorithms to distort how users see the world and whether and how they vote in elections — a controversy that briefly erupted in May.

Addressing the dangers of WMDs is, of course, a challenge. Fairness and the common good are hard to model mathematically, as O’Neil notes.

Starting with the recognition that algorithms are fallible is one element. Regulation could be another by, say, imposing some human judgement on the machines. Requiring models to incorporate feedback — essentially testing and refining them — or at least auditing them could be part of the answer too.

In some areas, the author suggests extending existing regulations covering the use of FICO scores or health data to cover new types of use by new types of companies, so that consumers are not left in the dark. Requiring greater transparency when models are used in employment-related and other evaluations of people could also help.

Finance always has had a penchant for mathematical magic, and that’s where it all started for the author, who also blogs at mathbabe.org. In the world of pre-crisis mortgage finance, plenty of inputs to Wall Street’s clever models turned out to be literally fraudulent.

“The math could multiply the horseshit,” O’Neil writes, “but it could not decipher it.”

The notion of “garbage in, garbage out” is hardly a new one. Weapons of Math Destruction nevertheless illustrates how Big Data, sometimes a force for good, can be abused.

 This article was originally published on www.thestar.com.my and can be viewed in full

(0)(0)