The use of big data by public institutions is increasingly shaping peoples’ lives. In the USA, algorithms influence the criminal justice system through risk assessment and predictive policing systems, drive energy allocation and change educational system through new teacher evaluation tools.
The belief is that the data knows best, that you can’t argue with the math, and that the algorithms ensure the work of public agencies is more efficient and effective. And, often, we simply have to maintain this trust because nobody can examine the algorithms.
But what happens when – not if – the data works against us? What is the consequence of the algorithms being “black boxed” and outside of public scrutiny? Behind this are two implications for ICT4D.
The Data Don’t Lie, Right?
Data scientist and Harvard PhD in Mathematics, Cathy O’Neill, says that clever marketing has tricked us to be intimidated by algorithms, to make us trust and fear algorithms simply because, in general, we trust and fear math.
O’Neill’s 2016 book, Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy, shows how when big data goes wrong teachers lose jobs, women don’t get promoted and global financial systems crash. Her key message: the era of blind faith in big data must end, and the black boxes must be opened.
Demand Algorithmic Accountability
It is very interesting, then, that New York City has a new law on the books to do just that and demand “algorithmic accountability” (presumably drawing on the Web Foundation’s report of the same name). According to MIT Technology Review, the city’s council passed America’s first bill to ban algorithmic discrimination in city government. The bill wants a task force to study how city agencies use algorithms and create a report on how to make algorithms more easily understandable to the public.
AI Now, a research institute at New York University focused on the social impact of AI, has offered a framework centered on what it calls Algorithmic Impact Assessments. Essentially, this calls for greater openness around algorithms, strengthening of agencies’ capacities to evaluate the systems they procure, and increased public opportunity to dispute the numbers and the math behind them.
We Need Data Storytellers
So, what does this mean for ICT4D? Two things, based on our commitment to being transparent and accountable for the data we collect. Firstly, organisations that mine big data need to become interpreters of their algorithms. Someone on the data science team needs to be able to explain the math to the public.
Back in 2014 the UN Secretary General proposed that “communities of ‘information intermediaries’ should be fostered to develop new tools that can translate raw data into information for a broader constituency of non-technical potential users and enable citizens and other data users to provide feedback.” You’ve noticed the increase in jobs for data scientists and data visualisation designers, right?
But it goes beyond that. With every report and outcome that draws on big data, there needs to be a “how we got here” explanation. Not just making the data understandable, but the story behind that data. Maybe the data visualiser does this, but maybe there’s a new role of data storyteller in the making.
The UN Global Pulse principle says we should “design, carry out, report and document our activities with adequate accuracy and openness.” At the same time, Forbes says data storytelling is an essential skill. There is clearly a connection here. Design and UI thinking will be needed to make sure the heavy lifting behind the data scenes can be easily explained, like you would to your grandmother. Is this an impossible ask? Well, the alternative is simply not an option anymore.
We Need Data Activists
Secondly, organisations that use someone else’s big data analysis – like many ICT4D orgs these days – need to take an activist approach. They need to ask questions about where the data comes from, what steps were taken to audit it for inherent bias, for an explanation of the “secret sauce” in the analysis. We need to demand algorithmic accountability” We are creators and arbiters of big data.
The issue extends beyond protecting user data and privacy, important as this is. It relates to transparency and comprehension. Now is the time, before it’s too late, to lay down the practices that ensure we all know how big data gets cooked up.
Image: CC by kris krüg
It’s terrific to see the need for greater openness and ‘algorithmic accountability’ finally being taken seriously, and the New York law is/will be a great precedent to build on. Cathy O’Neill, Global Pulse and others have done us all a great service by opening this up. Now we must ensure that in implementing improvements, i.e. strengthening agencies’ and citizens’ capacities to evaluate and dispute the data, that we don’t inadvertently recreate bias just further down the line.
Accurately interpreting and translating the ‘so what’ of the data in to policies, programs and investments requires deep knowledge and understanding of the context in which people live, and ideally the involvement of people and communities for whom decisions are being made. We know that is usually far from the reality (and sometimes not feasible) and that experts and government officials often interpret the data. Let’s pay real attention now to putting in place meaningful processes for citizens and marginalized populations who usually do not have a voice, to be involved in interpreting the ‘so what’ of the data, and beyond that, perhaps even more importantly, to be real partners in implementing the solutions. We also need to realize that many marginalized or discriminated populations are not even represented in the data in the first place, because they are not part of the ‘legal’ formal system. The importance of knowing who is ‘left out’ of the data cannot be understated.
And one last thing – PLEASE could we stop using the gender-biased cliche of ‘explaining things to our grandmothers’ – why not our grandfathers too? I know it is just an expression, but ironically it perpetuates the same gender biases that we are trying to get rid of here.
Thanks!
Thank you, Nancy, for the rich comments.
I completely note your point about grandmothers — sorry about that. In future I will use grandfather or grandparent.