Studying water disconnections: can you do it ethically?

Research Aug 10, 2020

Randomised Control Trials, or RCTs, are usually known as the "golden standard" of economics and social research: you compare a "treatment" group (benefiting from your intervention) to the "control" group (not benefiting), with enough randomness so that your study doesn't influence too much the results. It's long and expensive, but do it well enough and you win the Nobel prize in economics – or at least you find some ways to reduce poverty in your study area.

When your intervention is about providing something better (like a cash transfer), the ethics of choosing who benefits or not are complex. So what would you make of a study that disconnects water from residents of informal settlements? Well, we have one, and it's not an easy issue!

Enforcing Payment for Water and Sanitation Services in Nairobi's Slums

I strongly suggest that you read it all:

  1. The original study: Enforcing Payment for Water and Sanitation Services in Nairobi's Slums. It's not open access, but it's available on Sci-Hub for $5.
  2. It generated quite a bit of backlash on Twitter, and the authors replied with their comments on ethical issues.
  3. Finally, there's a working document started by Chris Prottas commenting on both the original paper and the reply. All excellent reading.

Let me try to summarise (there are also some Twitter threads embedded below for useful summaries). Nairobi City Water and Sewerage Company, is losing money, partly because some people don't pay their bills, and has to ration water. Like many water companies, they are afraid that threatening to cut off water from residents will have political repercussions.

But is it true? That's the point of this study: Nairobi Water asked advice from the World Bank. Like any bank, its advice boiled down to: get your money back, or repossess cut the service. As Nairobi Water was preparing to cut off water to non-payers, the study authors used that opportunity to test whether nudges would be effective before disconnections (in this case, explaining to tenants how to talk to their landlords 🙄), and whether disconnections were useful to resume payments. 97 connections were cut during this study, and most had it back 9 months after.

Map of Kayole Soweto in Nairobi showing areas served disconnection notices (red) or not (blue / black). Source: Original study

Let's consider this: at least 97 families (possibly more, as connections are often shared), cut off from the most basic service because their landlord didn't pay the bills, during up to nine months.

The results are shockingly surprising: talking to tenants has zero effect (as anyone who knows the peculiar landlord-tenant relationships in Nairobi can attest), but on the other side, cutting water off makes people pay again. Shocking, right? What is interesting is the side effect of making people pay:

We do not find evidence of negative impacts of the increased enforcement on landlord and tenant perceptions of fairness and quality of service delivery, on community activism, on the relationships of tenants with their landlords, or on child health. To counterbalance the effective increase in utility fees paid, landlords increased their rental income predominantly by renting out additional space in their compounds and by marginally increasing tenant rents. These results suggest that strict enforcement through disconnections increases payment and the financial position of the utility without incurring political costs.

There are many ethical issues at play here, neatly summarised in the commentary paper. But let's start from the premise: if you take the point of view of Nairobi Water, this study can make sense: you don't want to cut off water, as you usually want to try any other method to get payments, and you are rightly worried that you become the "bad guy" when you disconnect residents. So it can be useful to know what and what doesn't.

But the utility's view can be narrow. Nairobi has unique conditions: a complex rental housing market with many absentee landlords, and "water cartels" often displacing official efforts. There are strong power dynamics at play, and if you take the vantage point of the residents instead, you can turn the research on its head:

  • "Not incurring political costs" becomes "People don't protest";
  • "We find a precisely estimated null effect of the engagement intervention" becomes "Tenants don't have much power over their landlords"
  • "Landlords increased their rental income predominantly by renting out additional space in their compounds" becomes "Landlords further increased the unviable density of the informal settlement to extract profit".

The ethics response by the authors offers useful context, but hides too easily behind arguments such as:

[...] these interventions were designed and implemented by a Government. And those interventions are standard policies that many governments use over the world if you do pay for the electricity and water that you use.
[...] These interventions were not the invention of the minds of academics deigned to test a theory, but rather to help a government sort out a real-world problem that many developing country governments face.

This ignores the fact that researchers are never neutral. Like in quantum physics, you can't observe a system without affecting it: if you publish a paper saying that disconnections work – based on a mere 97 disconnections in a specific settlement of a specific city – then other governments and utilities will jump on this result and use it to justify their policies. And this is the other inequality at play: a few researchers from rich institutions in the Global North have the liberty to study what happens when you deprive impoverished residents of their water, without their consent.

What's next? The authors have been willing to discuss this issue, which is good, and will revise their paper to include more context. Bu there are many issues raised in the commentary, and I doubt that a revision will address them all. Beyond this study, this raises the question of such RCTs: do we need such big, expensive studies, sometimes with small study groups (97 disconnections!!), conducted by foreign organisations? Can you really create "evidence-based policy" from this, or merely justify your policy with scant evidence?

This is also a story of a water utility trying to expand too quickly, or at least to expand the number of connections – a good-looking metric – while neglecting the socio-economic side of such expansion. Sanitation is coming along too in Nairobi, are we going to see the same issue?

If you want to read more, below are some useful Twitter threads and summaries I recommend:

A new research experiment in Kenya raises questions about ethics
A randomized controlled trial in Kenya sparks concerns about whether marginalized communities lost access to water for the sake of an experiment. Now more people are asking how these trials are conducted and who is involved.


Cover image credits: Jouni Rajala on Unsplash
Spotted by: Ben Tidwell and Josh Budlender