How social enterprises and non-profits can demonstrate their impact.
Last year, an Asia-wide contest for social enterprises drew 1,080 entries from 31 countries. Participants were vying for a share of the total prize money of US$130,000, underscoring the stiff competition for funds.
The 2017 winners of the DBS-NUS Social Venture Challenge Asia, which seeks to support ventures that could best benefit society, include a producer of biodegradable bioplastics in Indonesia, a firm which aids Bhutanese women weavers and a company that uses the internet to help impoverished Indians apply for government and private financial aid.
Fundraising is vital to sustain and expand the work of social enterprises using innovation to tackle social problems and foster progress. Many such organisations, whether non-profit or commercial, rely on funding from wealthy individuals, charities and government.
To secure funding, however, social enterprises must increasingly prove that they are making a difference. To do so, they try to measure their impact. But their imprecise or broad definitions of “impact” may result in the collection of misleading data about what works. That, in turn, could lead to poor decisions and missed fundraising opportunities.
Haphazard evaluation in Uganda
Take for instance the experience of a non-profit organisation in Uganda in 2011. After eight years helping young people affected by conflicts, it was gaining public recognition and interest in it was rising. In view of that, the director of programmes felt that the organisation should show its donors that it was being run well and fulfilling its promise of transforming lives. This involved collecting data on how its programmes were carried out and having a positive impact on Ugandans.
Having mainly raised funds from small public donations and the sale of merchandise, the organisation had never assessed the efficacy of its programmes until then. And donors had not asked it to demonstrate accountability.
To address its lack of administrative or operational data, the organisation set up a monitoring and evaluation team and started tracking its activities. It also recruited an external company to carry out an evaluation of its work.
Halfway through the evaluation exercise, the director of programmes found that the contractor was using basic measurements to compare the lives of the participants before and after the programmes. Given that participants may systematically differ from non-participants, it quickly became clear that this before-and-after impact assessment would not provide the robust and credible insights the organisation was looking for. The correct comparison was the degree to which lives changed with the programmes – as opposed to without them – a difficult question that required careful attention to the design of impact assessment. In this case, the assessment ended up being a wasteful exercise with few valuable insights, something a non-profit can ill-afford.
Credible, actionable, responsible and transportable
The Goldilocks Challenge is a new book co-authored by Mary Kay Gugerty and Dean Karlan. It offers social enterprises a framework that helps them consider what kind of information, data systems and impact analysis strategy are just right for them – not too much and not too little, as in the Goldilocks parable.
Karlan, a professor of economics and finance at Northwestern University, visited INSEAD in April as a Distinguished Speaker. In an interview, he described the framework’s “CART” principles: credible, actionable, responsible and transportable.
Credible: This principle calls for data to accurately measure what they are supposed to, so that the analysis produces an accurate result. Such data must capture the essence of what social enterprises seek to measure, as well as reliably ensure that data collection methods are consistent and unbiased. While this may seem obvious, it can be challenging in practice. For instance, how “schooling” might be measured from one education organisation to the next might be entirely different. More importantly, the data analysis should be credible and yield accurate insights regarding impact.
Actionable: Firms should only collect data that they will use. They should also consider what specific actions they will take based on the findings, and whether they have the resources and commitment to take action. Under pressure to deliver data to donors, some organisations gather more than they need, wasting valuable resources and organisational time. Essentially, firms should collect data that will inform key decisions. They should also have clear plans contingent on what the data say.
Responsible: Social enterprises should ensure that the benefits of data collection outweigh the costs. Again as in the Goldilocks parable, there should be a balance between collecting too much information and too little. Firms should think about the efficacy of their data collection methods and whether the use of funds and respondents’ time are justified.
Transportable: The aim of this principle is to transport knowledge across time and create actionable knowledge that will help with the design of future programmes. Moreover, for data to be transportable, it should be placed in a generalisable theory, be made available to others and enable replication in different settings such as other countries or economic conditions.
Randomised trials uncover what works and what doesn’t
Karlan says having lots of data is useless without the right analytical tools to understand their meaning and implications. Indeed, gathering more data may not always be the right answer for social enterprises if there is no will to act on the findings or if the cost of measuring the impact is too high.
In recent decades, the collection and analysis of data have increased significantly, as it has become “radically cheaper” to do so, says Karlan. That has enabled many studies that were “simply not possible before” to become viable today.
In addition, the “big shift” in using randomised controlled trials has been instrumental in ascertaining the causal effects of social programmes, by examining how the lives of people change in their presence or absence of these programmes.
Such a study informed Teaching at the Right Level, a programme that was tested and carried out in African countries and India. Karlan and his colleagues set out to understand why, despite record high enrollment rates in Indian schools, 250 million primary school-aged children lacked basic reading, writing and numeracy skills. They found that reorganising classrooms by ability rather than age significantly improved test scores, helping children who were lagging behind.
“Sometimes it’s not a matter of finding out what works. Sometimes it’s a matter of finding out what doesn’t,” Karlan explains.
For example, different randomised trials across the world have found that microcredit schemes are failing to achieve their goal of helping the very poor increase their long-term earnings, according to Karlan. That was even though microcredit had become the main tool for governments, financial institutions and non-profit groups to lift people out of poverty.
In contrast, a social protection programme, which has been tested in seven countries, has managed to generate long-term income growth (between three and seven years) for the poor.
In the Philippines, the government wants to conduct a pilot trial of this programme and later expand it. Karlan will advise the government on how the programme can be adapted and run in a cost-effective manner.