Broken Feedback Loops and Evidence-Based Development

Fighting global poverty requires finding solutions to some of the world’s most complex problems. Solving any complex problem requires a clear and simple feedback loop. Learning from our mistakes over time is what leads us toward the solutions that work. The problem with development work is that this feedback loop is almost non-existent.

An important example of the broken feedback loop is apparent with popular microloan programs. Over 45% of employment in developing countries is generated by small businesses, yet over half of these business lack access to formal credit [1]. It has long been hypothesized that increasing access to credit in developing countries would support small businesses and lead to higher incomes for their employees. While this certainly seems like an effective intervention, we have to carefully evaluate its impact to understand if this program actually works.  

For an organization providing microloans in a developing country, analyzing the impact of this program may seem straightforward — if the businesses that they provide loans to grow over time, this program must be a successful intervention. But this is a broken feedback loop at work. This organization is failing to see that their analysis hasn’t accounted for the counterfactual. In order to understand whether their loans have had any impact, they need to create a working feedback loop that isolates the effect of their program.

In order to do this, the organization needs to understand both what is happening to the businesses that receive their loans, and what is happening to businesses who don’t receive loans at all. Measuring the success of the treated businesses establishes a trend but doesn’t tell us anything about the impact of their program. What if the businesses who received loans experienced significant growth over a two-year period, but so did many other businesses who don’t receive loans at all?

 This is the fundamental hole in our feedback loop. Had this organization only measured the success or failure of the business that were given loans, they would think that their program was a clear success. But if businesses that don’t receive any loans experience the same growth, how could the loans have had any impact at all? Perhaps the economy was growing, and these businesses would have experienced the same growth without loans. What if the organization (directly or indirectly) chosen to give loans to successful businesses and opted not to take a risk with less successful ones? Observing that the business who were given loans grew could simply be a reflection of the fact that they were already on track to be successful. On the other hand, if the businesses who received loans saw a meaningful rate of growth over this time period, but other comparable businesses experienced little to no growth, we can say with some level of confidence that these loans caused that growth.

Creating a counterfactual is what allows us to close the hole in our feedback loop and isolate the true impact of any intervention. When we start to truly understand the impact of our efforts, we can learn from our mistakes and build upon our successes. Fighting global poverty is one of the most difficult endeavors of our time. But like the many complicated challenges that we have overcome throughout history, this problem can be solved when we are guided by our own efforts. The lack of a natural feedback loop requires us to be more deliberate in our actions, and more precise in how we build upon them. Fixing our broken feedback loop is a step that will lead us toward the solutions that alleviate poverty and correct our actions along the way.

[1}    Small and Medium Enterprises (SMEs) Finance. World Bank Group. September, 2015. http://www.worldbank.org/en/topic/financialsector/brief/smes-finance