Measuring the immeasurable?

How to measure something which seems so intangible, like ‘advocacy’ and ‘improvements in the enabling environment’? This is the challenge we face with planning, monitoring and evaluation (PME) for the Voice for Change Partnership (V4CP) programme. The questions we must ask ourselves are: How do we know that we have made progress, how do we know if our strategy works or how do we monitor subtle changes? It’s not an easy task to make this more concrete and figure out how it can be measured. Besides, all civil society organisations (CSOs) involved and the SNV teams have to work with these methods. So keeping things user-friendly while also offering reliable monitoring tools are both important criteria.

So what exactly is our approach for PME? The following description might sound a bit technical but don’t be afraid, it’s short. Like with most Strategic Partnerships, the advocacy nature of the V4CP programme calls for qualitative measurement methods, besides quantitative methods. A qualitative method often used in our programme is a scoring rubric (also called ‘ladder’ or ‘QIS’). A scoring rubric consists of different scenarios which describe several situations and define varying degrees of success. For example, if we look at collaboration with government officials, then having no meetings at all represents the lowest degree of success. But if a government is following up and taking into account the interests of the CSOs the degree of success is considered very high. With of course multiple other scenarios in between these two ends of the rubric. The real test for working with such a method occurs when the CSOs apply it to an actual situation. Their first opportunity to do this was the baseline data collection, which started late 2016 and is currently being finalised.

How was this baseline conducted in practice? Let’s take the example of the result ‘Improved collaboration between CSOs, government and private sector’ and apply it to Rwanda. In Rwanda, 5 CSOs are jointly working on nutrition and then also worked on the baseline together. They discussed to what extent they currently find themselves cooperating with the government on this specific topic. Are they regularly participating in meetings with government officials on nutrition? Have they been asked to share their opinion on nutrition or related topics, such as food security or health? Or perhaps plans on nutrition have been made or are being developed by the government that take into account the interest of the CSOs? All these questions represent certain scenarios, and respectively a higher degree of collaboration between CSOs and government on the issue of nutrition.

At first, not all CSOs agreed on the scenario for their current situation because some considered themselves to be on a higher level of engagement. But a plenary discussion in which they exchanged opinions and information, including examples from other CSOs, eventually lead to a lower and more realistic agreed upon scenario for all 5 CSOs. That final scenario was inserted into the baseline, as well as the information that was exchanged to serve as ‘evidence’ for the chosen scenario.

So, measuring results such as collaboration is in fact possible and that is of course good news. But what we also learned from the baseline exercise is that the supporting ‘evidence’ for a certain scenario has to be concrete and relevant to the issue the CSOs want to promote, in this case nutrition. This scenario analysis has to be executed thoroughly so that next year we can build on it to show progress (or not). Another lesson learned is that achieving the ‘ideal’ situation or the highest scenario is not realistic in every country and for every topic. Some targets were over-ambitious, so we have to be careful about what can be achieved in the lifespan of the programme.

All in all using the qualitative measurement method, which is often new for our partners, is quite exciting and stimulates discussions among the CSOs and SNV. A good learning exercise!

This blog was written by Kim Groen - Project Officer of the Voice for Change Partnership programme.