Reflections on Collections of Data

What we learned digging through each other's archives.

Over 250 unique pieces of data were submitted for our Secondary Data Analysis. In reviewing more than 10 years of data on Holistic, Community-Led Development impact, key insights were learned for the three organizations.

Here are some reflections from each of the organizations:

Katie Bernhard – Head of Impact. Legado

The process of gathering, cataloguing, and sharing our existing data and M&E materials was extremely valuable for us as an organization. We went through an internal learning process to document the evolution of our theories of change and theories of action over time, so that we could situate our current M&E framework within the overall evolution of our approach and program. This way, we were able to share a clear and cohesive set of secondary data that aligned with our current MEL framework, and also produced an internal guide to help illustrate how our programmatic and MEL thinking has evolved over time.

I was excited to see nuanced analysis of Legado’s data on social cohesion dimensions of trust, social networks, and social capital. I was also interested in the finding that behavioral indicators are often more informative than relying on attitudinal questions for Holistic Community-Led Development evaluation. This is a takeaway that we plan to incorporate in our future evaluations.

Seeing the analysis of our current M&E system indicators, conceptual definitions, and data side-by-side with other organizations that implement Holistic Community-Led Development programs gave us critical insight into our own M&E. For example, seeing that all three organizations collect data on social cohesion, but define and measure social cohesion differently, was a helpful perspective for us. This conceptual mapping and the clarity that emerged from that effort is critical for ensuring conceptual consistency – not only for Legado and our MEL, but for the broader field of HCLD.

Kyla Korvne – Head of Monitoring, Evaluation, Research, and Learning. Tostan.

The process of collecting and submitting our data was a valuable opportunity for us to take stock of our archival data. This was particularly timely as we were just wrapping up a major overhaul of our M&E systems. As we experiment with ways to quantify and measure social cohesion in our communities, we have taken note of the findings regarding the importance of anchoring such measures in behaviors as opposed to attitudes. We will be watching for emerging patterns in our early tests of new measures.

We were also very interested in the insight that when it comes to views of harmful behaviors, people may evolve towards more nuanced and specific views about each separate behavior. This reflects something we have observed in our evaluations over the years. We believe it is indicative of the complexity of social norm change and the importance of creating distinct spaces for dialogue and deliberation on different practices.

Tostan has always been proud of the consistency of our results over the years. One lesson learned for us, both from the process of collecting and submitting our data but also from the analysis, was the importance of this consistency. We have a huge amount of archival data, but in many different formats as our approach to data cleaning has changed over the years. This made synthesis for analysis across time a challenge. Even more importantly, our questionnaires and tools have also changed over time, even in some cases very slightly, but this makes comparing the results difficult. This drove home for us the importance of being patient with any process of overhauling or editing our survey tools - the need for adjustment and adaptation is normal, but instead of making incremental changes over time. We would be better served to take note of adjustments needed and make them all, together, periodically. This way, instead of having 10 slightly different tools across 15 years, we would have, for example, two significantly different tools and the results of each could be more easily compared over time.

Mohamed Rogers – Monitoring, Evaluation and Learning Manager. One Village Partners

The collecting and submitting of data was a valuable process for me and my team. It helped us understand how important data management is and how much value exists in data we routinely collect. The process helped us do an in-depth search of all our data, even before my presence with the organization.

What really stood out was that the results were consistent overtime. The data that OVP collected over a period of time showed things kept getting better in important areas, which shows that our Community-led way of programming is strong. It became clear that our work is not just about short-term results, but also about building long-term community capacity and resilience.

Another important point of view was how our model can be used in many different situations and how it can be changed to fit those situations. Seeing our data next to that of other FSELI members made it clear our process of documentation needs to be strong. This was a key lesson learned, as it improves transparency and makes it easier for others to learn from our data.