Building From Our Own Foundations: Using existing data to design a better evaluation
A summary of the secondary data analysis of three community-led organizations.
The Full Spectrum Evidence and Learning Initiative (FSELI) is a partnership of organisations advocating for a foundational shift in our understanding of how international development actors engage with communities. Despite the growing interest in community-led models and examples of its effectiveness, the development sector still lacks sufficient evidence across multiple programs to explore how these approaches improve well-being. To address this gap, the FSELI aims to generate credible evidence around the Holistic Community-Led Development (HCLD) approach to help mainstream community-centered processes globally.
In the current phase, the FSELI is building a shared understanding of past work and designing a well-adapted evaluation of HCLD programming. As part of this journey, we recently completed a comprehensive Secondary Data Analysis (SDA) report. Here, we examine existing quantitative and qualitative data from three HCLD implementers: Tostan, One Village Partners (OVP), and Legado. We dive deep into the community-led activities captured by these three organisations’ teams to better understand how they have been measured and identify interesting patterns of heterogeneity. Understanding these patterns will help us build an evaluation design that will allow us to explore how, why, and when HCLD approaches work.
What We Did: The Secondary Data Analysis Process
The Institute of Development Studies (IDS) and IDinsight worked together to analyze existing Monitoring and Evaluation (M&E) data from the three implementing partners. Tostan, OVP, and Legado have admirably strong learning cultures. In fact, over 250 pieces of evidence collected since 2016 were submitted for consideration in this SDA review!
Our objective was to explore how key outcomes are currently measured. We wanted to see what past data could teach us about community dynamics, and to determine if the organisations’ current measurement tools are fit for answering the initiative’s broader research questions around HCLD.
What We Learned…
By analyzing this wealth of existing data, we identified several critical insights that will inform how we evaluate community-led development in the future:
We must harmonize how we define and measure key terms
We discovered a pressing need to align how we define and measure core concepts like collective action, resilience, well-being, and trust. Currently, different organizations approach these constructs differently. Selecting harmonized, context-appropriate measures will be one of the initiative’s most important next steps.
Governance views are nuanced, and community meetings may play a key role
We found evidence that people do not view governance as a single, simple idea. Perceptions varied by age and the precise question posed. Interestingly, individuals who frequently attend community meetings report more positive evaluations of leader performance and higher confidence in authorities. This link might indicate that these meetings play a crucial role in building confidence in governance, or may point to self-selection bias (e.g. people who already trust local authorities are more likely to attend meetings).
Social capital is not a unified concept, and requires comprehensive measurements
This SDA shows that HCLD programs often encounter communities with deep, near-universal bonding social capital (i.e. close ties with family and immediate neighbors), but noticeably lower linking social capital (relationships with people and institutions in authority). Furthermore, quantitative analysis reveals that within these relationships, concepts like perceived closeness, reliability, and willingness to help someone from a given group diverge significantly. This means people do not treat these ideas interchangeably; for example, a respondent might feel close to a group but not necessarily rely on them for help. Our evaluation design must consider multiple perspectives to help surface and address apparently conflicting evidence.
Communities’ definitions of well-being is dynamic and evolving
Community priorities change over time. Qualitative data showed that communities initially prioritize tangible infrastructure (like roads and buildings), but these definitions of well-being may evolve as a direct result of participating in the programs. Our future evaluation must be longitudinal to capture these evolving priorities.
Individual beliefs often change faster than community norms
When evaluating harmful behaviors, we noticed a gap between what individuals believe and what they think their community expects. People often personally report less harmful attitudes than the norms they perceive around them. This gap suggests that individual beliefs may evolve ahead of collective social expectations, or that they feel a need to distance their own beliefs and behaviors from those they attribute to their neighbors.
Views on harmful behaviors become more specific over time
When evaluating harmful behaviors, we noticed that people evolve from holding a consistent set of beliefs across different behaviors to developing more nuanced, specific views about each separate behavior. They might change their mind about one harmful practice without uniformly rejecting all others. Therefore, our evaluation design must measure different norms and harmful behaviors separately, and we must resist over-simplifying measures.
Qualitative data can illuminate causal linkages
Qualitative open-coding by IDS revealed valuable statements where participants described how they experienced change. This shows us that qualitative data can successfully illuminate the “why” and “how” behind program success, which will help us propose accurate causal linkages in the theories of action. To capture these often hidden mechanisms, our future evaluation methods must intentionally invite communities to share their own perceptions of how program activities contribute to the changes they experience.
… and What We’re Still Figuring Out
While the secondary data provided excellent insights, it also revealed where we will need to continue learning. Because this data was originally collected for internal monitoring, it often highlights that a change occurred but lacks the details necessary to explain the underlying mechanisms. Moving forward, the upcoming evaluation itself will be designed to answer the questions we cannot solve through past data alone:
What contextual conditions matter most?
We do not yet know which exact contextual conditions matter most for a program’s success. We have built a framework of hypotheses but our evaluation will need a dedicated effort to understand under what specific conditions the HCLD approach thrives.
How can alternative qualitative methods capture authentic community voices?
The qualitative methods used so far have been highly dialogic and tend to produce overwhelmingly positive data geared toward organizational reporting. To capture authentic community voices and less explicit forms of knowledge, we must design new, participatory approaches (like micronarratives) for our HCLD evaluation.
So… Now what?
This exercise highlighted the complex interplay of governance, social capital, and social norms within HCLD programming. It also laid bare the central challenge of evaluating community-led work: we must sensitively balance the need for comparable data across different contexts with the absolute necessity of preserving local meaning.
A strong evaluation requires a solid foundation. As the Full Spectrum Evidence and Learning Initiative (FSELI) moves to the next phase of this work, we will use these learnings as the building blocks to finalize our theories of change and construct a rigorous, fit-for-purpose evaluation design. Ultimately, laying this groundwork helps us prove that when communities are fully supported to lead their own development, sustainable and multidimensional change is possible.
Summary by Savannah Smith and the IDinsight team.
 | A guest post by
|