How the Rockefeller Foundation is approaching evaluation with developing country partners.
Across every sector of society, decision makers are struggling with the complexity and velocity of change in an increasingly interdependent world. The context of decision-making has evolved, and in many cases has been altered in revolutionary ways. In the decade ahead our lives will be more intensely shaped by transformative forces, including economic, environmental, geopolitical, societal, and technological seismic shifts.
As part of its response to this global dynamism, the Rockefeller Foundation has translated its 1913 mission of promoting the well-being of humanity into two overarching goals: expanding opportunity through more equitable growth, and strengthening resilience to acute crises and chronic stresses, whether man-made or ecological. Our vision is a world in which globalization's benefits are more widely shared and the inevitable challenges that accompany a world that is fast changing, diverse, and complex are more easily weathered.
The value of evaluation must ultimately be judged by its usefulness in helping to improve outcomes for target beneficiaries.
The Rockefeller Foundation structures its work around time-bound cross-sectoral initiatives that seek innovative solutions and support enabling environments to bring about change. The foundation's structure reflects its view that today's problems and solutions are multi-dimensional in scope and nature, and that they require multi-disciplinary responses at the intersection of fields.
Just as the Rockefeller Foundation's approach to philanthropy has evolved, so too has its approach to evaluation. With its mission to improve the well-being of humankind, its focus on impact, and much of its grantmaking in developing countries, the Rockefeller Foundation is committed to evaluation practices that are rigorous, innovative, inclusive of stakeholders' voices, and appropriate to the contexts in which the foundation works. This article discusses how the Rockefeller Foundation integrates the views of developing-region evaluators into its evaluation approaches, and highlights five key strategies:
Engaging stakeholders to develop shared outcomes.
Expanding capacity through use of non-staff monitoring and evaluation specialists to partner with grantees.
Sharing knowledge through learning forums and communities of practice.
Strengthening developing country evaluation practice and ownership of results.
Developing innovative methods and approaches to evaluation and learning.
Rethinking, Reshaping, and Reforming Evaluation
In November 2011, the Rockefeller Foundation brought together leaders from philanthropy and development at the Future of Philanthropy and Development forum, held at its conference center in Bellagio, Italy. In one of the keynote papers, Evaluating Development Philanthropy in a Changing World, Robert Picciotto, former vice president at the World Bank and now professor at King's College London, squarely tackled the role of evaluation in philanthropy and development. "The changing context and thinking on development has profound implications for development evaluation itself, and for the contribution evaluation can bring to the empowerment of people; and the effectiveness of development interventions by national governments and international partners and, increasingly, by non-state actors?foundations, philanthropists, and agencies that promote investing for impact."
Picciotto continued, "Pressing human needs are not being met by an official aid system that is short of resources, catering to multiple interests, and hobbled by massive coordination problems. By contrast, private giving for development is growing and has proven nimbler and more results-oriented than official aid. However, the philanthropic enterprise will not fulfill its potential unless it identifies and taps into its distinctive comparative advantage and coordinates its interventions with other development actors; embeds evaluation in its processes to achieve operational relevance, effectiveness, and efficiency; and demonstrates that it is accountable and responsive to its diverse stakeholders."
Developing country evaluation leaders have also articulated the need for a new approach to evaluation and the role it plays in improving the wellbeing of humankind?in particular, the lives of the poor and vulnerable in developing countries. At the January 2012 gathering of the Africa Evaluation Association's biannual conference in Accra, Ghana, African evaluation leaders and policy makers highlighted five steps foundations and development agencies must take if they aspire to play a meaningful role in social transformation.
Broaden the inclusion of key stakeholders in evaluation. Only when the voices of those whose lives we seek to improve are heard, respected, and internalized, will we be able to effectively evaluate what success should look like for the people we are most concerned about. Foundations and agencies need to take practical steps to include key stakeholders in the design of, conduct of, and learning from evaluation.
Regard evaluative knowledge as a public good and share it widely. Learning with our partners and stakeholders about what works and what doesn't work should be seen as a global public good not limited to boards and program teams, but shared widely with grantees, partners, and peers.
Address evaluation asymmetries between developed and developing regions. The majority of human and financial resources for evaluation emanate from agencies and foundations based in the developed world. With evaluators from developing countries playing a minor role, if any, many of them do not get sufficient experience to move into leadership roles. Mentoring, coaching, and training can strengthen the role, capacity, and resources of developing-country evaluators so that they can play key roles in conducting and using evaluation results for social transformation and accountability in their own countries.
Broaden the objects of evaluation to learn more beyond the individual grant or project to a more strategic assessment of portfolios of investments, policy change, new financing mechanisms, and sector-wide approaches that tell us more about what works and what does not in different contexts. Framing evaluation to take into account the drivers of unsustainability and causes of the challenge being addressed provides greater learning than narrower evaluations that focus only on the funder's specific intervention.
Invest in the development and application of innovative new methods and tools for evaluation and monitoring that reflect multidisciplinary and systems approaches to problems and complexity; invest in methods that assess network effectiveness and policy change; and use and adapt new technology to enable stakeholders to provide close to real-time data and feedback.
The Rockefeller Foundation's Approach
With its long history of supporting developing country institutions, the Rockefeller Foundation has responded to the call to action from developing-region evaluators by adopting the following approaches to planning, monitoring, and evaluating its work.
Shared Outcomes | An important underpinning of the Rockefeller Foundation's initiative-based approach is a fundamental recognition that the world's greatest challenges can't be solved alone. These challenges involve a complex mix of actors that are often globally interdependent across sectors and geographies. Networks, alliances, and coalitions of diverse stakeholders from governments, foundations, civil society, and business are increasingly seen by the foundation as a more powerful way to mobilize the vast range of resources and actions required to bring about sustained and transformational change on a significant scale.
Increasingly, the Rockefeller Foundation brings together grantees and partners from developed and developing countries to establish a common vision of the problem, outcomes, and indicators for success. Grantee agreements now include reference to the common vision of results and shared outcomes to which the grantee contributes, and foundation teams are expected to manage portfolios of grants and relationships with grantees towards that common vision. This shared-outcomes approach forms the basis for ongoing monitoring, evaluation, and reporting, and for learning dialogues with grantees and partners. (See "Shared Results Framework.")
Monitoring and Evaluation | Most foundations have capacity limitations on the amount of time that can be devoted to monitoring and learning with grantees and partners, visiting field projects, and working collaboratively?activities that we know contribute to greater collaborative learning and effective relationships. Recognizing these limitations, the foundation awards grants to monitoring and evaluation (M&E) groups and specialists in developing and developed countries who act as monitoring partners, or what we call "critical friends," 2 throughout the life of initiatives (typically, a five- to six-year period). They work with grantees to identify key learning questions, help to set up monitoring systems, and provide support in analyzing monitoring data. The most significant feature of the critical friends is that they build trust with grantees and partners to ask tough evaluative questions, and they support grantees in seeking and using feedback to make improvements throughout the life of the initiative. Periodic evaluations are conducted by independent teams to provide an objective assessment of progress toward outcomes and impact.
For example, the India-based nonprofit Participatory Research in Asia, in collaboration with the Ghana-based Institute for Policy Alternatives, works alongside Shack/Slum Dwellers International (SDI) which directly represents millions of urban poor slum dwellers in 33 countries. The aim of this critical friend partnership is to strengthen the participatory learning, monitoring, and evaluation systems and abilities of the urban poor networks to better capture and systematize learning and strengthen accountability with the goal of empowering the urban poor to achieve wider positive impacts. The critical friend role underpins a belief that federations of the urban poor are capable of changing their own situation for the better. As a result of this partnership, SDI has strengthened its ability to democratize learning, monitoring, and evaluation?continuing to place the tools, responsibility, and ability for change in the hands of its members.
This figure illustrates the framework around which Rockefeller Foundation staff, grantees, and partners develop a common vision of the results and impact that they seek to achieve collectively. The top frame represents the mission and strategy of the foundation?promoting the well-being of humanity in two overarching goals: expanding opportunity through more equitable growth, and strengthening resilience. The middle frame represents the medium-term outcomes that the foundation seeks to achieve during the life of the initiative (these change from initiative to initiative). The lower frame represents the work that grantees, partners, and staff do in their individual organizations to collectively bring about outcomes and ultimately improve the lives of beneficiaries. These shared results frameworks anchor the ongoing dialogue with grantees about progress toward achieving this vision and their contribution to the shared outcomes. It also serves as a framework for managing portfolios of grants and monitoring changes during the life of the initiative.
Learning Forums and Communities of Practice | Most of the Rockefeller Foundation's initiative teams convene grantees and partners annually to review progress, highlight lessons and challenges, celebrate successes, and identify improvements needed. Through these forums grantees learn from others in the field, meet new resource people, and adjust their strategies going forward. Although this practice does not guarantee impact, it increases the likelihood of it by creating a greater sense of ownership and shared outcomes, and it increases leverage by connecting grantees with new resource people, funders, and mentors. Increasingly, M&E grantees produce high-quality knowledge products as a public evaluation good to highlight what works, what does not, for whom, and under what conditions. Our aim is to establish with grantees a body of collaborative knowledge, shared lessons, and a culture that values evaluation as a resource for learning as well as for accountability.
For example, the Rockefeller Foundation has aligned with the South East Asia Community of Practice in Evaluating Climate Change Resilience (SEA Change), facilitated by the nonprofit organization Pact, to work on urban climate change resilience in ten Asian cities. This community of practice brings together evaluators, program managers, grantees, and policy makers concerned with learning what works in interventions aimed at adapting and building resilience to the effects of climate change and extreme weather events in Southeast Asia. Resources and lessons are shared through online learning, onsite convening of SEA Change participants, and coaching, mentoring, and training provided by members of the community of practice throughout the countries of Southeast Asia.
Addressing Asymmetries | The Rockefeller Foundation is supporting the formation and strengthening of regional developing-country networks and the first-ever regional institutions to train, coach, and mentor evaluators, and to partner with evaluators from other regions. Through these platforms and networks, the foundation aims to help rebalance the asymmetries of choice and opportunity for developing-country evaluators to control the evaluation process in their own localities and to improve the quality of evaluation by partnering with evaluation leaders globally.
One example of this is the African Evaluation Association (AfrEA), a pan-Africa umbrella organization comprising more than 25 national M&E associations in Africa, and a resource for individuals in countries where national evaluation bodies do not exist. AfrEA, which has more than 1,000 evaluators from all regions of Africa, receives Rockefeller Foundation funding to enable the formalization of its organizational, operational, and management structure, and to build communities of practice among its membership to tackle the most pressing evaluation challenges on the continent.
The Centers for Learning on Evaluation and Results, located in Africa and Asia, are another example of an effort aimed at addressing asymmetries in evaluation in developing countries. Together with a consortium of funders committed to building developing country capacity for taking charge of the evaluation agenda in their regions, the foundation is supporting regional centers3 in East and West Africa and South Asia to strengthen their skills, networks, and experience in monitoring and evaluation and results-based management capacity of public, private, and civil society development in the global south.
New Methods and Approaches | Traditional evaluation methods and approaches to learning, accountability, and feedback have not kept pace with the advances in technology and social media. The majority of evaluation practice is still largely paper-based despite great strides in technology, interactive web-based platforms, and multimedia tools that make real-time feedback from grantees and beneficiaries possible and accessible. The Rockefeller Foundation and its partners learned a great deal from Ushahidi, an open-source crowdsourcing project that allows users to send crisis information via mobile devices to map reports of violence or suffering. Inspired by the potential of these kinds of tools to democratize evaluation information, increase transparency, and lower the barriers for individuals to share information and stories, the foundation is supporting a number of innovative approaches to evaluation.
We are privileged to work in an expanding field in which our evaluation findings can change lives for the better.
One example is GlobalGiving's Story Telling project, an innovative way to gather local feedback from people in developing countries and to share it with communities, implementing organizations, and donors to create real-time feedback. With support from the Rockefeller Foundation, GlobalGiving successfully deployed a network of people in Kenya, Uganda, and Tanzania that has generated more than 20,000 tagged narratives from thousands of people. Some of GlobalGiving's partners are deriving actionable intelligence from these stories, and GlobalGiving is discovering patterns in the stories that inform its own operational and strategic decision-making processes.
Another example is BetterEvaluation, an online interactive community of evaluation practice developed by the Royal Melbourne Institute of Technology in partnership with the Institutional Learning and Change Initiative and the Overseas Development Institute, with support from the Rockefeller Foundation and Pact. BetterEvaluation provides advice, online support, and good practice examples to evaluators in developing and developed countries.
Reshaping Development Evaluation
Philanthropists and development practitioners have a golden opportunity to join together with grantees and partners in developing countries to reshape evaluation to better respond to global change and to serve our missions and goals more effectively. To do this, we must be prepared to rethink and reshape our evaluation practice in at least four ways. We must:
Embrace a broader set of voices in framing our approaches to evaluation.
View collaboration and partnerships between developed and developing areas as mutually beneficial toward a common goal of expanding and sharing evaluation knowledge as a public good aimed at achieving better development outcomes.
Recognize the need to address issues of accountability, transparency, ethics, culture, and independence.
Address asymmetries in individual and institutional capacities for undertaking, driving, and owning evaluation in developing regions by promoting opportunities for professional excellence, networks, and sustained global partnerships in the discipline of development evaluation.
The value of evaluation must ultimately be judged by its usefulness in helping to improve outcomes for target beneficiaries. The quest for impact is currently in the spotlight among foundations and development agencies as we seek collectively to maximize the positive benefits of our resources. We are privileged to work in an expanding field in which our evaluation findings can change lives for the better. Together with our peers, partners, and grantees, we can and should rethink, reshape, and reform the practice of evaluation to better meet that challenge.
See the complete evaluation supplement: http://evaluation.ssireview.org/
Notes
1 For more discussion of this topic, see Global Risks 2012, Seventh Edition, World Economic Forum, 2012.
2 The term "critical friend" refers to a partner that builds trust and engenders a reflective evaluative culture that is able to provide both negative and positive feedback to the grantee in a supportive way throughout the life of the work.
3 In South Asia The Jameel Poverty Action Lab South Asia at the Institute for Financial Management and Research, India, with the Center for Economic Research in Pakistan; In Africa the University of Witwatersrand with the Kenya Institute of Administration and Ghana Institute of Management and Public Administration.
Judith Rodin is president of the Rockefeller Foundation. Before joining the foundation, she was president of the University of Pennsylvania, and preceding that, provost of Yale University. Rodin is on the board of Citigroup, Comcast, and AMR.
Nancy MacPherson is managing director, evaluation, at the Rockefeller Foundation. Before joining the foundation she spent 25 years in development evaluation for international nonprofit organizations in Asia and Africa and for the United Nations.
.png)