Understanding Principles Focused Evaluation

Greetings and welcome. I'm Michael Quinn Patton, author of Principles Focused Evaluation, here to take you through what it is, how to do it, why to do it, with some examples. This video is about an hour long and based upon my Principles Focused Evaluation workshop. It's in three parts. The first part will cover the three overarching Principles Focused Evaluation questions, Definition of Effectiveness Principles, Truth and Reconciliation Principles Example, Principles-Driven Transformative Social Movements, and the Purpose and Niche of Principles-Focused Evaluation. Part two, we'll look at Principles versus Rules, an Extended Vibrant Communities Example, and the Guide Framework and Tool. And then in part three, we'll look at four major examples. and some minor examples. The major examples of principles focused evaluation are effective principles to help youth overcome homelessness, two-eyed seeing principle for transcultural collaboration, principles for international development aid, and the principles of the global alliance for the future of food. That's what we're going to cover in this video. Now principles are all around us. We grew up with principles telling us how we should behave, Teams have principles. Organizations have principles. Here are some classics to remind you what we're talking about. Be mission-driven. Do no harm. Nothing about us without us. A principle for traditionally underrepresented and unheard voices, indigenous people, people with disabilities who are now wanting to participate in research and evaluation that focuses on them, but with the principle of nothing about us without us. Stay focused on what's important. Build trust. Nurture relationships. Take care of yourself. Do unto others as you would have them do unto you. A golden rule, but also a principle. And speak truth to power. These are just examples of principles that surround us. Principles can be directed at individuals, Families can have principles. The family that plays together stays together. Teams have principles. Organizations express principles. Programs can be based on principles. Initiatives can be directed by principles. Indeed, actions of all kinds by anyone or any group anywhere at all times can be based on principles. The key here is can. Principles can guide, but the principles-focused evaluation question is do they guide? There are three primary principles focused evaluation questions to be asked about principles. To what extent are the principles meaningful to those who are expected to follow them? If meaningful, to what extent are the principles actually adhered to? And third, if the principles are adhered to, what results from following the principles? What difference does adhering to principles make? Principles-focused evaluation, then, is a way for principles-based programs with principles-driven leaders and staff to find out the extent to which their principles are meaningful, are adhered to, and lead them where they want to go. This video, then, will cover how to craft meaningful and evaluatable principles, distinguish principles from rules, values, and beliefs, We'll introduce the guide tool for conceptualizing and evaluating principles. We'll give you examples of principles and examples of principles-focused evaluation. So, let's get at it. On the cover of the book is featured a sculpture of an Inuksuk. Inuksuks are large stone sculptures in the wilderness of the Arctic by the Inuit people that provide... ways of navigating through the wilderness where there are no roads, where there are no signs. The Anaksaks become guidance. They help locate directions. They often are places where sacred objects and food are placed to help people through the journey in the wilderness. Well, that's what principles do. The word Anaksak means to act in the capacity of a human and that become the symbol of principles to give us the capacity to find our way through complex dynamic systems. And nunchucks represent the guidance that principles offer to find our way through difficult, tumultuous, uncertain terrain. Bill Easterly, the author of Tyranny of Experts, opens that book with the observation, it is critical to get the principles of action right before acting. And that's why principles can be so important in program and project design, as well as in tracking their implementation and evaluating. So, what is a principle? For our purposes, an effectiveness principle provides guidance about how to think or behave towards some desired result based on norms, values, beliefs, experience, and knowledge. The principle statement is a hypothesis about effectiveness until evaluated within some context to determine its relative meaningfulness, feasibility, and utility for those committed to and attempting to follow it. We are going to be looking at effectiveness principles. There are two aspects to this work. One is crafting principles. Just as evaluators historically got involved in helping clarify goals to make sure they were clear, specific, and measurable, so too we get involved in helping to express principles in ways that they can be evaluated in order to then evaluate their meaningfulness, adherence, and results. Nelson Mandela, certainly a highly principled man, said, you mustn't compromise your principles, but you mustn't humiliate the opposition. No one is more dangerous than one who is humiliated. Nelson Mandela and Bishop Tutu created the truth and reconciliation process in South Africa following the end of apartheid, and that process epitomizes being principles-driven by principles-driven leaders. The Mandela-Tutu principles of truth and reconciliation involved five principled steps. Bring together the oppressor and the oppressed, speak truth to each other. confess, forgive, and reconcile. Nelson Mandela was often asked how he could forgive the some 20 years of torture done to him, the effects on his family, upon his community, and he replied that you don't begin with forgiveness. There's a process that leads to forgiveness based upon principles of coming together with the oppressor, speaking truth to each other. confession, and then forgiveness and reconciliation. Each of these steps in the truth and reconciliation process, each of these principles can and has been evaluated. Bringing together the oppressor and the oppressed leads to principles focused in questions like who participated in the truth and reconciliation process, who represented oppressors, who represented victims of oppression, who from both groups did not participate and why. What was the scope of participation in the process compared to the numbers of people who were engaged in oppression and those who were victims of that oppression? These are just samples of examples of evaluating principles. I'm not going to read you the questions for each of the principles. You can pause the video if you want to read them for yourself. The point is that there are evaluation questions that guide the evaluation of the extent to which principles are meaningful, adhered to, and what results from following the principles. I did a book with two Canadian colleagues, Frances Wesley and Brenda Zimmerman, entitled Getting to Maybe, How the World Has Changed. We studied transformative social movements and found that they were typically problem-focused and principles-driven. They didn't begin with strategic plans or clear, specific, measurable goals and logic models and theories of change. They began with a problem that they wanted to take on, that they were driven by to change, and they had principles that guided how they engaged in that work. Mothers Against Drunk Driving, founded by Candice Leitner after her 13-year-old daughter was killed by a drunk driver, and she and other mothers began to come together. not with a grand plan about reducing the alcohol level for determining drunk driving or changing laws or increasing police spending, but to make sure that the punishment fit the crime and the drunk drivers were identified and didn't just get suspended sentences and put back on road again. and it created a lasting social movement that's now had a major impact around the world. Bangladeshi economist Mohamed Yunus began the microfinance movement through the Grameen Bank based upon the principle that low-income, especially women, ought to have access to capital. Principles guide ongoing engagement and adaptation as projects and programs unfold. adapting to new contexts, scaling principles from one place to another, guide systems change and transformation initiatives, and guide how to navigate complex dynamic systems. The niche of principles-focused evaluation then is it's a unit of analysis. It's the evaluation focus. The focus isn't just on attaining goals or upon testing a logic model. It's on the principles themselves. A major message of principles-focused evaluation is that principles can be evaluated and should be evaluated. It's an approach to innovative programming. When new initiatives start out, before they have fully strategized plans and implementation means, they have principles. It's a way, as I said, to navigate complex dynamic systems. And... Principles can be a way of doing business. You're going to see examples of that as we go along. And then, of course, it's an approach to evaluation. So the work of crafting principles we're going to take on, the work of evaluating principles we're going to take on. There are examples of principles that have had a major influence in the management world, in the business world, in the private sector world. Jim Collins and his team, They're important books like Good to Great and Great by Choice, where they study high-performing corporations, generate principles to guide corporations in being more effective. Ray Dalio, a major finance investor, has written a best-selling book on principles, both about principles for business and investing. but principles about living. These are examples of the importance of attention to principles in our current world. There are all kinds of books about principles, non-profit management principles, and alternative non-profit management reinvented principles, principles of trauma therapy, principles of public health practice, principles of emergency management. The six principles for exemplary teaching of English learners, servant leadership for higher education principles, and universal principles of design. Just a few examples of the many approaches to principles, all of which could be evaluated. And that's what principles-focused evaluation does. To help illustrate what principles are, I want to compare them to rules because it clarifies the particular distinction we're making about principles. Compare rules of engagement to principles of engagement. So for example, coming to a stop sign and stopping is a rule. You come to a complete and full stop. Rules are clear. They're evaluated by you either follow them or you don't follow them. But a principle would be represented by defensive driving. It tells you to pay attention to other drivers, to watch out for changes in weather, to be alert to what's going on, to not drive distracted, to not drive with a mobile phone going and talking on it, but to drive defensively. That doesn't give you the same specific guidance that a rule like a stop sign gives you, but it gives you direction and you have to interpret the principle for the context and conditions. where it applies recipes give very precise directions add a quarter teaspoonful of salt a cooking principle in contrast is to season to taste and situation who's coming what's the nature of the food what spices might you want to include and you adapt the meal to those conditions there are exercise rules thirty minutes of aerobic exercise every day in contrast A principle would be to exercise regularly at a level that supports health and is sustainable given your health. lifestyle, age, and capacity. Principles have to be interpreted and applied contextually. In evaluation, a rule is to deliver the report on time, the date specified in the contract. In contrast, a utilization-focused evaluation principle is to target delivery of findings to be timely and useful for informing decisions and learning opportunities, monitor situation developments that may affect timeliness, so that You don't just look at the contract deadline, but you look at when would findings be useful and adjust to that. That's a utilization-focused evaluation principle. One of my favorite examples is a transition from being rule-based to principles-based. In 1999, the American Academy of Pediatrics formulated guidance for parents in the form of a rule, no screens under age two. Screens were TVs and computers that were thought to be possibly harmful to a child's brain development, and so the rule was to prohibit such screens. But subsequently, after 20 years of research and study, they moved from the rule no screens under 2 to a principle, no screens under age 2 without interaction. Interaction is a guide, not a rule. It has to be adapted to the age of the child, six months, two years old, what's on the screen, who else is present. But the point is not to use screens as a babysitter, but to engage and interact with a child with whatever is on the screen. And therefore, it moves from a rule to a principle. Another example of moving from a rule to a principle comes from statisticians. In 2019, the American Statistician Journal published a special issue entitled Moving to a World Beyond P less than.05. And in that special issue, they posited the rule, don't say, quote, statistically significant, close quote. They wrote, we conclude, based on our review of the articles in this issue and the broader literature, that it is time to stop using the term statistically significant entirely. regardless of whether it was ever useful a declaration of statistical significance has to-day become meaningless yet they went on solid principles for the use of statistics do exist and they are well explained in this special issue we summarize our recommendations in two sentences totaling seven words accept uncertainty be thoughtful open and modest remember adam these are principles Statistical analysis principles. Accept uncertainty, be thoughtful, be open, be modest. The point of positing these principles is to move from a mechanistic pattern of simply accepting whatever appeared to be statistically significant to a thoughtful pattern of thinking about results, accepting uncertainty, being thoughtful is key, open, and modest. statistical analysis principles the article goes on to conclude with regard to moving from statistical significance to thoughtfulness that momentum seems to finally be on the side of reform the initial slow speed of progress should not be discouraging that is how all broad-based social movements move forward and we should be playing the long game but the ball is rolling downhill the current generation is inspired and impatient to carry this forward so let's do it let's move beyond statistically significant even if upheaval and disruption are inevitable for the time being it's worth it in a world beyond p less than point o five by breaking free from the bonds of statistical significance statistics in science and policy will be more significant than ever statisticians becoming principles focused and principles driven. Rules then for evaluation purposes simply amount to looking at compliance. Was the rule followed? But with principles we're talking about evaluating meaningfulness and relevance of the principles, adherence and results. Where do the principles take you? Going to turn now to an actual example of a principles focused evaluation. for a major initiative, an anti-poverty initiative in Canada. Vibrant Communities was established in April of 2002 with 15 communities and three national sponsors meeting for a three-day forum in Guelph, Ontario to create what they called the Vibrant Communities Initiative. They jointly developed an experiment designed to test a quote new close quote way to tackle poverty in a way that acknowledged the complex nature of poverty and the effects of poverty. and the challenge of achieving scale in poverty reduction efforts. The new way was not a model, but rather a set of five core principles that local communities agreed to follow in mounting locally unique campaigns. The five principles are poverty reduction, a focus on reducing poverty as opposed to alleviating the hardships of living in poverty, comprehensive thinking and action, addressing the interrelated causes of poverty rather than its individual symptoms multi-sectoral collaboration engaging individuals and organizations from at least four key sectors business government nonprofit and persons who've experienced poverty in a joint effort rather than just one sector four community asset building building on community strengths rather than focusing on its deficits and fifth learning and change Embracing a long-term process of learning and change rather than simply undertaking a series of specific interventions. These five principles both state what the principle is that they're going to follow and what they're going to not do. So focusing on reducing poverty as opposed to alleviating the hardships of poverty. Taking an assets approach rather than a deficits approach. And these principles provided the overall direction for the initiatives in 15 different communities across Canada, which did different things, which had different foci, different activities that they undertook, and different indicators of results, but were all operating under the same umbrella principles. The results of 10 years of this initiative were published in a publication available online called Inspired Learning, an evaluation of vibrant communities'national supports. We turn now to the GUIDE framework for effectiveness principles. GUIDE is a set of criteria and a tool for writing goals that can be evaluated. It's helpful to compare the GUIDE framework to SMART goals. SMART goals specify five criteria for what constitutes a well-written well conceptualized goal for purposes of program mission and evaluation smart goals are specific measurable achievable realistic and timely well that same kind of framing leads to criteria for what constitute high quality principles the guide framework for principles calls for the principles to provide guidance to be useful to be inspiring to be developmental and to be evaluable we're going to look at each of those criteria and how they play out in crafting and evaluating principles the guide framework is core to principles focused evaluation the g stands for providing guidance The principle is directional. It specifies how to go about doing something. It uses an active verb, do this to be effective. And it's distinct from an opposite or alternative approach. So one of the ways you know you've got a meaningful principle that provides guidance is you can articulate what would be an alternative principle or an alternative direction. So when we look at the vibrant communities principle of community asset building. The principle is to build on community strengths rather than focusing on its deficits. That provides clear direction. Seek strengths, identify strengths, build on strengths. It uses an active verb, build, and it's distinct from its opposite, which is to focus on deficits or problems. Build on community strengths rather than focusing on deficits illustrates the G, the guiding nature of a good principle. The second letter, U, is for useful. It's useful. if a principle points toward desired results. It's not just process, it's also results. So it's describing how to be effective. It can be used in making choices and decision making. And utility resides in being interpretable, doable, feasible, and actionable. So, again, community asset building. Build on community strengths rather than focusing on its deficits. It points toward the results of a community. using strengths, identifying strengths, that that's the way to be effective, that when making choices about what to do, those choices would be informed by this principle of building on strengths and that what it means to build on strengths should be interpretable, doable, feasible, and actionable. That a community can identify its strengths as opposed to worrying about its deficits and problems. If so, that makes the principle useful. A third letter, the I, is for inspiring. Principles inspire. They're values-based. Values are statements of belief. We believe in working on assets, but it becomes behavioral when it becomes a principle. So that building on, acting on assets expresses the value. valuing assets. It's meaningful to people in the community to identify and know what their assets are. And it hopefully evokes a sense of purpose for a community. And in fact, that's part of what the evaluation will determine. Is the principle inspiring? Is working for assets rather than deficits meaningful and inspiring? And when we look at that principle for vibrant communities, It allows us to test out, is that principle inspiring to people in the communities where they're using it? The D in guide is for developmental, which means adaptable. The principle can be applied in different contexts. It can be applied under conditions of context, complexity, and that it's enduring. Smart goals have a time limit on it. achieve the goal by such and such a time. Principles don't go out of date. They don't expire. They're enduring. And so the vibrant community principle, building on strengths rather than deficits, has to be given meaning within each of the different communities. What are the strengths of that community? How will they adapt those strengths to whatever projects and initiatives they undertake? And how will they continue to honor the... principle over time. And finally, the E in guide is for evaluable, that the principle is stated in a way that it can be documented and judged whether it's followed, that it can be examined for what results come from it, and can determine if it takes the community where they want to go. Does building on strengths rather than assets achieve reductions in poverty? The purpose of the Vibrant Communities Initiative. That's the guide framework, G-U-I-D-E. It provides direction, is useful, the principles inspiring, developmental and adaptable, and can be evaluated. We turn now to some concrete examples of principles-focused evaluation to help make this approach real. We begin with an initiative to help youth overcome homelessness that was principles-based. Six. different programs in the Minneapolis-St. Paul metropolitan area were funded by the Brimmer Foundation. Three emergency shelters, two drop-in centers, one street outreach collaborative in Minnesota were supported to enhance their programs, and the six executive directors came together to begin a learning process with each other that I facilitated. They shared their missions, their visions, their staff manuals. their organizational and operational procedures, and as they worked together a half day a month over a couple of months, I suggested to them that while they offered different services, served different populations in different geographical arenas, a part of what they seemed to have in common were some common principles. That led to articulation of their principles and a proposal to evaluate the principles by doing case studies. of youth who had experienced these programs to see if the principles were manifest in their stories. So that we identified the principles in draft form, collaboratively identified 14 homeless youth who had been through the programs, interviewed the youth, reviewed their case file, interviewed a nominated staff person, synthesized the information, and wrote case studies. reviewed the stories with youth, analyzed the stories looking for principles and emergent trends. And this became the dissertation of Nora Murphy Johnson, who did the interviews and facilitated the analysis. Here's an example of a principle, trusting youth-adult relationships. True caring by staff is profoundly important to youth. And so the principle was and is build relationships by interacting with youth in an honest, dependable, authentic, caring, and supportive way. So in the interviews done with youth, we examine their experiences to see if this principle showed up. This is an inductive approach to evaluating a principle. What we did was interview the youth about their experiences to see... if the principal showed up in their experiences without asking the youth directly about the principal. It's inductive then, it's indirect, in that we're looking for evidence of the principals in the experience of the youth. Here are two quotations from interviews that illustrate what was coded as illustrative of trusting youth-adult relationships. Pearl, her own... made up name for the interview and the case study said, and you'd be like, okay, I have all this on my plate. I have to dig in and look into the choices I'm making to make my life more complete. And I felt that on my own, I really couldn't. Not even the strongest person on God's green earth can do it. I couldn't do it. So I ended up reaching out to the youth shelter and they opened their arms. They were just like, come, just get here. And they got me back on track. Here's Maria's quote. If I was to sit in a room and think about like everything that happened to me or I've been through, I'll get to crying and feeling like I don't want to be on earth anymore, like I wanted to die. When I talk to somebody about it, it makes me feel better. The people I talk about it give me good advice. They tell me how much they like me and how good I'm doing. They just put good stuff in my head and then I think about it and realize I am a good person and everything's going to work out better. So these are two examples from the youth of their experiences that we coded as examples of building trusting adult-youth relationships. The importance of this principle is in contrast to an alternative principle of focusing only on the transaction, on the task and work of helping get young people off the streets and into housing without building relationships. Remember, the G in the guide framework is it provides guidance and that the guidance is clearly different from alternative guidance. There are other programs that aren't able to, don't have the staff capacity or the principle of building trusting relationships, but simply work on getting the work done. And that makes this principle stand out. So alternative programs would focus on helping youth. get their high school diploma, help with resume writing, help applying for housing, while minimizing opportunities for relationships to develop. I've seen manuals of programs that say we're not in the relationship business, we're in the business of getting kids off the street and into housing. While these programs said relationships are key to helping build the trust that will support getting young people off the streets and into housing. Different principles with different results. Another principle is being journey oriented. Interact with youth to help them understand the interconnectedness of past, present, and future as they decide where they want to go and how to get there. This is in contrast to the opposite principle again of an immediate outcomes focus. Getting the youth into housing, but not having them deal with the journey. that they were on. In fact, part of the way these programs emphasized the journey orientation was to refer to the youth as youth who happened to be homeless at this time, suggesting that they weren't homeless youth. as a label and identity, but were youth on their journey who were homeless at this time. Another example of a principle was being trauma-informed. Recognize that most homeless youth have experienced trauma. Build relationships, responses, and services on that knowledge. The opposite approach would be to set standards of behavior and compliance with rules. Trauma-informed programs meant that they would recognize that young people who've been traumatized might act out, might be assertive, might be resistant to following certain rules. And as long as they were not a threat to themselves or others, the programs would try to work with them. Whereas other programs, zero tolerance programs, would not allow youth to continue in a program if they did not conform to standards of behavior. This relates to yet another principle, being non-judgmental. To interact with youth without labeling or judging them on the basis of background, experiences, choices, or behaviors. Harm reduction is another principle. Contain the effects of risky behavior in the short term and seek to reduce its effects in the long term. The opposite of that principle is zero tolerance. So that a harm reduction strategy means that if a young person shows up, high, drunk, disheveled, acting out, that as long as they are not a threat to themselves and others, the program would try to help them, the shelter would let them in, and work with them. This contrasts with a zero tolerance principle where youth who are showing conditions of being high or drunk or disheveled or acting out are not allowed into the program or the shelter. Different principles. different ways of acting, and the evaluation then is examining the extent to which the principles are actually followed, and if they are followed, what the results are for the youth in terms of getting into housing. So these principles taken in total, being journey-oriented, trauma-informed care, non-judgmental engagement, harm reduction, trusting youth-adult relationships, a strength-based approach. positive youth development and holistic approach engage youth in a manner that recognizes mental physical spiritual and social health are interconnected and interrelated these principles together constituted a overall philosophy and way of conducting quite diverse programs in different parts of the city with different youth but what their coherence constituted was adherence to this set of principles. The publication, Nine Evidence-Based Guiding Principles to Help Youth Overcome Homelessness, is available online and goes into detail about the evidence and support of each of the principles. This also brings us to discussing three major ways in which evidence-based interventions can be viewed as effective. The first of these is a summative evaluation of a single program, grant, or model, where a conclusion is reached based on the evidence about the merit, worth, and significance of an individual program. A second form of evidence-based integration is meta-analysis of results for several different programs and grants, but all using the same model and measuring the same outcomes where those Separate project evaluations are integrated and aggregated through meta-analysis to reach a more valid and credible conclusion about the effectiveness of the model. The third kind of synthesis involves having different kinds of programs, as with youth homelessness, having different kinds of programs, working in different areas with different populations, but following the same principles. That's not a statistical aggregation. That's a qualitative synthesis of diverse interventions adhering to shared evidence-based principles. And that's what the youth homelessness example illuminated. A different example of principles-focused evaluation came from working with an after-school program for African-American students from grades one through five who were Engaging with African Kwanzaa Principles, the seven principles that this program had drawn on from African traditions, and the evaluation then engaged with the young people, the students, in finding out what these principles meant to them, how they understood them to apply to their lives and the schooling, how the parents viewed them, and to look at the extent to which the principles were meaningful to the students and their parents and the community, how much they were adhered to, how did they affect their behaviors. and what results did that have for their educational experiences. A different set of principles informed a Native American after-school program. Seven Ojibwe principles of taking the form of verbs in the Ojibwe language, in the English language as nouns, humility, honesty, wisdom, truth, respect, bravery, and love. How these principles as a part of an after-school curriculum were made meaningful to students and parents and how it affected their engagement in their schooling in their community. Those are examples of principles that can be evaluated where the evaluation helps enhance adherence and meaningfulness. So what we have are a number of arenas where the terrain is difficult. where action is unpredictable, complex dynamic situations, where you can't have rigid implementation models, but need to have adaptive, responsive programming anchored by principles. Areas like conflict mitigation, where it depends on who shows up at the table, what issues arise, and how the conflict negotiations unfold that are guided by principles. Leadership development. is principles focused. Community development, where people in the community come together and engage together about setting priorities and what they want to accomplish together, that is well done with principles. Public health education is an arena that is principles focused. Human-centered design, human resources management, diversity, equity, and inclusion initiatives, youth engagement, social impact investing, psychotherapy, wilderness hiking. Leave only footprints, take only pictures is a fundamental wilderness hiking principle. Agroecology, as opposed to industrial agriculture, is principles focused. These are simply examples of the many, many arenas where principles help provide guidance, meaningfulness, and direction through complex dynamic systems. Another principle. worthy of consideration is two-eyed seeing a universal guiding principle for transcultural collaboration here is albert marshall explaining what that is who i see really is it has to be your guiding principles as to how one should live while they were here on this planet Two Eyed Seeing, as brought to us by Elder Albert Marshall, Mi'kmaq Elder Descasoni, is recognizing that there are different ways of looking at the world and maybe two of those ways are the Western scientific way and the Indigenous peoples or the Aboriginal peoples way. and that two-eyed seeing refers to finding the strengths in both of those the western and the indigenous two-eyed seeing then is a way of looking at the world that combines western science with indigenous knowledge to give a more holistic perspective and that principle of two-eyed seeing can be evaluated for its meaningfulness its adherence and the way in which it illuminates the relationship between human in nature, the results of following the principle. We turn now to another example, a global example of evaluating principles. This example concerns the evaluation of the Paris Declaration for International Development Aid. The Paris Declaration on Aid Effectiveness was a practical, action-oriented roadmap to improve the quality of international development aid and its impact on country development it put forth a series of specific implementation measures and established a monitoring system to assess progress and ensure that donors and recipients held each other accountable for their commitments The Paris Declaration Agreement spotlighted five fundamental principles for making aid more effective. And in each of these principles, you can infer the baseline against which the new principles were being adopted. The first principle, country ownership. Developing countries set their own strategies for poverty reduction, improve their institutions, and tackle corruption. This was a change from... the donor countries determining the priorities that they wanted to fund. The new principle called for countries who were receiving aid to set the priorities for the aid they received. The second principle, alignment, called for donor countries to align behind the objectives of the host receiving countries and to use local systems instead of simply their own donor systems for delivering implementing aid the third called for harmonization donor countries coordinate simplify procedures and share information to avoid duplication the baseline against this new principle was that every donor country doing its own thing operating autonomously in its own silos results developing countries and donors shift focus to development results and results get measured And finally, mutual accountability. Donors and partners are accountable for development results mutually, the status quo being that accountability was one way. Recipients of aid were accountable to donors, but donors were not accountable to recipients, including accountability for following the Paris Declaration principles. Evaluation of implementation of the Paris Declaration principles involved 22 country case studies. and 18 donor studies. The country case studies involved evaluation teams in each country interviewing key informants, reviewing documentation, looking at how donor projects were being undertaken to examine the extent to which the aid honored the Paris Declaration principles. The likewise case studies of donor countries looked at their procedures their approaches, their funding patterns, their relationships with recipient countries, and to see how much they were adhering to the past declaration principles. The metrics for the principles were to look at the direction of the aid in the countries. Were they following the principles? If so, how far were they following and what was the distance that they followed them in? and with what speed direction distance and speed of implementing the principles the evaluation found that country ownership had advanced the farthest that alignment and harmonization improved unevenly that mutual accountability and management for results lagged the most and called for action on mutual accountability to be the most important need backed by transparency and a realistic acceptance and management of risks. The evaluation of the Paris Declaration principles on aid are available online and constitute a deductive approach to evaluating principles. You recall that the evaluation of the youth homelessness program was inductive. The youth were not asked specifically about the programs. and the principles. But in this case, key informants in the countries and the donor organizations were explicitly asked about the principles and adherence to them, which led to a high degree of what we call process use. Process use is the impacts of being involved in an evaluation process for those countries, donors, and participants involved. The 22 countries that were involved in the evaluation and the donor participants had higher levels of implementation because the evaluation teams were bringing the Paris Declaration principles into the spotlight, asking questions about it, and getting them to pay attention to what implementation processes they were following. So the evaluation becomes part of an intervention to disseminate the Paris Declaration aid principles. and then to get people to think about how they're implementing them, and indeed undertaking implementation so as to be able to show that they're complying with the Paris Declaration principles. Much higher engagement with the principles among those who are involved in the evaluation. One final example, the exemplar of an organization that is principles-focused, principles-driven, and bases evaluation on principles-focused evaluation. The Global Alliance for the Future of Food. The Global Alliance is an alliance of foundations, philanthropic foundations, committed to leveraging their shared resources to help shift food and agriculture systems towards greater sustainability, security, and equity. The Global Alliance represents more than 30 foundations from 10 countries on four continents with diverse interests and expertise spanning health, agriculture, food, conservation, cultural diversity, and community well-being. They have come together guided by a set of shared principles that both express the shared values of the Global Alliance and provide a diagnostic assessment and strategic innovation. intervention tool free of realizing, they say, our overall goal. They put out publication on the principles and the future of food. The principles express values, provide actionable direction, and encompass the change we want to make interpreted within and adapted to specific sustainable food and agricultural system context. The global alliance for the future food principles are. Renewability. Address the integrity of natural and social resources that are the foundation of a healthy planet and future generations in the face of changing global and local demands. Resilience. Support regenerative, durable, and economically adaptive systems in the face of a changing planet. Equity. Promote sustainable livelihoods and access to nutritious and just food systems. Diversity. Value our rich and diverse agricultural, ecological, and cultural heritage. Inclusive. Ensure meaningful and authentic engagement of diverse people and organizations in transparent deliberations, shared power, democratic decisions, and collective actions affecting food systems for the public good. Healthfulness. Advance the health and well-being of people, the environment, and the societies that depend on both. And interconnectedness. understand the implications of the interdependence of food people and the planet in a transition to more sustainable food and agricultural systems they have developed an overall evaluation system to examine and evaluate the principles for food systems transformation directed by pablo de adora the world's first blue marble evaluation representing the global perspective the blue marble global perspective of the global alliance for the future of food the global alliance evaluation team uses a rubric to assess the degree of alignment of each principle with whatever the work is that's going on across the top you see a definition of the levels of alignment based on the consequences of the work being assessed a deeper systemic understanding potential to guide transformational engagement, integration of principles. And along the side is a five-point rubric that begins with the principle significantly advanced, that continues where the principle is moderately advanced, and goes all the way down to the principle is not advanced. This rubric and this framework is a serious engagement with and manifestation of principles focused evaluation ruth richardson the former director of the global lines for the future of food said in that publication let me start with the punch line our world needs principles she went on to write as we grapple with the realities of covert nineteen a sharper focus is placed on the climate health and environmental crises in part because they are so interconnected in part because they all speak to the deep trouble we are in as a human community so what do we do how do we respond all too often especially faced with such calamity our instinct is to reach for solid answers firm metrics dependable pathways we look to financial investments innovation commitments from governments behavior change technology policy reform these are reasonable useful and important responses But time and again they are upheld as a solution without a cause. Financial investment to what end? Behavioral change to what end? Technology to what end? This is where principles come into play. some say that principles are nice words but have little use motherhood and apple-pie statements who could disagree with those but principles are powerful if used well they tell us where we need to go and how to get there they are both the destination and the compass she concludes these principles are our destination they define our vision of food systems they help us imagine a future of food that is full of biodiversity that provides access to healthy food for all that is resilient to shocks and stresses that upholds human dignity that is built on inclusive decision making we believe our food systems should be designed to account for these principles to nurture them and ultimately to realize them as essential defining features of our future of food these principles also guide our decisions and choices we make every day they are the compass we check at regular intervals to determine if we are on the right course whether that is related to a program, a report, a convening, or the way in which we engage with others in collaboration for food systems change. The Global Alliance for the Future of Food exemplifies principles-focused evaluation, and they use that evaluation as a guidance for being true to their principles, to keeping them meaningful, and to ensure that they take them where they want to go. Principles then provide direction, but not detailed prescription, are grounded in values about what matters, are based on evidence about how to be effective, must be interpreted and applied contextually, require judgment and application, inform choices at forks in a road, are the rudder for navigating complex dynamic systems, point to outcomes and impacts, and can be evaluated for both process, implementation, and results. That is what principles-focused evaluation is about.

It's in three parts. The first part will cover the three overarching Principles Focused Evaluation questions, Definition of Effectiveness Principles, Truth and Reconciliation Principles Example, Principles-Driven Transformative Social Movements, and the Purpose and Niche of Principles-Focused Evaluation. Part two, we'll look at Principles versus Rules, an Extended Vibrant Communities Example, and the Guide Framework and Tool.

And then in part three, we'll look at four major examples. and some minor examples. The major examples of principles focused evaluation are effective principles to help youth overcome homelessness, two-eyed seeing principle for transcultural collaboration, principles for international development aid, and the principles of the global alliance for the future of food.

That's what we're going to cover in this video. Now principles are all around us. We grew up with principles telling us how we should behave, Teams have principles.

Organizations have principles. Here are some classics to remind you what we're talking about. Be mission-driven.

Do no harm. Nothing about us without us. A principle for traditionally underrepresented and unheard voices, indigenous people, people with disabilities who are now wanting to participate in research and evaluation that focuses on them, but with the principle of nothing about us without us. Stay focused on what's important.

Build trust. Nurture relationships. Take care of yourself. Do unto others as you would have them do unto you.

A golden rule, but also a principle. And speak truth to power. These are just examples of principles that surround us. Principles can be directed at individuals, Families can have principles.

The family that plays together stays together. Teams have principles. Organizations express principles.

Programs can be based on principles. Initiatives can be directed by principles. Indeed, actions of all kinds by anyone or any group anywhere at all times can be based on principles. The key here is can.

Principles can guide, but the principles-focused evaluation question is do they guide? There are three primary principles focused evaluation questions to be asked about principles. To what extent are the principles meaningful to those who are expected to follow them? If meaningful, to what extent are the principles actually adhered to?

And third, if the principles are adhered to, what results from following the principles? What difference does adhering to principles make? Principles-focused evaluation, then, is a way for principles-based programs with principles-driven leaders and staff to find out the extent to which their principles are meaningful, are adhered to, and lead them where they want to go.

This video, then, will cover how to craft meaningful and evaluatable principles, distinguish principles from rules, values, and beliefs, We'll introduce the guide tool for conceptualizing and evaluating principles. We'll give you examples of principles and examples of principles-focused evaluation. So, let's get at it. On the cover of the book is featured a sculpture of an Inuksuk.

Inuksuks are large stone sculptures in the wilderness of the Arctic by the Inuit people that provide... ways of navigating through the wilderness where there are no roads, where there are no signs. The Anaksaks become guidance.

They help locate directions. They often are places where sacred objects and food are placed to help people through the journey in the wilderness. Well, that's what principles do.

The word Anaksak means to act in the capacity of a human and that become the symbol of principles to give us the capacity to find our way through complex dynamic systems. And nunchucks represent the guidance that principles offer to find our way through difficult, tumultuous, uncertain terrain. Bill Easterly, the author of Tyranny of Experts, opens that book with the observation, it is critical to get the principles of action right before acting.

And that's why principles can be so important in program and project design, as well as in tracking their implementation and evaluating. So, what is a principle? For our purposes, an effectiveness principle provides guidance about how to think or behave towards some desired result based on norms, values, beliefs, experience, and knowledge. The principle statement is a hypothesis about effectiveness until evaluated within some context to determine its relative meaningfulness, feasibility, and utility for those committed to and attempting to follow it. We are going to be looking at effectiveness principles.

There are two aspects to this work. One is crafting principles. Just as evaluators historically got involved in helping clarify goals to make sure they were clear, specific, and measurable, so too we get involved in helping to express principles in ways that they can be evaluated in order to then evaluate their meaningfulness, adherence, and results.

Nelson Mandela, certainly a highly principled man, said, you mustn't compromise your principles, but you mustn't humiliate the opposition. No one is more dangerous than one who is humiliated. Nelson Mandela and Bishop Tutu created the truth and reconciliation process in South Africa following the end of apartheid, and that process epitomizes being principles-driven by principles-driven leaders.

The Mandela-Tutu principles of truth and reconciliation involved five principled steps. Bring together the oppressor and the oppressed, speak truth to each other. confess, forgive, and reconcile. Nelson Mandela was often asked how he could forgive the some 20 years of torture done to him, the effects on his family, upon his community, and he replied that you don't begin with forgiveness. There's a process that leads to forgiveness based upon principles of coming together with the oppressor, speaking truth to each other.

confession, and then forgiveness and reconciliation. Each of these steps in the truth and reconciliation process, each of these principles can and has been evaluated. Bringing together the oppressor and the oppressed leads to principles focused in questions like who participated in the truth and reconciliation process, who represented oppressors, who represented victims of oppression, who from both groups did not participate and why. What was the scope of participation in the process compared to the numbers of people who were engaged in oppression and those who were victims of that oppression?

These are just samples of examples of evaluating principles. I'm not going to read you the questions for each of the principles. You can pause the video if you want to read them for yourself. The point is that there are evaluation questions that guide the evaluation of the extent to which principles are meaningful, adhered to, and what results from following the principles.

I did a book with two Canadian colleagues, Frances Wesley and Brenda Zimmerman, entitled Getting to Maybe, How the World Has Changed. We studied transformative social movements and found that they were typically problem-focused and principles-driven. They didn't begin with strategic plans or clear, specific, measurable goals and logic models and theories of change. They began with a problem that they wanted to take on, that they were driven by to change, and they had principles that guided how they engaged in that work.

Mothers Against Drunk Driving, founded by Candice Leitner after her 13-year-old daughter was killed by a drunk driver, and she and other mothers began to come together. not with a grand plan about reducing the alcohol level for determining drunk driving or changing laws or increasing police spending, but to make sure that the punishment fit the crime and the drunk drivers were identified and didn't just get suspended sentences and put back on road again. and it created a lasting social movement that's now had a major impact around the world.

Bangladeshi economist Mohamed Yunus began the microfinance movement through the Grameen Bank based upon the principle that low-income, especially women, ought to have access to capital. Principles guide ongoing engagement and adaptation as projects and programs unfold. adapting to new contexts, scaling principles from one place to another, guide systems change and transformation initiatives, and guide how to navigate complex dynamic systems. The niche of principles-focused evaluation then is it's a unit of analysis. It's the evaluation focus.

The focus isn't just on attaining goals or upon testing a logic model. It's on the principles themselves. A major message of principles-focused evaluation is that principles can be evaluated and should be evaluated. It's an approach to innovative programming.

When new initiatives start out, before they have fully strategized plans and implementation means, they have principles. It's a way, as I said, to navigate complex dynamic systems. And... Principles can be a way of doing business. You're going to see examples of that as we go along.

And then, of course, it's an approach to evaluation. So the work of crafting principles we're going to take on, the work of evaluating principles we're going to take on. There are examples of principles that have had a major influence in the management world, in the business world, in the private sector world.

Jim Collins and his team, They're important books like Good to Great and Great by Choice, where they study high-performing corporations, generate principles to guide corporations in being more effective. Ray Dalio, a major finance investor, has written a best-selling book on principles, both about principles for business and investing. but principles about living. These are examples of the importance of attention to principles in our current world. There are all kinds of books about principles, non-profit management principles, and alternative non-profit management reinvented principles, principles of trauma therapy, principles of public health practice, principles of emergency management.

The six principles for exemplary teaching of English learners, servant leadership for higher education principles, and universal principles of design. Just a few examples of the many approaches to principles, all of which could be evaluated. And that's what principles-focused evaluation does.

To help illustrate what principles are, I want to compare them to rules because it clarifies the particular distinction we're making about principles. Compare rules of engagement to principles of engagement. So for example, coming to a stop sign and stopping is a rule. You come to a complete and full stop.

Rules are clear. They're evaluated by you either follow them or you don't follow them. But a principle would be represented by defensive driving. It tells you to pay attention to other drivers, to watch out for changes in weather, to be alert to what's going on, to not drive distracted, to not drive with a mobile phone going and talking on it, but to drive defensively. That doesn't give you the same specific guidance that a rule like a stop sign gives you, but it gives you direction and you have to interpret the principle for the context and conditions.

where it applies recipes give very precise directions add a quarter teaspoonful of salt a cooking principle in contrast is to season to taste and situation who's coming what's the nature of the food what spices might you want to include and you adapt the meal to those conditions there are exercise rules thirty minutes of aerobic exercise every day in contrast A principle would be to exercise regularly at a level that supports health and is sustainable given your health. lifestyle, age, and capacity. Principles have to be interpreted and applied contextually. In evaluation, a rule is to deliver the report on time, the date specified in the contract. In contrast, a utilization-focused evaluation principle is to target delivery of findings to be timely and useful for informing decisions and learning opportunities, monitor situation developments that may affect timeliness, so that You don't just look at the contract deadline, but you look at when would findings be useful and adjust to that.

That's a utilization-focused evaluation principle. One of my favorite examples is a transition from being rule-based to principles-based. In 1999, the American Academy of Pediatrics formulated guidance for parents in the form of a rule, no screens under age two.

Screens were TVs and computers that were thought to be possibly harmful to a child's brain development, and so the rule was to prohibit such screens. But subsequently, after 20 years of research and study, they moved from the rule no screens under 2 to a principle, no screens under age 2 without interaction. Interaction is a guide, not a rule. It has to be adapted to the age of the child, six months, two years old, what's on the screen, who else is present. But the point is not to use screens as a babysitter, but to engage and interact with a child with whatever is on the screen.

And therefore, it moves from a rule to a principle. Another example of moving from a rule to a principle comes from statisticians. In 2019, the American Statistician Journal published a special issue entitled Moving to a World Beyond P less than.05.

And in that special issue, they posited the rule, don't say, quote, statistically significant, close quote. They wrote, we conclude, based on our review of the articles in this issue and the broader literature, that it is time to stop using the term statistically significant entirely. regardless of whether it was ever useful a declaration of statistical significance has to-day become meaningless yet they went on solid principles for the use of statistics do exist and they are well explained in this special issue we summarize our recommendations in two sentences totaling seven words accept uncertainty be thoughtful open and modest remember adam these are principles Statistical analysis principles. Accept uncertainty, be thoughtful, be open, be modest. The point of positing these principles is to move from a mechanistic pattern of simply accepting whatever appeared to be statistically significant to a thoughtful pattern of thinking about results, accepting uncertainty, being thoughtful is key, open, and modest.

statistical analysis principles the article goes on to conclude with regard to moving from statistical significance to thoughtfulness that momentum seems to finally be on the side of reform the initial slow speed of progress should not be discouraging that is how all broad-based social movements move forward and we should be playing the long game but the ball is rolling downhill the current generation is inspired and impatient to carry this forward so let's do it let's move beyond statistically significant even if upheaval and disruption are inevitable for the time being it's worth it in a world beyond p less than point o five by breaking free from the bonds of statistical significance statistics in science and policy will be more significant than ever statisticians becoming principles focused and principles driven. Rules then for evaluation purposes simply amount to looking at compliance. Was the rule followed?

But with principles we're talking about evaluating meaningfulness and relevance of the principles, adherence and results. Where do the principles take you? Going to turn now to an actual example of a principles focused evaluation. for a major initiative, an anti-poverty initiative in Canada.

Vibrant Communities was established in April of 2002 with 15 communities and three national sponsors meeting for a three-day forum in Guelph, Ontario to create what they called the Vibrant Communities Initiative. They jointly developed an experiment designed to test a quote new close quote way to tackle poverty in a way that acknowledged the complex nature of poverty and the effects of poverty. and the challenge of achieving scale in poverty reduction efforts. The new way was not a model, but rather a set of five core principles that local communities agreed to follow in mounting locally unique campaigns.

The five principles are poverty reduction, a focus on reducing poverty as opposed to alleviating the hardships of living in poverty, comprehensive thinking and action, addressing the interrelated causes of poverty rather than its individual symptoms multi-sectoral collaboration engaging individuals and organizations from at least four key sectors business government nonprofit and persons who've experienced poverty in a joint effort rather than just one sector four community asset building building on community strengths rather than focusing on its deficits and fifth learning and change Embracing a long-term process of learning and change rather than simply undertaking a series of specific interventions. These five principles both state what the principle is that they're going to follow and what they're going to not do. So focusing on reducing poverty as opposed to alleviating the hardships of poverty.

Taking an assets approach rather than a deficits approach. And these principles provided the overall direction for the initiatives in 15 different communities across Canada, which did different things, which had different foci, different activities that they undertook, and different indicators of results, but were all operating under the same umbrella principles. The results of 10 years of this initiative were published in a publication available online called Inspired Learning, an evaluation of vibrant communities'national supports.

We turn now to the GUIDE framework for effectiveness principles. GUIDE is a set of criteria and a tool for writing goals that can be evaluated. It's helpful to compare the GUIDE framework to SMART goals.

SMART goals specify five criteria for what constitutes a well-written well conceptualized goal for purposes of program mission and evaluation smart goals are specific measurable achievable realistic and timely well that same kind of framing leads to criteria for what constitute high quality principles the guide framework for principles calls for the principles to provide guidance to be useful to be inspiring to be developmental and to be evaluable we're going to look at each of those criteria and how they play out in crafting and evaluating principles the guide framework is core to principles focused evaluation the g stands for providing guidance The principle is directional. It specifies how to go about doing something. It uses an active verb, do this to be effective.

And it's distinct from an opposite or alternative approach. So one of the ways you know you've got a meaningful principle that provides guidance is you can articulate what would be an alternative principle or an alternative direction. So when we look at the vibrant communities principle of community asset building. The principle is to build on community strengths rather than focusing on its deficits.

That provides clear direction. Seek strengths, identify strengths, build on strengths. It uses an active verb, build, and it's distinct from its opposite, which is to focus on deficits or problems. Build on community strengths rather than focusing on deficits illustrates the G, the guiding nature of a good principle. The second letter, U, is for useful.

It's useful. if a principle points toward desired results. It's not just process, it's also results.

So it's describing how to be effective. It can be used in making choices and decision making. And utility resides in being interpretable, doable, feasible, and actionable.

So, again, community asset building. Build on community strengths rather than focusing on its deficits. It points toward the results of a community. using strengths, identifying strengths, that that's the way to be effective, that when making choices about what to do, those choices would be informed by this principle of building on strengths and that what it means to build on strengths should be interpretable, doable, feasible, and actionable.

That a community can identify its strengths as opposed to worrying about its deficits and problems. If so, that makes the principle useful. A third letter, the I, is for inspiring.

Principles inspire. They're values-based. Values are statements of belief. We believe in working on assets, but it becomes behavioral when it becomes a principle. So that building on, acting on assets expresses the value.

valuing assets. It's meaningful to people in the community to identify and know what their assets are. And it hopefully evokes a sense of purpose for a community. And in fact, that's part of what the evaluation will determine. Is the principle inspiring?

Is working for assets rather than deficits meaningful and inspiring? And when we look at that principle for vibrant communities, It allows us to test out, is that principle inspiring to people in the communities where they're using it? The D in guide is for developmental, which means adaptable. The principle can be applied in different contexts.

It can be applied under conditions of context, complexity, and that it's enduring. Smart goals have a time limit on it. achieve the goal by such and such a time. Principles don't go out of date.

They don't expire. They're enduring. And so the vibrant community principle, building on strengths rather than deficits, has to be given meaning within each of the different communities.

What are the strengths of that community? How will they adapt those strengths to whatever projects and initiatives they undertake? And how will they continue to honor the... principle over time.

And finally, the E in guide is for evaluable, that the principle is stated in a way that it can be documented and judged whether it's followed, that it can be examined for what results come from it, and can determine if it takes the community where they want to go. Does building on strengths rather than assets achieve reductions in poverty? The purpose of the Vibrant Communities Initiative.

That's the guide framework, G-U-I-D-E. It provides direction, is useful, the principles inspiring, developmental and adaptable, and can be evaluated. We turn now to some concrete examples of principles-focused evaluation to help make this approach real.

We begin with an initiative to help youth overcome homelessness that was principles-based. Six. different programs in the Minneapolis-St.

Paul metropolitan area were funded by the Brimmer Foundation. Three emergency shelters, two drop-in centers, one street outreach collaborative in Minnesota were supported to enhance their programs, and the six executive directors came together to begin a learning process with each other that I facilitated. They shared their missions, their visions, their staff manuals. their organizational and operational procedures, and as they worked together a half day a month over a couple of months, I suggested to them that while they offered different services, served different populations in different geographical arenas, a part of what they seemed to have in common were some common principles.

That led to articulation of their principles and a proposal to evaluate the principles by doing case studies. of youth who had experienced these programs to see if the principles were manifest in their stories. So that we identified the principles in draft form, collaboratively identified 14 homeless youth who had been through the programs, interviewed the youth, reviewed their case file, interviewed a nominated staff person, synthesized the information, and wrote case studies. reviewed the stories with youth, analyzed the stories looking for principles and emergent trends. And this became the dissertation of Nora Murphy Johnson, who did the interviews and facilitated the analysis.

Here's an example of a principle, trusting youth-adult relationships. True caring by staff is profoundly important to youth. And so the principle was and is build relationships by interacting with youth in an honest, dependable, authentic, caring, and supportive way. So in the interviews done with youth, we examine their experiences to see if this principle showed up.

This is an inductive approach to evaluating a principle. What we did was interview the youth about their experiences to see... if the principal showed up in their experiences without asking the youth directly about the principal. It's inductive then, it's indirect, in that we're looking for evidence of the principals in the experience of the youth. Here are two quotations from interviews that illustrate what was coded as illustrative of trusting youth-adult relationships.

Pearl, her own... made up name for the interview and the case study said, and you'd be like, okay, I have all this on my plate. I have to dig in and look into the choices I'm making to make my life more complete. And I felt that on my own, I really couldn't. Not even the strongest person on God's green earth can do it.

I couldn't do it. So I ended up reaching out to the youth shelter and they opened their arms. They were just like, come, just get here.

And they got me back on track. Here's Maria's quote. If I was to sit in a room and think about like everything that happened to me or I've been through, I'll get to crying and feeling like I don't want to be on earth anymore, like I wanted to die.

When I talk to somebody about it, it makes me feel better. The people I talk about it give me good advice. They tell me how much they like me and how good I'm doing. They just put good stuff in my head and then I think about it and realize I am a good person and everything's going to work out better. So these are two examples from the youth of their experiences that we coded as examples of building trusting adult-youth relationships.

The importance of this principle is in contrast to an alternative principle of focusing only on the transaction, on the task and work of helping get young people off the streets and into housing without building relationships. Remember, the G in the guide framework is it provides guidance and that the guidance is clearly different from alternative guidance. There are other programs that aren't able to, don't have the staff capacity or the principle of building trusting relationships, but simply work on getting the work done.

And that makes this principle stand out. So alternative programs would focus on helping youth. get their high school diploma, help with resume writing, help applying for housing, while minimizing opportunities for relationships to develop.

I've seen manuals of programs that say we're not in the relationship business, we're in the business of getting kids off the street and into housing. While these programs said relationships are key to helping build the trust that will support getting young people off the streets and into housing. Different principles with different results. Another principle is being journey oriented.

Interact with youth to help them understand the interconnectedness of past, present, and future as they decide where they want to go and how to get there. This is in contrast to the opposite principle again of an immediate outcomes focus. Getting the youth into housing, but not having them deal with the journey. that they were on.

In fact, part of the way these programs emphasized the journey orientation was to refer to the youth as youth who happened to be homeless at this time, suggesting that they weren't homeless youth. as a label and identity, but were youth on their journey who were homeless at this time. Another example of a principle was being trauma-informed.

Recognize that most homeless youth have experienced trauma. Build relationships, responses, and services on that knowledge. The opposite approach would be to set standards of behavior and compliance with rules.

Trauma-informed programs meant that they would recognize that young people who've been traumatized might act out, might be assertive, might be resistant to following certain rules. And as long as they were not a threat to themselves or others, the programs would try to work with them. Whereas other programs, zero tolerance programs, would not allow youth to continue in a program if they did not conform to standards of behavior.

This relates to yet another principle, being non-judgmental. To interact with youth without labeling or judging them on the basis of background, experiences, choices, or behaviors. Harm reduction is another principle.

Contain the effects of risky behavior in the short term and seek to reduce its effects in the long term. The opposite of that principle is zero tolerance. So that a harm reduction strategy means that if a young person shows up, high, drunk, disheveled, acting out, that as long as they are not a threat to themselves and others, the program would try to help them, the shelter would let them in, and work with them.

This contrasts with a zero tolerance principle where youth who are showing conditions of being high or drunk or disheveled or acting out are not allowed into the program or the shelter. Different principles. different ways of acting, and the evaluation then is examining the extent to which the principles are actually followed, and if they are followed, what the results are for the youth in terms of getting into housing.

So these principles taken in total, being journey-oriented, trauma-informed care, non-judgmental engagement, harm reduction, trusting youth-adult relationships, a strength-based approach. positive youth development and holistic approach engage youth in a manner that recognizes mental physical spiritual and social health are interconnected and interrelated these principles together constituted a overall philosophy and way of conducting quite diverse programs in different parts of the city with different youth but what their coherence constituted was adherence to this set of principles. The publication, Nine Evidence-Based Guiding Principles to Help Youth Overcome Homelessness, is available online and goes into detail about the evidence and support of each of the principles. This also brings us to discussing three major ways in which evidence-based interventions can be viewed as effective. The first of these is a summative evaluation of a single program, grant, or model, where a conclusion is reached based on the evidence about the merit, worth, and significance of an individual program.

A second form of evidence-based integration is meta-analysis of results for several different programs and grants, but all using the same model and measuring the same outcomes where those Separate project evaluations are integrated and aggregated through meta-analysis to reach a more valid and credible conclusion about the effectiveness of the model. The third kind of synthesis involves having different kinds of programs, as with youth homelessness, having different kinds of programs, working in different areas with different populations, but following the same principles. That's not a statistical aggregation. That's a qualitative synthesis of diverse interventions adhering to shared evidence-based principles.

And that's what the youth homelessness example illuminated. A different example of principles-focused evaluation came from working with an after-school program for African-American students from grades one through five who were Engaging with African Kwanzaa Principles, the seven principles that this program had drawn on from African traditions, and the evaluation then engaged with the young people, the students, in finding out what these principles meant to them, how they understood them to apply to their lives and the schooling, how the parents viewed them, and to look at the extent to which the principles were meaningful to the students and their parents and the community, how much they were adhered to, how did they affect their behaviors. and what results did that have for their educational experiences.

A different set of principles informed a Native American after-school program. Seven Ojibwe principles of taking the form of verbs in the Ojibwe language, in the English language as nouns, humility, honesty, wisdom, truth, respect, bravery, and love. How these principles as a part of an after-school curriculum were made meaningful to students and parents and how it affected their engagement in their schooling in their community.

Those are examples of principles that can be evaluated where the evaluation helps enhance adherence and meaningfulness. So what we have are a number of arenas where the terrain is difficult. where action is unpredictable, complex dynamic situations, where you can't have rigid implementation models, but need to have adaptive, responsive programming anchored by principles.

Areas like conflict mitigation, where it depends on who shows up at the table, what issues arise, and how the conflict negotiations unfold that are guided by principles. Leadership development. is principles focused. Community development, where people in the community come together and engage together about setting priorities and what they want to accomplish together, that is well done with principles. Public health education is an arena that is principles focused.

Human-centered design, human resources management, diversity, equity, and inclusion initiatives, youth engagement, social impact investing, psychotherapy, wilderness hiking. Leave only footprints, take only pictures is a fundamental wilderness hiking principle. Agroecology, as opposed to industrial agriculture, is principles focused. These are simply examples of the many, many arenas where principles help provide guidance, meaningfulness, and direction through complex dynamic systems.

Another principle. worthy of consideration is two-eyed seeing a universal guiding principle for transcultural collaboration here is albert marshall explaining what that is who i see really is it has to be your guiding principles as to how one should live while they were here on this planet Two Eyed Seeing, as brought to us by Elder Albert Marshall, Mi'kmaq Elder Descasoni, is recognizing that there are different ways of looking at the world and maybe two of those ways are the Western scientific way and the Indigenous peoples or the Aboriginal peoples way. and that two-eyed seeing refers to finding the strengths in both of those the western and the indigenous two-eyed seeing then is a way of looking at the world that combines western science with indigenous knowledge to give a more holistic perspective and that principle of two-eyed seeing can be evaluated for its meaningfulness its adherence and the way in which it illuminates the relationship between human in nature, the results of following the principle.

We turn now to another example, a global example of evaluating principles. This example concerns the evaluation of the Paris Declaration for International Development Aid. The Paris Declaration on Aid Effectiveness was a practical, action-oriented roadmap to improve the quality of international development aid and its impact on country development it put forth a series of specific implementation measures and established a monitoring system to assess progress and ensure that donors and recipients held each other accountable for their commitments The Paris Declaration Agreement spotlighted five fundamental principles for making aid more effective.

And in each of these principles, you can infer the baseline against which the new principles were being adopted. The first principle, country ownership. Developing countries set their own strategies for poverty reduction, improve their institutions, and tackle corruption. This was a change from...

the donor countries determining the priorities that they wanted to fund. The new principle called for countries who were receiving aid to set the priorities for the aid they received. The second principle, alignment, called for donor countries to align behind the objectives of the host receiving countries and to use local systems instead of simply their own donor systems for delivering implementing aid the third called for harmonization donor countries coordinate simplify procedures and share information to avoid duplication the baseline against this new principle was that every donor country doing its own thing operating autonomously in its own silos results developing countries and donors shift focus to development results and results get measured And finally, mutual accountability. Donors and partners are accountable for development results mutually, the status quo being that accountability was one way. Recipients of aid were accountable to donors, but donors were not accountable to recipients, including accountability for following the Paris Declaration principles.

Evaluation of implementation of the Paris Declaration principles involved 22 country case studies. and 18 donor studies. The country case studies involved evaluation teams in each country interviewing key informants, reviewing documentation, looking at how donor projects were being undertaken to examine the extent to which the aid honored the Paris Declaration principles. The likewise case studies of donor countries looked at their procedures their approaches, their funding patterns, their relationships with recipient countries, and to see how much they were adhering to the past declaration principles.

The metrics for the principles were to look at the direction of the aid in the countries. Were they following the principles? If so, how far were they following and what was the distance that they followed them in?

and with what speed direction distance and speed of implementing the principles the evaluation found that country ownership had advanced the farthest that alignment and harmonization improved unevenly that mutual accountability and management for results lagged the most and called for action on mutual accountability to be the most important need backed by transparency and a realistic acceptance and management of risks. The evaluation of the Paris Declaration principles on aid are available online and constitute a deductive approach to evaluating principles. You recall that the evaluation of the youth homelessness program was inductive.

The youth were not asked specifically about the programs. and the principles. But in this case, key informants in the countries and the donor organizations were explicitly asked about the principles and adherence to them, which led to a high degree of what we call process use. Process use is the impacts of being involved in an evaluation process for those countries, donors, and participants involved. The 22 countries that were involved in the evaluation and the donor participants had higher levels of implementation because the evaluation teams were bringing the Paris Declaration principles into the spotlight, asking questions about it, and getting them to pay attention to what implementation processes they were following.

So the evaluation becomes part of an intervention to disseminate the Paris Declaration aid principles. and then to get people to think about how they're implementing them, and indeed undertaking implementation so as to be able to show that they're complying with the Paris Declaration principles. Much higher engagement with the principles among those who are involved in the evaluation. One final example, the exemplar of an organization that is principles-focused, principles-driven, and bases evaluation on principles-focused evaluation.

The Global Alliance for the Future of Food. The Global Alliance is an alliance of foundations, philanthropic foundations, committed to leveraging their shared resources to help shift food and agriculture systems towards greater sustainability, security, and equity. The Global Alliance represents more than 30 foundations from 10 countries on four continents with diverse interests and expertise spanning health, agriculture, food, conservation, cultural diversity, and community well-being.

They have come together guided by a set of shared principles that both express the shared values of the Global Alliance and provide a diagnostic assessment and strategic innovation. intervention tool free of realizing, they say, our overall goal. They put out publication on the principles and the future of food. The principles express values, provide actionable direction, and encompass the change we want to make interpreted within and adapted to specific sustainable food and agricultural system context. The global alliance for the future food principles are.

Renewability. Address the integrity of natural and social resources that are the foundation of a healthy planet and future generations in the face of changing global and local demands. Resilience. Support regenerative, durable, and economically adaptive systems in the face of a changing planet. Equity.

Promote sustainable livelihoods and access to nutritious and just food systems. Diversity. Value our rich and diverse agricultural, ecological, and cultural heritage. Inclusive.

Ensure meaningful and authentic engagement of diverse people and organizations in transparent deliberations, shared power, democratic decisions, and collective actions affecting food systems for the public good. Healthfulness. Advance the health and well-being of people, the environment, and the societies that depend on both.

And interconnectedness. understand the implications of the interdependence of food people and the planet in a transition to more sustainable food and agricultural systems they have developed an overall evaluation system to examine and evaluate the principles for food systems transformation directed by pablo de adora the world's first blue marble evaluation representing the global perspective the blue marble global perspective of the global alliance for the future of food the global alliance evaluation team uses a rubric to assess the degree of alignment of each principle with whatever the work is that's going on across the top you see a definition of the levels of alignment based on the consequences of the work being assessed a deeper systemic understanding potential to guide transformational engagement, integration of principles. And along the side is a five-point rubric that begins with the principle significantly advanced, that continues where the principle is moderately advanced, and goes all the way down to the principle is not advanced. This rubric and this framework is a serious engagement with and manifestation of principles focused evaluation ruth richardson the former director of the global lines for the future of food said in that publication let me start with the punch line our world needs principles she went on to write as we grapple with the realities of covert nineteen a sharper focus is placed on the climate health and environmental crises in part because they are so interconnected in part because they all speak to the deep trouble we are in as a human community so what do we do how do we respond all too often especially faced with such calamity our instinct is to reach for solid answers firm metrics dependable pathways we look to financial investments innovation commitments from governments behavior change technology policy reform these are reasonable useful and important responses But time and again they are upheld as a solution without a cause.

Financial investment to what end? Behavioral change to what end? Technology to what end?

This is where principles come into play. some say that principles are nice words but have little use motherhood and apple-pie statements who could disagree with those but principles are powerful if used well they tell us where we need to go and how to get there they are both the destination and the compass she concludes these principles are our destination they define our vision of food systems they help us imagine a future of food that is full of biodiversity that provides access to healthy food for all that is resilient to shocks and stresses that upholds human dignity that is built on inclusive decision making we believe our food systems should be designed to account for these principles to nurture them and ultimately to realize them as essential defining features of our future of food these principles also guide our decisions and choices we make every day they are the compass we check at regular intervals to determine if we are on the right course whether that is related to a program, a report, a convening, or the way in which we engage with others in collaboration for food systems change. The Global Alliance for the Future of Food exemplifies principles-focused evaluation, and they use that evaluation as a guidance for being true to their principles, to keeping them meaningful, and to ensure that they take them where they want to go.

Principles then provide direction, but not detailed prescription, are grounded in values about what matters, are based on evidence about how to be effective, must be interpreted and applied contextually, require judgment and application, inform choices at forks in a road, are the rudder for navigating complex dynamic systems, point to outcomes and impacts, and can be evaluated for both process, implementation, and results. That is what principles-focused evaluation is about.

Transcript for:Understanding Principles Focused Evaluation

Transcript for:
Understanding Principles Focused Evaluation