Improving Development Evaluation David Wand

In this episode 7, part 3, Dr. Wand continues his discussion with evaluation expert Yvonne Okeke about Equitas www.equitas.org Performance Measurement Framework (PMF) for a $18 Million project in Burkina Faso, Haiti, Kenya, Tanzania, Senegal entitled Achieving Equality through Human Rights Education. You can learn more about the project here: https://w05.international.gc.ca/projectbrowser-banqueprojets/project-projet/details/P006852001 The experts conclude that Equitas cannot make the claim that their project is achieving its outcomes because only 1 of their 20 outcome indicators in their PMF properly measures the project outcomes.
This discussion focuses on why Equitas chose to promote gender equality in 5 countries that are not democracies, according to the 2023 Democracy Index with the 2023 Freedom House index noting that Kenya, Tanzania, and Senegal were only 'partly free.'
They recommend that this project promoting gender equality might wish to focus on democratic countries in Africa such as Ghana, Lesotho, Namibia, South Africa, and Botswana, according to the 2023 Democracy Index.
They also discuss the 2 outcome indicators that, in combination, properly measure and evaluate the achievement of improvements in leadership skills to mobilize communities in promoting gender equality.
The PMF and complete summary of all 20 outcome indicators for this project is available by emailing evaluatecanadaaid@gmail.com.
#internationaldevelopment
#evaluation

In this episode 7, part 2, Dr. Wand discusses with evaluation expert Yvonne Okeke Equitas www.equitas.org Performance Measurement Framework (PMF) for a $18 Million project in Burkina Faso, Haiti, Kenya, Tanzania, Senegal entitled Achieving Equality through Human Rights Education. You can learn more about the project here: https://w05.international.gc.ca/projectbrowser-banqueprojets/project-projet/details/P006852001 The experts conclude that Equitas cannot make the claim that their project is achieving its outcomes because only 1 of their 20 outcome indicators in their PMF properly measures the project outcomes. This discussion focuses on the flaws with the PMF and provides solutions to how the PMF and the overall evaluation of the project can be improved. In brief, those solutions include:
Replace your self-reporting bias (i.e. perceived capacity, levels of confidence, transferring knowledge) from your project beneficiaries (i.e. women, men, women organizations, civil society organizations, others) with external, objective measures of their technical capacity to promote and lobby for gender equality. Replace your subjective and frequency measures reported by the duty bearers with objective behaviours such as increased expenditures on gender equality programs 0r legislation passed that reflects better gender equality. Your current measures of empowerment in the PMF are not valid measures of empowerment.Either introduce comparison groups or measure your target groups more frequently to support your claims that your project outcomes have been achieved due to the project. The PMF and complete summary of all 20 outcome indicators for this project is available by emailing evaluatecanadaaid@gmail.com.
#internationaldevelopment
#evaluation

In this episode 7, part 1, Dr. Wand critiques the Equitas Performance Measurement Framework (PMF) for a $18 Million project in Burkina Faso, Senegal, Kenya, Tanzania, and Haiti entitled Achieving Equality through Human Rights Education. You can learn more about the project here: https://w05.international.gc.ca/projectbrowser-banqueprojets/project-projet/details/P006852001 He concludes that the PMF is flawed and that Equitas www.equitas.org cannot make the claim that its project services have achieved its project outcomes. He also describes the services that the project delivers and invites listeners to request a copy of the PMF and his critique. Finally, he invites a Equitas representative to attend Part 2 of the episode to respond to his critique and provide solutions as to how to improve the evaluation of the project.

In this episode 6, part 2, Dr. Wand discusses with evaluation expert Dr. Jenny Jorgenson Colleges & Institutes Canada www.collegesinstitutes.ca Performance Measurement Framework (PMF) for a $18 Million project in Senegal entitled A Thousand women: I am woman, I exist, I participate. You can learn more about the project here: https://w05.international.gc.ca/projectbrowser-banqueprojets/project-projet/details/P006852001. The experts conclude that Colleges & Institutes Canada cannot make the claim that their project is achieving its outcomes because only 1 of their 17 outcome indicators in their PMF properly measures the project outcomes. This discussion focuses on the flaws with the PMF and provides solutions to how the PMF and the overall evaluation of the project can be improved. In brief, those solutions include:
Replace your self-reporting bias of your project beneficiaries (i.e. vulnerable women) from your focus groups with measuring others (e.g. community members, others within their households) on awareness of the need for women to participate in decision-making bodies;Measure women's incomes and their ability to make decisions that show their power before you claim that your project women have increased their 'empowerment'. Your current measures of empowerment in the PMF are not valid measures of empowerment.Either introduce comparison groups or measure your target groups more frequently to support your claims that awareness levels or empowerment levels have increased. Currently, you only measure your target groups once per year in your PMF. This is inadequate to support your claim that your leadership training and entrepreneurship training are responsible for increases in women empowerment.The PMF and complete summary of all 17 outcome indicators for this project is available by emailing evaluatecanadaaid@gmail.com.

In this episode 6, part 1, Dr. Wand critiques the Colleges & Institutes Canada www.collegesinstitutes.ca Performance Measurement Framework (PMF) for a $18 Million project in Senegal entitled A Thousand women: I am, I exist, I participate. You can learn more about the project here: https://w05.international.gc.ca/projectbrowser-banqueprojets/project-projet/details/P006852001 He concludes that the PMF is flawed and that Colleges & Institutes Canada cannot make the claim that its project services have achieved its project outcomes. He also describes the services that the project delivers and invites listeners to request a copy of the PMF and his critique. Finally, he invites a Colleges & Institutes Canada representative to attend Part 2 of the episode to respond to his critique and provide solutions as to how to improve the evaluation of the project.

In this episode 5, Part 2, Dr. Wand discusses with evaluation expert Benjamin Serebour and MEDA representative Yasir Dildar the Mennonite Economic Development Associates www.meda.org Performance Measurement Framework (PMF) for a $19,040,307 Million project in Africa, Americas, and Asia entitled A New Partnership for Sustainable Impact Investing in Frontier Markets. You can learn more about this project at https://w05.international.gc.ca/projectbrowser-banqueprojets/project-projet/details/D000120001. This discussion focuses the flaws with the PMF and provides solutions to how the PMF can be improved. In brief, those solutions include:
Stop using self-reporting/Most Significant Change from individuals trained/mentored by MEDA as a method to claim that these individuals have experienced increased levels of skill/knowledge/technical ability. Replace this with objective, technical measures of technical ability. If this is too expensive, take a sample. If these individuals and organizations refuse this, do not provide the training and mentoring. You have that leverage.For organizations or 'entities' that MEDA claims to be increasing their standards of reporting on gender and environmental issues due to their training/mentoring, replace the number reporting on these standards with the percentage achieving this minimum standard using your technical audits and put this in your PMF.MEDA cannot find comparison groups that wish to participate in being measured on their project outcomes. Fair enough. Take your treatment group and use half of them as your comparison group and deliver your training/mentoring to that group later. See Chapter 5 Quasi-Experimentation Handbook of Practical Program Evaluation Second Edition ISBN 0-7879-6713-0 for other options without using comparison groups.Include country level reporting in your PMF as there are bound to be variations in performance between countries.