EvalNetwork EvalNetwork

James Pann interviews Melanie Hwalek, Ph.D., a program evaluation consultant, to discuss the retrospective pretest (RPT) design, focusing on its practical applications and the findings from her recent research detailed in the paper, “Designing a Questionnaire with Retrospective PrePost Items: Format Matters.” RPT is particularly useful for evaluating changes in participants’ perceptions or self-assessments following interventions such as workshops or training sessions. It can be used to address common survey research challenges encountered by evaluation consultants and researchers.

Historical Background and Evolution of RPT

Melanie traces the origins of RPT back to 1947, when it was first used to evaluate training impacts on soldiers’ attitudes. She highlights the significant milestones in RPT’s development, including its discussion in Campbell and Stanley’s seminal 1963 book on quasi-experimental designs, which solidified its methodological relevance.

Advantages of Retrospective Pretest Surveys

Practicality: Melanie emphasizes RPT’s practicality, particularly where pretesting is unfeasible, or participants are unaware of their intervention until it happens. This method consolidates data collection at a single point, thereby simplifying logistical challenges and reducing potential biases associated with traditional pre/post-testing methods.

Reduction of Response Shift Bias: A significant advantage of RPT is its ability to mitigate response shift bias. This occurs when participants’ understanding of the measured concept changes due to the intervention. For example, after a training session, participants might realize they knew less than they initially thought. RPT asks participants to reassess their prior state of knowledge or attitudes postintervention, leading to potentially more accurate change measurements. This can prevent misleading outcomes like the boomerang effect, where participants report decreased knowledge or skills postintervention—not because the intervention failed, but because their enhanced understanding reveals a previous overestimation of their capabilities.

Disadvantages and Limitations

Despite its benefits, RPT has limitations, including reliance on autobiographical memory, which can be unreliable over long periods. It may also be unsuitable for children or certain interventions where defining a clear ‘before’ state is challenging.

Insights from Dr. Hwalek’s Study on Retrospective Pretest Layouts

Melanie’s recent study, as detailed in her paper, investigated the impact of different RPT questionnaire layouts on data quality. The study involved 1,941 caregivers participating in training workshops, comparing six layouts to see which minimized errors like inattentiveness and the boomerang effect.

Key Findings:



* Best Layout: Layout 1 was found to be the most effective. It placed questions in the center, with ‘before’ responses on the left and ‘now’ responses on the right. This layout significantly reduced inattentiveness and minimized the boomerang effect, indicating that it helped participants better understand and respond accurately to the survey.

* Implications for Evaluators: These findings underscore the need for careful consideration of survey design in RPTs to enhance data reliability and validity.



Conclusion

The interview with Dr. Hwalek provides comprehensive insights into the retrospective pretest design, reinforcing its utility in evaluating the impact of interventions and assisting program evaluation consultants.

David Fetterman is a leading expert in empowerment evaluation, an approach that emphasizes collaboration, participation, and capacity building. He has written extensively on the topic, and his work has been used in a wide range of settings, including government agencies, non-profit organizations, and businesses.

David’s work focuses on helping people evaluate their programs and initiatives and use the findings to improve their work. He believes that empowerment evaluation is essential for building a more just and equitable society. David is known for his innovative work, commitment to social justice, and ability to make complex ideas accessible to a wide audience.

Understanding Empowerment Evaluation: A Shift in Traditional Evaluation Models

Empowerment evaluation revolutionized the way we view and conduct evaluations. David explains how this approach places the evaluated individuals in control, transforming them from subjects to leaders of the evaluation process. This shift not only uncovers real, often overlooked issues but also ensures more sustainable results by fostering a sense of ownership and self-efficacy among the participants.

Integration of Empowerment Evaluation in Organizations: A Case Study

David discusses the seamless integration of empowerment evaluation within organizational structures. He highlights a case where a school system successfully internalized evaluation, making it a natural and credible part of their daily operations and program implementation. This integration allowed for timely and data-driven mid-course corrections, ensuring the longevity and effectiveness of the evaluative practices within the organization.

Empowering Communities through Participatory Evaluation: An Oregon Case Study

This episode further explores the impact of empowerment evaluation on community development. Fetterman shares an inspiring story from Oregon, where children, armed with collected cigarette butts, advocated for smoke-free policies. This participatory approach, Fetterman notes, empowers communities to set and achieve their own goals, fostering independent growth and development.

Microskills for Facilitating Empowerment Evaluation Meetings

David delves into the essential microskills required for effectively facilitating evaluation meetings. From personal and communication skills to technical advising, these microskills ensure that the evaluation process is respectful, inclusive, and constructive, laying the foundation for meaningful and lasting change.

Navigating Cultural and Political Realities in Project Facilitation

The episode also touches on the challenges of navigating diverse cultural and political landscapes in project facilitation. David shares personal experiences, emphasizing the importance of listening and adapting to local perspectives to ensure the success and acceptance of the projects abroad.

Utilizing Evaluation and Empowerment for Positive Community Change

David highlights the transformative power of evaluation and empowerment in enacting positive community change. He shares a compelling story of a community member’s journey from opposition to advocacy for a teenage pregnancy prevention program, underscoring the importance of understanding and addressing underlying concerns within the community.

Maintaining Self-Awareness and Reflection in Project Work

In the podcast, David also discusses the critical role of self-awareness and reflection in project work. He shares insights on receiving and learning from feedback, working closely with community partners, and ensuring the focus remains on strengthening the community rather than asserting control.

Making Evaluation Principles Explicit for Project Success

He explains the significance of making evaluation principles explicit to ensure project success. He recounts a project experience in East Palo ...

Whether you're an educator, a student, or simply someone passionate about youth development and educational programs, this podcast episode with Tiffany Berry, PhD., promises to give you insights into the complex world of evaluation. She is the Dean and a full research professor in the Division of Behavioral & Organizational Sciences at Claremont Graduate University, where she also received her Ph.D.



Her research interests include Educational Program Evaluation, Educational Curricula, Comprehensive School Reform, after-school services, social-emotional learning, and other areas. She has published over 75 technical evaluation reports and peer-reviewed articles in leading evaluation and youth development journals.



See XXXXX for a detailed description of the interview.



OUTLINE:

0:00 Overview of episode

2:07 Evaluation models and conceptual models she uses

8:56 Common challenges Tiffany encountered evaluating educational programs

11:59 Challenges in measuring youth development outcomes

17:36 Changing her evaluation approach due to unforeseen circumstances or changes in the program

23:00 How to stay current in the field of evaluation

28:10 Advice for those interested in becoming an evaluator

32:41 Advantages and disadvantages of getting a graduate degree in evaluation versus other disciplines

38:15 Credentialing in evaluation in the US

44:32 The uniqueness of Claremont’s evaluation program

49:40 Claremont’ Doctoral program in evaluation

55:09 Building a professional network in the evaluation field



CONNECT WITH JAMES:

- Subscribe to this YouTube channel: https://www.youtube.com/user/jamesmpa...

- LinkedIn: https://www.linkedin.com/in/pannjames/



Please reach out with comments and questions. Thanks!

In this episode, I talk with Jim McDavid, Ph.D., about his experience with mindfulness and meditation practice, how it has influenced him, and how it affects how he views and practices evaluation. Our conversation also covers practical wisdom, Jim’s interest in the environment, and challenges associated with determining cause and effect in evaluation.



Jim is Professor Emeritus at the School of Public Administration, University of Victoria, which he joined as faculty in 1980. He was a recipient of the University of Victoria Alumni Teaching Award, and he received the University of Victoria’s highest academic honor, the UVic Distinguished Professorship Award. He has contributed significantly to the field of evaluation, and the reason I reached out to him related to his work related to mindfulness and evaluation.



00:00 Introduction

01:57 Jim’s definition of mindfulness

05:20 The ethical dimension of mindfulness

08:49 How Jim’s mindfulness practice has evolved over time

11:46 Difference between Transcendental Meditation and Vipassana

15:33 The spiritual dimension of meditation

22:10 Subjectivity of cause and effect

29:34 How mindfulness opens up the mind

35:52 Mindfulness can support evaluation practice

40:32 Connection to practical wisdom

44:45 Importance of being present and really listening in evaluation contexts

49:20 Judgment and evaluation

50:46 Starting a mindfulness practice



For more go to: https://evalnetwork.com/mindfulness-meets-evaluation/

In this episode, James Pann, Ph.D., interviews Sheila Robinson, Ed.D., about the topic of surveys and response rates. We focus on the significance of response rates in surveys and the steps that can be taken to maximize them.



Sheila is a career educator and professional learning designer with experience in K-12 public education and higher education, a certified program evaluator, and Certified Presentation Specialist (CPS™) with particular interests in survey design, data visualization, and presentation design.



Read more here: https://evalnetwork.com/maximize-your-survey-response-rates



Timeline

00:40 Why Sheila is interested in survey research

03:50 Why survey response rate is critical

05:15 The optimal response rate

07:37 Ensuring accurate demographic representation of the sample

09:30 How to improve response rate

12:01 The survey invitation message is critical

13:26 Transparency in the length of time to complete the survey is important

15:19 Survey reminders can be used tactically to improve the response rate

16:43 Incentives should be used carefully

20:46 Timing of incentives and the principle of reciprocity

23:08 Building survey completion time into program activities can increase response rate

26:17 Importance of piloting the survey prior to use

28:15 Who sends the survey is important

33:20 How to reach Sheila



EPISODE LINKS:

- Sheila's website: https://www.sheilabrobinson.com/

- Sheila's Linkedin: https://www.linkedin.com/in/sheilabrobinson/



CONNECT WITH JAMES:

- Subscribe to this YoutTube channel: https://www.youtube.com/user/jamesmpann?sub_confirmation=1

- LinkedIn: https://www.linkedin.com/in/pannjames/



Please reach out with comments and questions. Thanks!

/*! elementor - v3.21.0 - 30-04-2024 */

.elementor-widget-text-editor.elementor-drop-cap-view-stacked .elementor-drop-cap{background-color:#69727d;color:#fff}.elementor-widget-text-editor.elementor-drop-cap-view-framed .elementor-drop-cap{color:#69727d;border:3px solid;background-color:transparent}.elementor-widget-text-editor:not(.elementor-drop-cap-view-default) .elementor-drop-cap{margin-top:8px}.elementor-widget-text-editor:not(.elementor-drop-cap-view-default) .elementor-drop-cap-letter{width:1em;height:1em}.elementor-widget-text-editor .elementor-drop-cap{float:left;text-align:center;line-height:1;font-size:50px}.elementor-widget-text-editor .elementor-drop-cap-letter{display:inline-block} James Pann, Ph.D., interviews David Victorson, Ph.D., of True North Treks, a nonprofit organization whose mission is to empower young adults and caregivers affected by cancer to “find direction through connection” and mindfulness.As a child, David grew up surrounded by nature and its many restorative benefits. Therefore, when he went on to complete his postdoctoral fellowship in psychosocial oncology as a psychologist, he saw an opportunity to bring nature’s gifts to the young adult cancer patients he was seeing.In 2008, he co-founded True North Treks to fill some of the unmet needs of these cancer survivors and their caretakers and help them get their lives back on track. The reconnecting power of nature, coupled with mindfulness and meditation laid the basis for these restorative journeys.David goes on to discuss one of the most reported unmet needs: isolation. Many of the young cancer patients/survivors feel like they don’t know anybody like them. These treks allow the opportunity for deep social connection with others going through the same or similar experiences. These needs and solutions developed into three key points.







/*! elementor - v3.21.0 - 30-04-2024 */

.elementor-heading-title{padding:0;margin:0;line-height:1}.elementor-widget-heading .elementor-heading-title[class*=elementor-size-]>a{color:inherit;font-size:inherit;line-height:inherit}.elementor-widget-heading .elementor-heading-title.elementor-size-small{font-size:15px}.elementor-widget-heading .elementor-heading-title.elementor-size-medium{font-size:19px}.elementor-widget-heading .elementor-heading-title.elementor-size-large{font-size:29px}.elementor-widget-heading .elementor-heading-title.elementor-size-xl{font-size:39px}.elementor-widget-heading .elementor-heading-title.elementor-size-xxl{font-size:59px}True North Treks 3 Crucial Connections







1) Connection with nature (after going through something as unnatural as cancer treatment);2) Connection with peers who get it and have walked a similar path;3) Connection with oneself through mindful awareness practices, such as meditation and yoga.While it may sound like a therapy session at first, David emphasizes the lack of an explicit group therapy aspect. The guides are trained never to question the participants about their cancer and instead simply sit back and allow them to speak their minds. Often, the participants will immediately start talking about their cancer experience on their own.The guides, primarily mental health professionals, are taught to be themselves and simply bring mindfulness coaching. The participants benefit from the mindfulness and yoga experience and being with each other in the outdoors. That said, a “therapeutic” aspect tends to emerge on its own when the participants find themselves with several others just like themselves.Being one of the 3 Crucial Connections, David defines what mindfulness means on the treks.