The Method to Our Madness

staceymerola
  • 5.0 (3)
  • Social Sciences
  • Biweekly

The Method to Our Madness is a podcast that explores the art, ethics, and impact of research and evaluation. Hosted by award-winning evaluator and musician Dr. Stacey Merola, and William Pate, who together have close to 50 years of evaluation experience, each episode dives into the stories behind the data. Join our two hosts as they discuss their trials and tribulations as program evaluators: how they engage stakeholders, ask questions, measure change, and translate findings into action.  From equity-driven frameworks to creative data storytelling, this show brings together thought leaders, community advocates, and methodologists to rethink what meaningful evidence looks like. Whether you're a seasoned researcher or just curious about how evaluation shapes the world, tune in for insights that are rigorous, resonant, and real. 

  1. Oct 28

    Before and After: Getting the Most Out of One-Group, Pre-Post Designs

    Can’t afford a randomized controlled trial? You’re not alone, and a pretest-posttest design may get you further than you think. In this episode, Stacey Merola and Bill Pate dig into pre-post designs: where they fall short, how to make them stronger, and why they still matter for nonprofits pilot-testing a program or going after funding to scale. They start with the threats to validity that can undermine a simple before-and-after comparison: regression to the mean, maturation, test sensitization, rater drift and attrition. Then they turn to practical ways to shore up the design. They cover retrospective pre-post tests, including new research on how survey layout affects what people remember, and Bill’s marine biology program where self-ratings went down after training. They also cover gain scores and propensity score matching for Department of Education projects, delayed post-tests to measure whether effects last, and non-equivalent dependent variables as a low-cost stand-in for a comparison group. Finally, they get into the analysis: when to use a paired t-test, when to switch to the Wilcoxon signed-rank test, and how careful documentation can help your evidence stand up to funders. In this episode: Why pre-post designs still have a place in program evaluation Common threats to validity, and how to spot them Getting the most out of retrospective pre-post surveys Gain scores, propensity score matching and What Works Clearinghouse standards Delayed post-tests and interrupted time series Paired t-tests vs. Wilcoxon signed-rank tests References mentioned: Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Hwalek, M., Pierce, J., & Straub, V. (2024). Designing a questionnaire with retrospective pre-post items: Format matters. Evaluation and Program Planning, Volume 103. https://doi.org/10.1016/j.evalprogplan.2024.102411 Episode notes can be found here. Podcast music was written and produced by Stacey Merola. Podcast artwork was created by Zoe Targino.

  2. Oct 14

    Asking the Hard Questions: Researching Campus Sexual Assault

    Content note: This episode includes a discussion of sexual assault and gender-based violence. It also contains some explicit language.  How do you study something as sensitive as sexual assault, and do it ethically? In Episode 23, Stacey and William talk with Dr. Alison Cares, an associate professor of sociology at the University of Central Florida who researches gender-based violence on college campuses. Alison explains how she recruits survivors through surveys, flyers and Reddit threads. She describes how informed consent and empowerment models protect participants, and why IRBs and grant reviewers sometimes underestimate how resilient survivors are. She also explains why campus climate surveys can produce very different numbers, like Cornell’s widely reported 35%, depending on how assault is defined and over what period. The conversation then turns to prevention: what bystander intervention can and can’t do, why community-level change matters, and why teaching consent has to start early. Mentioned in this episode: Sexual Citizens: Sex, Power, and Assault on Campus by Jennifer S. Hirsch and Shamus Khan (W. W. Norton, 2021) A summary of this episode can be found here.  The podcast music was written and produced by Stacey Merola. The podcast artwork was created by Zoe Targino.

  3. Oct 1

    Librarian in the Loop: What You Can Gain from Having a Research Librarian on Your Team

    What can a research librarian find that a search engine, or an AI tool, can't? In this episode, Stacey Merola and William Pate talk with Matthew Von Hendy, a research librarian and the founder of Green Heron Information Services, about where library science and evaluation meet. Matthew worked at science and technology libraries in the DC area, including EPA, NASA and the National Academies, before he started consulting for evaluators in 2012. He tells stories from that work, like tracking down an obscure federal survey question about limited-English speakers and finding hard-to-locate survey items on fish consumption. He also explains what makes a literature search work: the reference interview, structured protocols like PRISMA, and the persistence it takes to find gray literature such as dissertations, government reports and patents. The conversation also covers: Why people have predicted that technology would replace librarians for decades, from microfiche to the web to AI, and why it hasn't happened What AI can and can't do for research, including the risk of made-up citations and the trouble that's causing in courtrooms The real cost of doing research yourself versus hiring an expert, including the person who spent 35 hours avoiding a $35 article How a single networking dinner led to lasting collaborations among independent evaluators How a firefighting-foam research project turned into a popular PFAS newsletter and made Matthew "the PFAS dude" This episode is for evaluators, researchers and anyone who has ever wondered whether their search missed something important. Learn more about Matthew's work at www.greenheroninformation.com. A summary of the episode can be found here.  The podcast music was written and produced by Stacey Merola. The podcast artwork was created by Zoe Targino.

  4. Sep 16

    AI in Qualitative Research: Promise, Pitfalls, and the Human Element

    Can AI really handle coding qualitative data, or is it just a shortcut that costs you in the end? In this episode, Stacey and Bill sit down with Dr. Asher Beckwitt, a researcher with 26 years of experience across 5,000 qualitative and mixed-methods studies, to unpack where AI genuinely helps with qualitative data analysis, and where it falls short. Asher walks through choosing compliant, privacy-safe AI tools, why human verification of AI outputs is non-negotiable, and how to document AI use for transparency and IRB compliance (including the often-overlooked step of fully deleting data from AI systems). They also break down the differences between grounded theory, thematic analysis, and phenomenology, and where AI fits (or doesn't) in each.  A must-listen for researchers weighing AI tools against methodological rigor in their qualitative research projects.  This episode also touches a fish consumption study Asher led for a tribal nation using community-based participatory research methods. This was a separate, non-AI project worth a listen on its own. Dr. Asher Beckwitt can be reached at asherbeckwitt@gmail.com or www.asherconsult.com. A summary of this episode can be found here.  The podcast music was written and produced by Stacey Merola. The podcast artwork was created by Zoe Targino.

  5. Sep 2

    Equipment Cut in Half: Lessons in Planning from Systems Engineering

    In this episode, Stacey and Bill talk with Joan Wagner about how systems engineering and industrial engineering (IE) work together. The former includes defining and verifying requirements, while the later involves optimizing processes and efficiency. Highlights include: The "Systems Engineering V" model and the importance of setting clear project boundaries Why Lean and Kaizen work best when the people doing the work drive the improvements A story about building stakeholder buy-in with a resistant shop-floor manager Costly planning failures, including a machine built too tall for its doorway Where Six Sigma works well, and where it gets misapplied The ongoing search for better systems-engineering documentation tools Joan's outreach work with ISE Day, bringing IE education to K–12 students Episode notes can be found here.  The podcast music was written and produced by Stacey Merola. The podcast artwork was created by Zoe Targino.

  6. Aug 19

    Actionable Evaluation: From Health Data to Policy Recommendations

    Focusing on policy recommendations as part of the evaluation is critical because health data without policy recommendations grounded in data creates a dangerous blind spot for policymakers, often leading to well-intentioned regulations that fail on the ground.    While quantitative metrics can pinpoint a drop in Medicaid enrollment, for example, they cannot capture the confusing administrative hurdles, conflicting agency rules, or frontline operational friction causing those trends. By integrating targeted focus groups and interviews into the data findings, evaluators can diagnose the actual root causes of systemic failures which can then provide opportunities for strong policy recommendations and policy frameworks.    In this episode, we're joined by Nalini Pande, Managing Director at Sappho Health Strategies, to talk about the unique challenges of evaluating healthcare programs and translating results into policy. Drawing on her deep background in Medicaid, Medicare, and ACA policy, Nalini makes the case for weaving policy considerations into an evaluation from day one, rather than adding them on at the end.   A summary of this episode can be found here.    The music for this podcast was written and produced by Stacey Merola. The artwork was created by Zoe Targino.

  7. Aug 5

    Summer Vibes: Mixed Methods, Moose, and Making Deals

    What do mixed methods interrupted time series designs, business development, and moose have in common? More than you might think. In this episode of The Method to Our Madness, we explore the power of mixed methods interrupted time series designs and how they help researchers and evaluators understand change over time. We also share insights from the world of business development, including the challenges and opportunities of building partnerships, pursuing new work, and growing a business. Along the way, we take an unexpected detour into the world of moose, because sometimes the most memorable conversations happen when data, strategy, and wildlife collide. Whether you're interested in research methods, professional growth, or simply enjoy hearing smart people wander off on a tangent, this episode offers practical insights, candid reflections, and a few moose stories you didn't know you needed. Podcast music written and produced by Stacey Merola. Podcast artwork was created by Zoe Targino.

  8. Jul 23

    Knowledge Maps: Seeing the Whole Picture

    What if your logic model is too simple to capture how your program actually works? Hosts Dr. Stacey Merola, and Bill Pate sit down with Dr. Bernadette Wright to explore knowledge mapping. A knowledge map is a visual approach that uses circles and arrows to show the interconnected web of causes and effects behind real-world outcomes, instead of forcing everything into a rigid input-output framework. Dr. Wright walks through how knowledge maps are built (often with 20-25+ interconnected concepts), why multiple reviewers matter, and how the process helps teams surface feedback loops, catch oversimplified solutions, and get aligned on what's actually driving results. The conversation also covers popular mapping tools, how the approach compares to traditional logic models, and where generative AI might fit into the future of concept mapping. A must-listen for evaluators, researchers, and program leaders who want a clearer picture of complexity. An example of a knowledge map can be found here. Source: Strategic Knowledge Mapping: The Co-Creation of Useful Knowledge A summary of the episode can be found here.  The podcast music was written and produced by Stacey Merola. The podcast artwork was created by Zoe Targino.

See All (24)

Ratings & Reviews

5
out of 5
3 Ratings

About

The Method to Our Madness is a podcast that explores the art, ethics, and impact of research and evaluation. Hosted by award-winning evaluator and musician Dr. Stacey Merola, and William Pate, who together have close to 50 years of evaluation experience, each episode dives into the stories behind the data. Join our two hosts as they discuss their trials and tribulations as program evaluators: how they engage stakeholders, ask questions, measure change, and translate findings into action.  From equity-driven frameworks to creative data storytelling, this show brings together thought leaders, community advocates, and methodologists to rethink what meaningful evidence looks like. Whether you're a seasoned researcher or just curious about how evaluation shapes the world, tune in for insights that are rigorous, resonant, and real. 

Information

  • Creator
    staceymerola
  • Years Active
    2025 - 2026
  • Episodes
    24
  • Rating
    Clean
  • Copyright
    © Copyright 2025 All rights reserved.
  • Show Website
    The Method to Our Madness