Women gained seven grade points with Mainstay, making chatbot student performance impossible to ignore in AI college courses. In this episode: Women in an undergraduate Microeconomics course saw their final grades increase by seven percentage points due to course-specific chatbot messages, demonstrating how AI improved grades.The Mainstay chatbot improved chatbot student performance by providing timely administrative nudges and support links, rather than generating academic content.Research at Georgia State University showed that chatbot student engagement was most effective for the middle 80 percent of students, offering vital scaffolding without necessarily building transferable study habits.Implementing AI in higher ed for course support requires significant human design, oversight, and existing infrastructure for it to be effective, as evidenced by the NBER working paper.To successfully adopt AI college courses, educators should first audit specific student friction points and measure real outcomes and staff workload, rather than focusing solely on claims of AI improved grades.Chapters: 00:00 — Cold open & welcome00:30 — Women gained seven points with chatbot student performance01:00 — Mainstay chatbot: Nudging, not generating01:45 — The invisible administration and chatbot student engagement02:45 — Strongest effects for the middle 80 percent03:45 — Gender effects and the Microeconomics course04:30 — Performance vs. capability: No transfer of skills05:30 — Implementation details for successful AI college courses06:15 — Research credibility and limitations of the NBER paper07:00 — Advice for leaders: Audit the problem firstHow can teachers use AI marking safely? The study suggests AI is best used for administrative nudges like reminders and connecting students to resources, rather than for academic decision-making or grading content directly. What specific benefits did the Mainstay chatbot provide in AI college courses? The Mainstay chatbot improved chatbot student performance by sending personalized text messages for deadlines, missing work alerts, encouragement, and links to tutoring, leading to higher grades and engagement, particularly for women. Did AI improved grades for all students? While students assigned to the chatbot were generally more likely to earn an A or B, the most significant gain was observed in women in the Microeconomics course, who earned grades seven points higher; men showed no comparable effect in this study. Featuring: Dan Fitzpatrick, NBER, Georgia State University, Mainstay, Registry of Efficacy and Effectiveness Studies, National Institute for Student Success, Katharine E. Meyer, Lindsay C. Page, Catherine Mata. Follow AI in Education with Dan Fitzpatrick for more on AI in education.
