Engagement Happens Between the Replies
New research reveals what students really think about support delivered by AI-powered chatbots in the classroom.
When Mainstay first began supporting large-scale gateway courses with conversational AI, the question on everyone’s mind was: will it work? Early studies answered with a clear yes, documenting measurable improvements in persistence and outcomes when students received timely, proactive nudges via text from a friendly, school-branded chatbot.
But a more recent question is now emerging: if this approach works online, will it still help when students are back in classrooms, more connected to one another and their instructors?
A new working paper by Catherine Mata (Brown University), Katharine Meyer (Brookings Institution), and Lindsay Page (Brown University) explores this exact question. The results are still unfolding, but one thing is already clear: students are paying attention.
A New Study, a New Setting — and Familiar Signals
In this randomized controlled trial, researchers and university faculty and staff collaborated to implement an AI-powered chatbot in an in-person, introductory college course. The bot delivered messages designed to keep students on track, offering nudges about deadlines, study tips, encouragement, and resources — just as it had in earlier implementations in online courses.
The study was designed to test whether the original model would hold up when students returned to in-person learning — when they were surrounded by peers and faculty, not isolated behind screens. Researchers focused on whether the intervention could remain distinct enough to measure its real impact. To do so, they analyzed anonymized chatbot engagement data from Mainstay — such as message rates and opt-outs — alongside an end-of-course survey that captured students’ perspectives on how the chatbot influenced their experience.
But along the way, the research revealed something deeper. Students’ attitudes toward the chatbot — and the messages it delivered — were overwhelmingly positive. Key findings included:
- Opt-out rates were extremely low. While 6% of students technically opted out, many of those students had already dropped the course. The vast majority remained opted-in for the chatbot and continued receiving support.
- Engagement was broader than replies. While “active” response rates hovered around 30% — not uncommon for text-based nudge campaigns — nearly 90% of students reported reading the messages, and three out of four said the messages were helpful.
- Passive engagement is still meaningful. A large number of messages were not designed to prompt replies. They were reminders, encouragement, nudges. That most students didn’t respond should not be read as a sign of student apathy.
This adds nuance to how we interpret student interaction. A lack of response doesn’t necessarily mean lack of engagement. In fact, this study suggests the opposite.
The Message Is Getting Through
When students choose not to opt out — when they keep reading even if they don’t write back — they’re sending a signal. That signal is not always visible in a dashboard, but it matters.
In the study, students reported that they appreciated the messages even when they didn’t reply. They noticed them. They remembered them. They often cited specific examples of how a timely reminder or encouraging note helped them feel more supported or more confident in managing their coursework.
This echoes what we’ve seen in previous research, including previous research done in large online courses documented in this EdWorkingPaper, and in AI-driven classroom support at institutions like Georgia State, West Texas A&M, and Hillsborough Community College.
Together, these studies suggest that conversational AI can be a durable, human-centered layer of support — not just online, but in physical classrooms, too.
From Research to Practice
For Mainstay’s partners in higher education, the implications are powerful.
We know that proactive support works. We’ve seen it improve FAFSA completion, increase re-enrollment, and boost GPA and pass rates in gateway courses. This latest study affirms that even when students are together in person — and even when they’re not replying directly to chatbot outreach — they’re still benefiting from it.
So what should institutions take away?
- Don’t over-interpret silence. Just because a student didn’t respond doesn’t mean the message didn’t land.
- Value the opt-in. Students staying in the conversation — especially when they don’t have to — is a form of engagement.
- Design with listening in mind. Not every message needs a reply. Some just need to be seen. And if this study is any indication, they are.
Want to learn more?
If you’re interested in exploring how Mainstay’s research-backed nudges can help your students — in person or online — check out our resources and case studies.