In late 2015, Georgia State came to us with a major challenge. They were grappling with runaway summer melt that had increased from 12% to nearly 19% in a few short years.
They knew that text messaging was the best way to engage the current generation of students, but they didn’t know how to manage personalized conversations with every admitted student with their existing staff on a reasonable budget. Here’s what happened…
Say Hello to Pounce
To help scale Georgia State’s personalized student communication, we created “Pounce”—a custom virtual assistant for GSU admissions. Pounce was designed to help students by sending timely reminders and relevant information about enrollment tasks, collecting key survey data, and instantly answering students’ many questions around the clock.
A Randomized Controlled Trial with Mainstay
RCTs are the gold standard for evaluating program effectiveness and are the most reliable way to determine what works and what doesn’t.
With the help of Professors Hunter Gehlbach and Lindsay Page, we implemented an RCT at Georgia State — half of the incoming freshman class were given access to Pounce and the other half were selected for the control group. Finally, the day came when we were ready to launch with thousands of students…
Day One: Will Our Heads Explode?
In early April 2016, we switched Pounce on, and 3,114 admitted students started texting, representing a 90% opt-in rate among admitted students with valid U.S. cell phone numbers.
While we were thrilled with the student engagement, we could never have predicted how transformative this platform would be for our admission partners. In working hand-in-hand with GSU’s team, we saw how Pounce supercharged response times and the level of support the admissions team was able to offer. It was like wearing an Ironman suit for communication – a single person could manage literally thousands of conversations simultaneously!
By program’s end, 63% of all students in the treatment group had engaged with Pounce on at least 3 separate days throughout the enrollment process and had exchanged an average of 60 messages. Of the 50,000-plus student messages received, less than one percent required the attention of Georgia State staff—the rest were handled by Pounce or Mainstay staff overseeing the virtual assistant’s learning process.
We also found that first generation and Pell-receiving students sent on average 9.4% and 31.7% more messages to Pounce, respectively. For schools looking for alternatives to engage underserved students, Pounce seems to have resonated particularly well with these two groups.
We also saw an interesting breakdown in the demographics of engagement by race with Asian-American, Hispanic, and African-Americans sending 12.3%, 12%, and 3.4% more messages than the average GSU student respectively.
How was Pounce received?
So how do all of these messages translate into student satisfaction and performance? Simply put, students loved chatting with Pounce: 80% gave it 4 or 5 out of 5 stars and 94% recommended that Georgia State introduce Pounce to next year’s admitted class.
Among their favorite characteristics, students noted that they appreciated Pounce’s casual language, ease of accessibility, and personalization of messages. One student said, “It was the easiest part of enrollment.”
In our focus groups with students who engaged with Pounce, we learned that some of the common themes were that students didn’t feel judged for asking what might seem like a “stupid” question, they appreciated the instantaneous responses, especially when they asked questions at all hours of the night, like this:
Overall, Pounce made for a pretty great guide for the admissions and enrollment process at GSU.
What’s truly impressive is that throughout the course of the RCT, the treatment group (those students who texted with Pounce) had significantly higher completion rates of key enrollment steps than the control group. Furthermore, since students communicated their decision changes to Pounce in real-time, GSU always had a better pulse on the enrollment intention of students in the treatment group.
Without further ado, here are the results for the students who committed to Georgia State in terms of percent impact on various enrollment tasks:
It’s worth noting, by the way, that the one negative number for FAFSA Verification is actually a good thing. It means more students had flawless FAFSAs and thus did not need further verification from the GSU financial aid office.
Now for the main event. For students who committed by June 1 — GSU’s priority deadline — the treatment effect was a 3.3% increase in enrollment and a 21.4% reduction in summer melt. This is a significant contribution to the overall record year for enrollment at the university.
It’s also worth noting that we received 9.4% and 31.7% more messages on average from first-generation college students and Pell-eligible students, respectively.
Commenting on the results he saw with students, Georgia State’s Assistant Vice President for Undergraduate Admissions, Scott Burke, said:
Following the success of last year’s interventions on their summer melt, we are excited to expand our partnership with Georgia State this year. Not only will we be increasing our scope of work with their recruitment and admissions efforts, but we will also be working on exciting new initiatives around student success.
To build artificial intelligence that can understand language is certainly a daunting task, but we are greatly encouraged by the progress our technology has made over the past twelve months, and we’re even more impressed with how enthusiastic students were about Mainstay.
But don’t take our word for it. Here’s Scott Burke talking about his experience with Mainstay: